Analyse network performance between CEA and SWH

This issue is to analyze and diagnose the network connectivity between the CEA and SWH.

Though a direct ssh connection between uffizi and the bastion, the bandwidth is around 700Mb which seems decent for a 1Gbps mutualized connection

iperf output:

[  5] 0.0000-10.0966 sec   817 MBytes   679 Mbits/sec  26061=0:26033:1:3:0:0:0:24

Through the ipsec VPN, the bandwith is around half slower:

[  6] 0.0000-10.2555 sec   391 MBytes   319 Mbits/sec  12492=0:12489:1:1:0:0:0:1

Protocol :

• iperf server on uffizi:

uffizi% iperf -s -e -i 1
------------------------------------------------------------
Server listening on TCP port 5001 with pid 1146745
Read buffer size:  128 KByte (Dist bin width=16.0 KByte)
TCP window size:  128 KByte (default)
------------------------------------------------------------

• ssh tunnel established between uffizi and the bastion

uffizi ~ % ssh -R 5001:localhost:5001 swh@[public ip|vpnized ip]

• iperf client on the bastion

swh@angrenost:~$ iperf -c localhost -e -i 1 -r

changed milestone to %Scale-out objstorage in production as primary objstorage [Roadmap - Tooling and infrastructure]

assigned to @vsellier

changed the description

mentioned in issue #5187 (closed)

For the record and to remove any doubts regarding the replayers performaces using a ssh tunnel to replay the data consume almost the same amount of bandwidth than the iperf.

If the compression is activated on the ssh tunnel, the bandwidth is around 160Mbits but winery is the bottleneck here (needs more tuning).

marked this issue as related to #5203

changed title from [CEA] Analyse network performance between CEA and SWH to Analyse network performance between CEA and SWH

added platform::CEA label

A new attempt with a direct ssh connection to bastion seems now limited around 100Mbits:

swh@angrenost:~$ iperf -c localhost -e -i 1 -r
listener bind failed: Address already in use
------------------------------------------------------------
Client connecting to localhost, TCP port 5001 with pid 1265749 (1 flows)
Write buffer size: 131072 Byte
TOS set to 0x0 (Nagle on)
TCP window size: 2.50 MByte (default)
------------------------------------------------------------
[  1] local 127.0.0.1%lo port 35420 connected with 127.0.0.1 port 5001 (sock=4) (icwnd/mss/irtt=319/32741/68) (ct=0.13 ms) on 2024-01-25 10:59:44 (CET)
[ ID] Interval            Transfer    Bandwidth       Write/Err  Rtry     Cwnd/RTT(var)        NetPwr
[  1] 0.0000-1.0000 sec  10.6 MBytes  89.3 Mbits/sec  86/0          1     3197K/4875(5145) us  2291
[  1] 1.0000-2.0000 sec  2.44 MBytes  20.4 Mbits/sec  20/0          1     3197K/5892(8395) us  434
[  1] 2.0000-3.0000 sec  3.56 MBytes  29.9 Mbits/sec  29/0          3     3197K/8938(13615) us  418
[  1] 3.0000-4.0000 sec  8.62 MBytes  72.4 Mbits/sec  69/0          2     3197K/1163(1866) us  7776
[  1] 4.0000-5.0000 sec  10.4 MBytes  87.0 Mbits/sec  83/0          5     3197K/5517(9732) us  1972
[  1] 5.0000-6.0000 sec  4.48 MBytes  37.6 Mbits/sec  36/0          1     1598K/1736(2973) us  2706
[  1] 6.0000-7.0000 sec  2.75 MBytes  23.0 Mbits/sec  22/0          0      767K/379(343) us  7598
[  1] 7.0000-8.0000 sec  7.62 MBytes  64.0 Mbits/sec  61/0          1      767K/517(668) us  15465
[  1] 8.0000-9.0000 sec  11.8 MBytes  98.6 Mbits/sec  94/0          0      767K/211(13) us  58392
[  1] 9.0000-10.0000 sec  8.62 MBytes  72.4 Mbits/sec  69/0          0      767K/197(17) us  45908
[  1] 10.0000-11.0596 sec   104 KBytes   802 Kbits/sec  1/0          0      639K/203(21) us  494
[  1] 0.0000-11.0596 sec  71.0 MBytes  53.8 Mbits/sec  570/0         14      639K/203(21) us  33149

Let's open the subject with the CEA

mentioned in issue swh/devel/swh-core#4431

I checked a packet capture of the ipsec traffic.

Most of the packets are fragmented.

It seems the MTUs are configured by default on all the interfaces (1500) which is not very well for the packets encapsulation

After updating the firewall configuration to adapt the MSS and opening the ICMP traffic, it seems the performances are better around 300Mbit/s.

Let's see after a couple of days the bandwidth fluctuation

This option net.inet.ipsec.async_crypto was activated on the firewall during the early tests to distribute the load across all the cores.

The cpu load on the firewall is around 80% off cpu.

The default configuration was restored to check the impact and the traffic dropped again around 100Mbit.

I guess the next step is to evaluate the impact of the esp configuration on the traffic which is currently esp=aes256-sha512-modp2048!

It seems modp2048 (DH14) is the recommended group configuration.

The first step could be testing sha384 instead of sha512 as hash algorithm

and aes192 as second step

FTR, the last ANSSI recommandation is from 2015: https://cyber.gouv.fr/publications/recommandations-de-securite-relatives-ipsec

After different algorithm tests, it seems the most effective until now is AES_CBC_128/HMAC_SHA1_96/NO_EXT_SEQ with a transfer rate around 330Mbits. The processor is no more overloaded, neither on the firewall nor on the bastion.

Initial: aes256-sha512-modp2048!
aes128gcm16-sha256-modp4096 2023-02--29 16:45 (KO)
aes192-sha256-modp2048! 2023-02--29 17:00
aes256-sha256-modp2048! 2023-03-04 23:30 
(auto) AES_CBC_128/HMAC_SHA1_96/NO_EXT_SEQ <---- best for now
(auto) AES_CBC_128/HMAC_SHA1_96 <--- slower

After poking at the configuration explicitely, the esp option was removed from the configuration to let openswarn deciding of the ciphers to use.

The next step is to activate the bench of the cipher to be sure something effective is used.

Even if the openswarm documentation specify that by default no weak encryption is used, all this is temporary to try to identify the bottlenecks.

marked this issue as related to #5276

The current configuration is fixed to aes128-sha1-modp2048 until further investigations.

The speed is better ~350Mbits but according to the ssh speed, we should probably be able to transfer x1.5 faster (very approximative guestimate)

no more work on this now winery is live

closed