Loss of throughput rate of the bridge

Y

yuqing.zhou

Hello everyone, I am currently using jail. Due to the requirements of the business, the IPs within the jail need to be able to communicate directly with the IPs of the physical network cards. I tested both if_bridge and netgraph methods. The service could be running. However, when the physical network card is a 25G network card, the throughput rate of the jail's IP via if_bridge can only reach 15G, while via netgraph it is only 14G and there are retransmissions. Is there any good solution that can meet the requirements and reduce the loss of throughput rate?

The test results of the two bridging modes of "jail" to "jail" are as follows：

if_bridge：
Code: 
root@test:~ # iperf3 -c 192.168.100.109 -t 20
Connecting to host 192.168.100.109, port 5201
[  5] local 192.168.100.110 port 33730 connected to 192.168.100.109 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.05   sec  1.71 GBytes  14.0 Gbits/sec    0   2.01 MBytes
[  5]   1.05-2.06   sec  1.58 GBytes  13.4 Gbits/sec    0   2.01 MBytes
[  5]   2.06-3.06   sec  1.55 GBytes  13.3 Gbits/sec    0   2.01 MBytes
[  5]   3.06-4.06   sec  1.55 GBytes  13.4 Gbits/sec    0   2.01 MBytes
[  5]   4.06-5.06   sec  1.57 GBytes  13.5 Gbits/sec    0   2.01 MBytes
[  5]   5.06-6.05   sec  1.57 GBytes  13.5 Gbits/sec    0   2.01 MBytes
[  5]   6.05-7.05   sec  1.53 GBytes  13.1 Gbits/sec    0   2.01 MBytes
[  5]   7.05-8.06   sec  1.53 GBytes  13.0 Gbits/sec    0   2.01 MBytes
[  5]   8.06-9.06   sec  1.47 GBytes  12.6 Gbits/sec    0   2.01 MBytes
[  5]   9.06-10.06  sec  1.54 GBytes  13.2 Gbits/sec    0   2.01 MBytes
[  5]  10.06-11.06  sec  2.26 GBytes  19.5 Gbits/sec    0   2.01 MBytes
[  5]  11.06-12.06  sec  2.07 GBytes  17.8 Gbits/sec    0   2.01 MBytes
[  5]  12.06-13.05  sec  1.51 GBytes  13.0 Gbits/sec    0   2.01 MBytes
[  5]  13.05-14.06  sec  1.54 GBytes  13.1 Gbits/sec    0   2.01 MBytes
[  5]  14.06-15.06  sec  1.61 GBytes  13.8 Gbits/sec    0   2.01 MBytes
[  5]  15.06-16.06  sec  1.98 GBytes  17.1 Gbits/sec    0   2.01 MBytes
[  5]  16.06-17.06  sec  2.16 GBytes  18.5 Gbits/sec    0   2.01 MBytes
[  5]  17.06-18.06  sec  2.07 GBytes  17.7 Gbits/sec    0   2.01 MBytes
[  5]  18.06-19.06  sec  1.46 GBytes  12.6 Gbits/sec    0   2.01 MBytes
[  5]  19.06-20.06  sec  1.60 GBytes  13.7 Gbits/sec    0   2.01 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-20.06  sec  33.9 GBytes  14.5 Gbits/sec    0            sender
[  5]   0.00-20.06  sec  33.9 GBytes  14.5 Gbits/sec                  receiver

iperf Done.
root@test:~ # iperf3 -c 192.168.100.109 -t 20 -R
Connecting to host 192.168.100.109, port 5201
Reverse mode, remote host 192.168.100.109 is sending
[  5] local 192.168.100.110 port 51384 connected to 192.168.100.109 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.06   sec  2.31 GBytes  18.7 Gbits/sec
[  5]   1.06-2.06   sec  2.25 GBytes  19.4 Gbits/sec
[  5]   2.06-3.06   sec  2.26 GBytes  19.4 Gbits/sec
[  5]   3.06-4.00   sec  2.10 GBytes  19.2 Gbits/sec
[  5]   4.00-5.06   sec  2.33 GBytes  18.9 Gbits/sec
[  5]   5.06-6.06   sec  2.24 GBytes  19.3 Gbits/sec
[  5]   6.06-7.06   sec  2.28 GBytes  19.6 Gbits/sec
[  5]   7.06-8.02   sec  1.92 GBytes  17.2 Gbits/sec
[  5]   8.02-9.02   sec  1.64 GBytes  14.1 Gbits/sec
[  5]   9.02-10.02  sec  1.65 GBytes  14.2 Gbits/sec
[  5]  10.02-11.01  sec  1.65 GBytes  14.2 Gbits/sec
[  5]  11.01-12.01  sec  1.69 GBytes  14.5 Gbits/sec
[  5]  12.01-13.01  sec  2.27 GBytes  19.6 Gbits/sec
[  5]  13.01-14.01  sec  1.86 GBytes  16.0 Gbits/sec
[  5]  14.01-15.01  sec  1.65 GBytes  14.2 Gbits/sec
[  5]  15.01-16.01  sec  1.65 GBytes  14.2 Gbits/sec
[  5]  16.01-17.00  sec  1.63 GBytes  14.1 Gbits/sec
[  5]  17.00-18.00  sec  2.16 GBytes  18.6 Gbits/sec
[  5]  18.00-19.06  sec  2.41 GBytes  19.5 Gbits/sec
[  5]  19.06-20.06  sec  2.27 GBytes  19.6 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-20.06  sec  40.2 GBytes  17.2 Gbits/sec    0            sender
[  5]   0.00-20.06  sec  40.2 GBytes  17.2 Gbits/sec                  receiver
netgraph：
Code: 
root@test3:~ # iperf3 -c 192.168.100.112
Connecting to host 192.168.100.112, port 5201
[  5] local 192.168.100.111 port 32542 connected to 192.168.100.112 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.02   sec  1.74 GBytes  14.7 Gbits/sec  227    175 KBytes
[  5]   1.02-2.01   sec  1.68 GBytes  14.5 Gbits/sec  243   1.40 MBytes
[  5]   2.01-3.01   sec  1.72 GBytes  14.7 Gbits/sec  235    744 KBytes
[  5]   3.01-4.00   sec  1.70 GBytes  14.8 Gbits/sec  235    201 KBytes
[  5]   4.00-5.06   sec  1.81 GBytes  14.7 Gbits/sec  259    131 KBytes
[  5]   5.06-6.06   sec  1.70 GBytes  14.7 Gbits/sec  234   1.50 MBytes
[  5]   6.06-7.06   sec  1.69 GBytes  14.6 Gbits/sec  243   52.5 KBytes
[  5]   7.06-8.01   sec  1.67 GBytes  15.1 Gbits/sec  224    166 KBytes
[  5]   8.01-9.00   sec  1.74 GBytes  15.0 Gbits/sec  255   1.27 MBytes
[  5]   9.00-10.06  sec  1.86 GBytes  15.1 Gbits/sec  258    254 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.06  sec  17.3 GBytes  14.8 Gbits/sec  2413            sender
[  5]   0.00-10.06  sec  17.3 GBytes  14.8 Gbits/sec                  receiver

iperf Done.

root@test3:~ # iperf3 -c 192.168.100.112 -R
Connecting to host 192.168.100.112, port 5201
Reverse mode, remote host 192.168.100.112 is sending
[  5] local 192.168.100.111 port 29443 connected to 192.168.100.112 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.06   sec  1.37 GBytes  11.1 Gbits/sec
[  5]   1.06-2.06   sec  1.46 GBytes  12.6 Gbits/sec
[  5]   2.06-3.06   sec  1.50 GBytes  12.9 Gbits/sec
[  5]   3.06-4.00   sec  1.30 GBytes  11.9 Gbits/sec
[  5]   4.00-5.00   sec  1.21 GBytes  10.4 Gbits/sec
[  5]   5.00-6.01   sec   754 MBytes  6.30 Gbits/sec
[  5]   6.01-7.00   sec  1.45 GBytes  12.4 Gbits/sec
[  5]   7.00-8.06   sec  1.54 GBytes  12.5 Gbits/sec
[  5]   8.06-9.00   sec  1.35 GBytes  12.3 Gbits/sec
[  5]   9.00-10.06  sec  1.52 GBytes  12.3 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.06  sec  13.4 GBytes  11.5 Gbits/sec  827            sender
[  5]   0.00-10.06  sec  13.4 GBytes  11.5 Gbits/sec                  receiver

iperf Done.

host machine：
Code: 
[root@mfsbsd roothome]$ iperf3 -c 192.168.100.108 -t 20
Connecting to host 192.168.100.108, port 5201
[  5] local 192.168.100.107 port 58399 connected to 192.168.100.108 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.01   sec  2.75 GBytes  23.4 Gbits/sec    0   2.00 MBytes
[  5]   1.01-2.04   sec  2.93 GBytes  24.5 Gbits/sec    0   2.00 MBytes
[  5]   2.04-3.06   sec  2.95 GBytes  24.7 Gbits/sec    0   2.00 MBytes
[  5]   3.06-4.06   sec  2.84 GBytes  24.4 Gbits/sec    0   2.00 MBytes
[  5]   4.06-5.06   sec  2.81 GBytes  24.2 Gbits/sec    0   2.00 MBytes
[  5]   5.06-6.06   sec  2.81 GBytes  24.2 Gbits/sec    0   2.00 MBytes
[  5]   6.06-7.06   sec  2.86 GBytes  24.6 Gbits/sec    0   2.00 MBytes
[  5]   7.06-8.06   sec  2.87 GBytes  24.7 Gbits/sec    0   2.00 MBytes
[  5]   8.06-9.06   sec  2.86 GBytes  24.6 Gbits/sec    0   2.00 MBytes
[  5]   9.06-10.06  sec  2.86 GBytes  24.6 Gbits/sec    0   2.00 MBytes
[  5]  10.06-11.06  sec  2.86 GBytes  24.6 Gbits/sec    0   2.00 MBytes
[  5]  11.06-12.06  sec  2.87 GBytes  24.7 Gbits/sec    0   2.00 MBytes
[  5]  12.06-13.06  sec  2.87 GBytes  24.6 Gbits/sec    0   2.00 MBytes
[  5]  13.06-14.06  sec  2.83 GBytes  24.3 Gbits/sec    0   2.00 MBytes
[  5]  14.06-15.06  sec  2.83 GBytes  24.3 Gbits/sec    0   2.00 MBytes
[  5]  15.06-16.06  sec  2.85 GBytes  24.5 Gbits/sec    0   2.00 MBytes
[  5]  16.06-17.06  sec  2.83 GBytes  24.3 Gbits/sec    0   2.00 MBytes
[  5]  17.06-18.06  sec  2.84 GBytes  24.4 Gbits/sec    0   2.00 MBytes
[  5]  18.06-19.06  sec  2.86 GBytes  24.6 Gbits/sec    0   2.00 MBytes
[  5]  19.06-20.06  sec  2.86 GBytes  24.6 Gbits/sec    0   2.00 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-20.06  sec  57.1 GBytes  24.4 Gbits/sec    0            sender
[  5]   0.00-20.06  sec  57.1 GBytes  24.4 Gbits/sec                  receiver

iperf Done.
[root@mfsbsd roothome]$ iperf3 -c 192.168.100.108 -t 20 -R
Connecting to host 192.168.100.108, port 5201
Reverse mode, remote host 192.168.100.108 is sending
[  5] local 192.168.100.107 port 29839 connected to 192.168.100.108 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.06   sec  2.81 GBytes  22.7 Gbits/sec
[  5]   1.06-2.06   sec  2.63 GBytes  22.7 Gbits/sec
[  5]   2.06-3.06   sec  2.87 GBytes  24.6 Gbits/sec
[  5]   3.06-4.06   sec  2.75 GBytes  23.7 Gbits/sec
[  5]   4.06-5.06   sec  2.64 GBytes  22.7 Gbits/sec
[  5]   5.06-6.06   sec  2.80 GBytes  24.1 Gbits/sec
[  5]   6.06-7.06   sec  2.48 GBytes  21.3 Gbits/sec
[  5]   7.06-8.02   sec  2.67 GBytes  23.9 Gbits/sec
[  5]   8.02-9.06   sec  2.49 GBytes  20.5 Gbits/sec
[  5]   9.06-10.01  sec  2.30 GBytes  21.0 Gbits/sec
[  5]  10.01-11.06  sec  3.02 GBytes  24.6 Gbits/sec
[  5]  11.06-12.06  sec  2.74 GBytes  23.6 Gbits/sec
[  5]  12.06-13.06  sec  2.38 GBytes  20.5 Gbits/sec
[  5]  13.06-14.00  sec  2.19 GBytes  19.9 Gbits/sec
[  5]  14.00-15.03  sec  2.36 GBytes  19.7 Gbits/sec
[  5]  15.03-16.06  sec  2.38 GBytes  19.9 Gbits/sec
[  5]  16.06-17.06  sec  2.33 GBytes  20.0 Gbits/sec
[  5]  17.06-18.06  sec  2.28 GBytes  19.6 Gbits/sec
[  5]  18.06-19.06  sec  2.27 GBytes  19.5 Gbits/sec
[  5]  19.06-20.00  sec  2.13 GBytes  19.5 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-20.00  sec  50.5 GBytes  21.7 Gbits/sec    0            sender
[  5]   0.00-20.00  sec  50.5 GBytes  21.7 Gbits/sec                  receiver

iperf Done.
You have new mail in /var/mail/root
[root@mfsbsd roothome]$
 
iperf3 for anything >10G is a CPU-benchmark.
Try using -P and start with ~half the number of available cores. With fast storage (NVMe) and slow CPU it might also help to use a file (-F) as data source instead of generating random data.

Also: raw, single-connection bandwidth has nothing to do with real world network performance. It might help diagnosing fundamental problems with the network, but iperf3 is too much CPU-bound to be a valid benchmark, especially if the host/jail you are testing from/to is supposed to pass traffic (e.g. a router/gateway)

That being said, how is your jail networking set up (full stack from the physical interface to the jail) - are vlan interfaces involved?
If your NICs support SR-IOV, you might want to pass through VFs to the jail. This adds by far the least (i.e. no) overhead.
 
You must log in or register to reply here.
Back
Top