Backport of if_bridge patch for better performance to 12.1

zirias@ · May 18, 2020

Hi all,

here's a recent commit to head (-CURRENT): https://svnweb.freebsd.org/base?view=revision&revision=360345

I thought it might be worthwile to test that with 12.1, so I backported the patch. So far, I only did a quick smoke test with two VNET jails using epair devices connected by a bridge. Using dd and nc, I managed to reach a data rate of 240 MB/s on my old desktop PC, so that doesn't look like an improvement, but at least it's a smoke test that the bridge still works

Maybe the epair devices or my aging CPU are limiting performance, I don't know…

So, if you feel advanturous enough to trust my backporting work, please try the patch attached here, maybe it improves some setups

Of course, no guarantees I didn't break anything.

sahlawat · May 24, 2020

A 4x network performance patch, I had to try this; really wanted this to work, but ...

1. The patch applies clean and compiles with 12.1-p5 r361429M
2. No system crashes - I have 4 bridges with 5 VLANs and a 2 interface lagg port connecting 45 jails with VNET interfaces.

Performed 2 simple iperf3 tests:
1. Jail (hub) to Jail (.123)
2. External pfsense (fw) to Jail (.123)
Jail 192.168.0.123 was running the iperf3 server in both cases

Performance Before Patch:

hub ~ # iperf3 -c 192.168.0.123
Connecting to host 192.168.0.123, port 5201
[ 5] local 192.168.0.50 port 43309 connected to 192.168.0.123 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 374 MBytes 3.13 Gbits/sec 0 2.00 MBytes
[ 5] 1.00-2.00 sec 384 MBytes 3.22 Gbits/sec 0 2.00 MBytes
[ 5] 2.00-3.00 sec 382 MBytes 3.20 Gbits/sec 0 2.00 MBytes
[ 5] 3.00-4.00 sec 382 MBytes 3.20 Gbits/sec 0 2.00 MBytes
[ 5] 4.00-5.00 sec 385 MBytes 3.23 Gbits/sec 0 2.00 MBytes
[ 5] 5.00-6.00 sec 386 MBytes 3.24 Gbits/sec 0 2.00 MBytes
[ 5] 6.00-7.00 sec 384 MBytes 3.22 Gbits/sec 0 2.00 MBytes
[ 5] 7.00-8.00 sec 389 MBytes 3.26 Gbits/sec 0 2.00 MBytes
[ 5] 8.00-9.00 sec 387 MBytes 3.25 Gbits/sec 0 2.00 MBytes
[ 5] 9.00-10.00 sec 387 MBytes 3.25 Gbits/sec 0 2.00 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 3.75 GBytes 3.22 Gbits/sec 0 sender
[ 5] 0.00-10.00 sec 3.75 GBytes 3.22 Gbits/sec receiver
iperf Done.

[2.4.5-RELEASE][admin@fw]/root: iperf3 -c 192.168.0.123
Connecting to host 192.168.0.123, port 5201
[ 5] local 192.168.0.5 port 41841 connected to 192.168.0.123 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 112 MBytes 939 Mbits/sec 0 65.1 KBytes
[ 5] 1.00-2.00 sec 108 MBytes 903 Mbits/sec 0 65.1 KBytes
[ 5] 2.00-3.00 sec 112 MBytes 938 Mbits/sec 0 65.1 KBytes
[ 5] 3.00-4.00 sec 111 MBytes 934 Mbits/sec 0 65.1 KBytes
[ 5] 4.00-5.00 sec 112 MBytes 936 Mbits/sec 0 65.1 KBytes
[ 5] 5.00-6.00 sec 110 MBytes 919 Mbits/sec 0 65.1 KBytes
[ 5] 6.00-7.00 sec 112 MBytes 939 Mbits/sec 0 65.1 KBytes
[ 5] 7.00-8.00 sec 112 MBytes 938 Mbits/sec 0 65.1 KBytes
[ 5] 8.00-9.00 sec 112 MBytes 939 Mbits/sec 0 65.1 KBytes
[ 5] 9.00-10.00 sec 112 MBytes 938 Mbits/sec 0 65.1 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.09 GBytes 932 Mbits/sec 0 sender
[ 5] 0.00-10.01 sec 1.09 GBytes 932 Mbits/sec receiver
iperf Done.

Performance After Patch:

hub ~ # iperf3 -c 192.168.0.123
Connecting to host 192.168.0.123, port 5201
[ 5] local 192.168.0.50 port 44597 connected to 192.168.0.123 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 29.3 MBytes 246 Mbits/sec 0 64.9 KBytes
[ 5] 1.00-2.00 sec 26.1 MBytes 219 Mbits/sec 0 64.9 KBytes
[ 5] 2.00-3.00 sec 32.7 MBytes 274 Mbits/sec 0 64.9 KBytes
[ 5] 3.00-4.00 sec 35.6 MBytes 298 Mbits/sec 0 64.9 KBytes
[ 5] 4.00-5.00 sec 28.3 MBytes 238 Mbits/sec 0 64.9 KBytes
[ 5] 5.00-6.00 sec 35.2 MBytes 296 Mbits/sec 0 64.9 KBytes
[ 5] 6.00-7.00 sec 28.6 MBytes 240 Mbits/sec 0 64.9 KBytes
[ 5] 7.00-8.00 sec 32.4 MBytes 272 Mbits/sec 0 64.9 KBytes
[ 5] 8.00-9.00 sec 25.5 MBytes 214 Mbits/sec 0 64.9 KBytes
[ 5] 9.00-10.00 sec 36.2 MBytes 304 Mbits/sec 0 64.9 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 310 MBytes 260 Mbits/sec 0 sender
[ 5] 0.00-10.00 sec 310 MBytes 260 Mbits/sec receiver
iperf Done.

[2.4.5-RELEASE][admin@fw]/root: iperf3 -c 192.168.0.123
Connecting to host 192.168.0.123, port 5201
[ 5] local 192.168.0.5 port 16638 connected to 192.168.0.123 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 112 MBytes 937 Mbits/sec 0 65.1 KBytes
[ 5] 1.00-2.00 sec 112 MBytes 939 Mbits/sec 0 65.1 KBytes
[ 5] 2.00-3.00 sec 112 MBytes 939 Mbits/sec 0 65.1 KBytes
[ 5] 3.00-4.00 sec 112 MBytes 939 Mbits/sec 0 65.1 KBytes
[ 5] 4.00-5.00 sec 109 MBytes 917 Mbits/sec 0 65.1 KBytes
[ 5] 5.00-6.00 sec 112 MBytes 938 Mbits/sec 0 65.1 KBytes
[ 5] 6.00-7.00 sec 112 MBytes 936 Mbits/sec 0 65.1 KBytes
[ 5] 7.00-8.00 sec 112 MBytes 937 Mbits/sec 0 65.1 KBytes
[ 5] 8.00-9.00 sec 112 MBytes 938 Mbits/sec 0 65.1 KBytes
[ 5] 9.00-10.00 sec 112 MBytes 939 Mbits/sec 0 65.1 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.09 GBytes 936 Mbits/sec 0 sender
[ 5] 0.00-10.01 sec 1.09 GBytes 935 Mbits/sec receiver
iperf Done.

Given the inter-jail throughput drop I restored the 12.1 bridge files.

Next - will standup another server with 13-current and see what performance I get on that branch.

pos · May 28, 2020

I have read of Krystofs great bridge work, and I welcome it *very* much.

The existing bridge code is not super. But works. I do not understand why you see so bad numbers... I have a virtual linux on top of FreeBSD 12.0. The virtual CentOS 7 server is connected over a FreeBSD bridge on Bhyve host. That virtual server gets 7.2 Gbit/s (over 7 routers hops to a target on the internet from my location). Speed test site is "bredbandskollen.se" with CLI client from http://www.bredbandskollen.se/om/mer-om-bbk/bredbandskollen-cli/ (also on git hub). Yes, I have a 10 Gbit internet connection. If not using a FreeBSD bridge I get 8.5 to 9 Gbit. Using a bridge in linux KVM in the same way I do not se any performance drop at all over the bridge. So if the systems are correctly configured the FreeBSD bridges do give half decent performance (but eat resources if you push much traffic).

So my personal opinion is to wait and upgrade to FreeBSD 13 if it is included there. But of course, if you have time available. Why not

sahlawat · Jun 11, 2020

Took a bit longer than expected, had to upgrade my testbed to 25G to make sure I was not hitting any network bandwidth limit.

Unfortunately this specific testing shows degraded base network performance in 13-current, so I can't really do any useful bridge performance comparisons across 12.1-release and 13-current. Will revisit this once we have 13.0 RC builds.

Previously (post above) I had applied the back-ported patch to 12.1 and done "jail to jail on same server" testing on my primary server.
FreeBSD 12.1 vanilla:

Code:

[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 3.75 GBytes 3.22 Gbits/sec 0 sender
[ 5] 0.00-10.00 sec 3.75 GBytes 3.22 Gbits/sec receiver

FreeBSD 12.1 w/ back-ported patch:

Code:

[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 310 MBytes 260 Mbits/sec 0 sender
[ 5] 0.00-10.00 sec 310 MBytes 260 Mbits/sec receiver

The "jail to external server" testing was throttled by the 1G network bandwidth to the external server, so did not provide any insights. Given the jail to jail throughput drop of the back-ported patch I am back to using the 12.1 vanilla bridge files.

13-current Vs 12.1-release - simple iperf test

#on both lab17 & lab18:

Code:

ifconfig bridge1 create
ifconfig bridge1 addm bge0 up
ifconfig bridge2 create
ifconfig bridge2 addm bge1 up
ifconfig bridge11 create
ifconfig bridge11 addm bnxt0 up
ifconfig bridge12 create
#ifconfig bridge12 addm bnxt1 up
#ifconfig bridge12 deletem bnxt1

#on lab17:

Code:

ifconfig bnxt0 10.0.11.17/24 up
ifconfig bnxt1 10.0.12.17/24 up

#on lab18:

Code:

ifconfig bnxt0 10.0.11.18/24 up
ifconfig bnxt1 10.0.12.18/24 up

lab17 & lab18 servers are connected over 25G SFP28 links via a switch.
bnxt0 interfaces are in VLAN 11 and bnxt1 interfaces are in VLAN 12

Code:

root@lab17:~ # uname -a
FreeBSD lab17 13.0-CURRENT FreeBSD 13.0-CURRENT #0 r361567: Thu May 28 04:59:08 UTC 2020     [email]root@releng1.nyi.freebsd.org[/email]:/usr/obj/usr/src/amd64.amd64/sys/GENERIC  amd64
root@lab17:~ #

root@lab18:~ # uname -a
FreeBSD lab18 13.0-CURRENT FreeBSD 13.0-CURRENT #0 r361567: Thu May 28 04:59:08 UTC 2020     [email]root@releng1.nyi.freebsd.org[/email]:/usr/obj/usr/src/amd64.amd64/sys/GENERIC  amd64
root@lab18:~ #[/codfe]


Bridged Interfaces: server to server
[code]
Accepted connection from 10.0.11.18, port 57363
[  5] local 10.0.11.17 port 5201 connected to 10.0.11.18 port 20451
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   152 MBytes  1.28 Gbits/sec
[  5]   1.00-2.00   sec   258 MBytes  2.16 Gbits/sec
[  5]   2.00-3.00   sec   262 MBytes  2.20 Gbits/sec
[  5]   3.00-4.00   sec   270 MBytes  2.27 Gbits/sec
[  5]   4.00-5.00   sec   278 MBytes  2.33 Gbits/sec
[  5]   5.00-6.00   sec   281 MBytes  2.36 Gbits/sec
[  5]   6.00-7.00   sec   280 MBytes  2.35 Gbits/sec
[  5]   7.00-8.00   sec   276 MBytes  2.32 Gbits/sec
[  5]   8.00-9.00   sec   252 MBytes  2.12 Gbits/sec
[  5]   9.00-10.00  sec   265 MBytes  2.22 Gbits/sec
[  5]  10.00-10.00  sec   113 KBytes  2.56 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  2.51 GBytes  2.16 Gbits/sec                  receiver
-----------------------------------------------------------

root@lab18:~ # iperf3 -c 10.0.11.17
Connecting to host 10.0.11.17, port 5201
[  5] local 10.0.11.18 port 20451 connected to 10.0.11.17 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   153 MBytes  1.28 Gbits/sec    7    218 KBytes
[  5]   1.00-2.00   sec   258 MBytes  2.16 Gbits/sec   15    299 KBytes
[  5]   2.00-3.00   sec   262 MBytes  2.20 Gbits/sec   12    242 KBytes
[  5]   3.00-4.00   sec   270 MBytes  2.27 Gbits/sec   10    229 KBytes
[  5]   4.00-5.00   sec   277 MBytes  2.33 Gbits/sec   13    252 KBytes
[  5]   5.00-6.00   sec   281 MBytes  2.36 Gbits/sec   15    286 KBytes
[  5]   6.00-7.00   sec   280 MBytes  2.35 Gbits/sec   12    321 KBytes
[  5]   7.00-8.00   sec   276 MBytes  2.32 Gbits/sec   11    336 KBytes
[  5]   8.00-9.00   sec   253 MBytes  2.12 Gbits/sec  135    348 KBytes
[  5]   9.00-10.00  sec   265 MBytes  2.22 Gbits/sec   13    308 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  2.52 GBytes  2.16 Gbits/sec  243             sender
[  5]   0.00-10.00  sec  2.51 GBytes  2.16 Gbits/sec                  receiver

iperf Done.[/code


Direct Interfaces: server to server
[code]
Accepted connection from 10.0.12.18, port 24137
[  5] local 10.0.12.17 port 5201 connected to 10.0.12.18 port 19922
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   403 MBytes  3.38 Gbits/sec
[  5]   1.00-2.00   sec   757 MBytes  6.35 Gbits/sec
[  5]   2.00-3.00   sec   758 MBytes  6.35 Gbits/sec
[  5]   3.00-4.00   sec   760 MBytes  6.38 Gbits/sec
[  5]   4.00-5.00   sec   759 MBytes  6.37 Gbits/sec
[  5]   5.00-6.00   sec   760 MBytes  6.37 Gbits/sec
[  5]   6.00-7.00   sec   758 MBytes  6.36 Gbits/sec
[  5]   7.00-8.00   sec   759 MBytes  6.37 Gbits/sec
[  5]   8.00-9.00   sec   759 MBytes  6.37 Gbits/sec
[  5]   9.00-10.00  sec   759 MBytes  6.37 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  7.06 GBytes  6.07 Gbits/sec                  receiver
-----------------------------------------------------------

root@lab18:~ # iperf3 -c 10.0.12.17
Connecting to host 10.0.12.17, port 5201
[  5] local 10.0.12.18 port 19922 connected to 10.0.12.17 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   404 MBytes  3.39 Gbits/sec   14    268 KBytes
[  5]   1.00-2.00   sec   757 MBytes  6.35 Gbits/sec    2    354 KBytes
[  5]   2.00-3.00   sec   758 MBytes  6.35 Gbits/sec    6    259 KBytes
[  5]   3.00-4.00   sec   760 MBytes  6.38 Gbits/sec    4    349 KBytes
[  5]   4.00-5.00   sec   759 MBytes  6.37 Gbits/sec    6    254 KBytes
[  5]   5.00-6.00   sec   760 MBytes  6.37 Gbits/sec    4    344 KBytes
[  5]   6.00-7.00   sec   758 MBytes  6.36 Gbits/sec    6    247 KBytes
[  5]   7.00-8.00   sec   759 MBytes  6.37 Gbits/sec    4    338 KBytes
[  5]   8.00-9.00   sec   759 MBytes  6.37 Gbits/sec   15    269 KBytes
[  5]   9.00-10.00  sec   759 MBytes  6.37 Gbits/sec    4    356 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  7.06 GBytes  6.07 Gbits/sec   65             sender
[  5]   0.00-10.00  sec  7.06 GBytes  6.07 Gbits/sec                  receiver

iperf Done.

jail to jail on same server

Code:

Accepted connection from 10.0.11.117, port 43039
[  5] local 10.0.11.127 port 5201 connected to 10.0.11.117 port 55213
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   173 MBytes  1.45 Gbits/sec
[  5]   1.00-2.00   sec   182 MBytes  1.52 Gbits/sec
[  5]   2.00-3.00   sec   180 MBytes  1.51 Gbits/sec
[  5]   3.00-4.00   sec   185 MBytes  1.55 Gbits/sec
[  5]   4.00-5.00   sec   184 MBytes  1.54 Gbits/sec
[  5]   5.00-6.00   sec   181 MBytes  1.52 Gbits/sec
[  5]   6.00-7.00   sec   154 MBytes  1.29 Gbits/sec
[  5]   7.00-8.00   sec   133 MBytes  1.12 Gbits/sec
[  5]   8.00-9.00   sec   136 MBytes  1.14 Gbits/sec
[  5]   9.00-10.00  sec   137 MBytes  1.15 Gbits/sec
[  5]  10.00-10.00  sec   771 KBytes  1.70 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  1.61 GBytes  1.38 Gbits/sec                  receiver
-----------------------------------------------------------

test-117 ~ # iperf3 -c 10.0.11.127
Connecting to host 10.0.11.127, port 5201
[  5] local 10.0.11.117 port 55213 connected to 10.0.11.127 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   175 MBytes  1.46 Gbits/sec    0   1.61 MBytes
[  5]   1.00-2.00   sec   182 MBytes  1.52 Gbits/sec    0   1.61 MBytes
[  5]   2.00-3.00   sec   179 MBytes  1.50 Gbits/sec    0   1.61 MBytes
[  5]   3.00-4.00   sec   185 MBytes  1.55 Gbits/sec    0   1.61 MBytes
[  5]   4.00-5.00   sec   184 MBytes  1.55 Gbits/sec    0   1.61 MBytes
[  5]   5.00-6.00   sec   181 MBytes  1.52 Gbits/sec    0   1.61 MBytes
[  5]   6.00-7.00   sec   153 MBytes  1.29 Gbits/sec    0   1.61 MBytes
[  5]   7.00-8.00   sec   133 MBytes  1.12 Gbits/sec    0   1.61 MBytes
[  5]   8.00-9.00   sec   136 MBytes  1.14 Gbits/sec    0   1.61 MBytes
[  5]   9.00-10.00  sec   137 MBytes  1.15 Gbits/sec    0   1.61 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.61 GBytes  1.38 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  1.61 GBytes  1.38 Gbits/sec                  receiver

iperf Done.

jail to jail on different servers

Code:

Accepted connection from 10.0.12.128, port 56410
[  5] local 10.0.12.127 port 5201 connected to 10.0.12.128 port 61265
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   126 MBytes  1.06 Gbits/sec
[  5]   1.00-2.00   sec   192 MBytes  1.61 Gbits/sec
[  5]   2.00-3.00   sec   173 MBytes  1.45 Gbits/sec
[  5]   3.00-4.00   sec   156 MBytes  1.31 Gbits/sec
[  5]   4.00-5.00   sec   170 MBytes  1.43 Gbits/sec
[  5]   5.00-6.00   sec   175 MBytes  1.46 Gbits/sec
[  5]   6.00-7.00   sec   173 MBytes  1.45 Gbits/sec
[  5]   7.00-8.00   sec   182 MBytes  1.53 Gbits/sec
[  5]   8.00-9.00   sec   195 MBytes  1.64 Gbits/sec
[  5]   9.00-10.00  sec   188 MBytes  1.58 Gbits/sec
[  5]  10.00-10.02  sec  2.47 MBytes  1.37 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.02  sec  1.69 GBytes  1.45 Gbits/sec                  receiver
-----------------------------------------------------------

test-128 ~ # iperf3 -c 10.0.12.127
Connecting to host 10.0.12.127, port 5201
[  5] local 10.0.12.128 port 61265 connected to 10.0.12.127 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   129 MBytes  1.08 Gbits/sec    0   1.61 MBytes
[  5]   1.00-2.00   sec   192 MBytes  1.61 Gbits/sec    0   1.61 MBytes
[  5]   2.00-3.00   sec   173 MBytes  1.45 Gbits/sec    0   1.61 MBytes
[  5]   3.00-4.00   sec   155 MBytes  1.30 Gbits/sec    0   1.61 MBytes
[  5]   4.00-5.00   sec   171 MBytes  1.43 Gbits/sec    0   1.61 MBytes
[  5]   5.00-6.00   sec   174 MBytes  1.46 Gbits/sec    0   1.61 MBytes
[  5]   6.00-7.00   sec   173 MBytes  1.45 Gbits/sec    0   1.61 MBytes
[  5]   7.00-8.00   sec   183 MBytes  1.54 Gbits/sec    0   1.61 MBytes
[  5]   8.00-9.00   sec   195 MBytes  1.63 Gbits/sec    0   1.61 MBytes
[  5]   9.00-10.00  sec   188 MBytes  1.58 Gbits/sec    0   1.61 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.69 GBytes  1.45 Gbits/sec    0             sender
[  5]   0.00-10.02  sec  1.69 GBytes  1.45 Gbits/sec                  receiver

iperf Done.

Code:

root@lab17:~ # uname -a
FreeBSD lab17 12.1-RELEASE FreeBSD 12.1-RELEASE r354233 GENERIC  amd64
root@lab17:~ #

root@lab18:~ # uname -a
FreeBSD lab18 12.1-RELEASE FreeBSD 12.1-RELEASE r354233 GENERIC  amd64
root@lab18:~ #

Bridged Interfaces: server to server

Code:

Accepted connection from 10.0.11.18, port 61273
[  5] local 10.0.11.17 port 5201 connected to 10.0.11.18 port 17033
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   310 MBytes  2.60 Gbits/sec
[  5]   1.00-2.00   sec   551 MBytes  4.62 Gbits/sec
[  5]   2.00-3.00   sec   589 MBytes  4.94 Gbits/sec
[  5]   3.00-4.00   sec   591 MBytes  4.96 Gbits/sec
[  5]   4.00-5.00   sec   592 MBytes  4.97 Gbits/sec
[  5]   5.00-6.00   sec   594 MBytes  4.99 Gbits/sec
[  5]   6.00-7.00   sec   594 MBytes  4.98 Gbits/sec
[  5]   7.00-8.00   sec   593 MBytes  4.98 Gbits/sec
[  5]   8.00-9.00   sec   593 MBytes  4.97 Gbits/sec
[  5]   9.00-10.00  sec   593 MBytes  4.98 Gbits/sec
[  5]  10.00-10.00  sec   337 KBytes  4.93 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  5.47 GBytes  4.70 Gbits/sec                  receiver
-----------------------------------------------------------

root@lab18:~ # iperf3 -c 10.0.11.17
Connecting to host 10.0.11.17, port 5201
[  5] local 10.0.11.18 port 17033 connected to 10.0.11.17 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   312 MBytes  2.61 Gbits/sec   28    390 KBytes
[  5]   1.00-2.00   sec   551 MBytes  4.63 Gbits/sec   78    254 KBytes
[  5]   2.00-3.00   sec   589 MBytes  4.94 Gbits/sec   27    191 KBytes
[  5]   3.00-4.00   sec   591 MBytes  4.96 Gbits/sec   32    335 KBytes
[  5]   4.00-5.00   sec   592 MBytes  4.96 Gbits/sec   28    304 KBytes
[  5]   5.00-6.00   sec   595 MBytes  4.99 Gbits/sec   29    275 KBytes
[  5]   6.00-7.00   sec   593 MBytes  4.98 Gbits/sec   28    238 KBytes
[  5]   7.00-8.00   sec   593 MBytes  4.98 Gbits/sec   33    366 KBytes
[  5]   8.00-9.00   sec   593 MBytes  4.97 Gbits/sec   26    348 KBytes
[  5]   9.00-10.00  sec   593 MBytes  4.98 Gbits/sec   31    315 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  5.47 GBytes  4.70 Gbits/sec  340             sender
[  5]   0.00-10.00  sec  5.47 GBytes  4.70 Gbits/sec                  receiver

iperf Done.

Direct Interfaces: server to server

Code:

Accepted connection from 10.0.12.18, port 24603
[  5] local 10.0.12.17 port 5201 connected to 10.0.12.18 port 30690
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   997 MBytes  8.36 Gbits/sec
[  5]   1.00-2.00   sec  1.82 GBytes  15.6 Gbits/sec
[  5]   2.00-3.00   sec  1.83 GBytes  15.7 Gbits/sec
[  5]   3.00-4.00   sec  1.83 GBytes  15.7 Gbits/sec
[  5]   4.00-5.00   sec  1.83 GBytes  15.7 Gbits/sec
[  5]   5.00-6.00   sec  1.83 GBytes  15.7 Gbits/sec
[  5]   6.00-7.00   sec  1.83 GBytes  15.8 Gbits/sec
[  5]   7.00-8.00   sec  1.84 GBytes  15.8 Gbits/sec
[  5]   8.00-9.00   sec  1.83 GBytes  15.7 Gbits/sec
[  5]   9.00-10.00  sec  1.83 GBytes  15.7 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  17.4 GBytes  15.0 Gbits/sec                  receiver
-----------------------------------------------------------

root@lab18:~ # iperf3 -c 10.0.12.17
Connecting to host 10.0.12.17, port 5201
[  5] local 10.0.12.18 port 30690 connected to 10.0.12.17 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   998 MBytes  8.37 Gbits/sec   16    386 KBytes
[  5]   1.00-2.00   sec  1.82 GBytes  15.6 Gbits/sec   38    375 KBytes
[  5]   2.00-3.00   sec  1.83 GBytes  15.7 Gbits/sec   16    379 KBytes
[  5]   3.00-4.00   sec  1.83 GBytes  15.7 Gbits/sec   11   1.41 KBytes
[  5]   4.00-5.00   sec  1.83 GBytes  15.7 Gbits/sec   16    205 KBytes
[  5]   5.00-6.00   sec  1.83 GBytes  15.7 Gbits/sec    8    212 KBytes
[  5]   6.00-7.00   sec  1.83 GBytes  15.8 Gbits/sec    9    239 KBytes
[  5]   7.00-8.00   sec  1.84 GBytes  15.8 Gbits/sec    8    267 KBytes
[  5]   8.00-9.00   sec  1.83 GBytes  15.7 Gbits/sec    7    289 KBytes
[  5]   9.00-10.00  sec  1.83 GBytes  15.7 Gbits/sec    6    314 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  17.4 GBytes  15.0 Gbits/sec  135             sender
[  5]   0.00-10.00  sec  17.4 GBytes  15.0 Gbits/sec                  receiver

iperf Done.

jail to jail on same server

Code:

Accepted connection from 10.0.11.117, port 31176
[  5] local 10.0.11.127 port 5201 connected to 10.0.11.117 port 38861
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   203 MBytes  1.71 Gbits/sec
[  5]   1.00-2.00   sec   342 MBytes  2.87 Gbits/sec
[  5]   2.00-3.00   sec   343 MBytes  2.88 Gbits/sec
[  5]   3.00-4.00   sec   315 MBytes  2.64 Gbits/sec
[  5]   4.00-5.00   sec   224 MBytes  1.88 Gbits/sec
[  5]   5.00-6.00   sec   233 MBytes  1.96 Gbits/sec
[  5]   6.00-7.00   sec   241 MBytes  2.02 Gbits/sec
[  5]   7.00-8.00   sec   243 MBytes  2.04 Gbits/sec
[  5]   8.00-9.00   sec   243 MBytes  2.04 Gbits/sec
[  5]   9.00-10.00  sec   243 MBytes  2.04 Gbits/sec
[  5]  10.00-10.00  sec   851 KBytes  3.64 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  2.57 GBytes  2.21 Gbits/sec                  receiver
-----------------------------------------------------------

test-117 ~ # iperf3 -c 10.0.11.127
Connecting to host 10.0.11.127, port 5201
[  5] local 10.0.11.117 port 38861 connected to 10.0.11.127 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   204 MBytes  1.71 Gbits/sec    0    495 KBytes
[  5]   1.00-2.00   sec   342 MBytes  2.87 Gbits/sec    0    995 KBytes
[  5]   2.00-3.00   sec   343 MBytes  2.88 Gbits/sec    0   1.38 MBytes
[  5]   3.00-4.00   sec   315 MBytes  2.64 Gbits/sec    0   1.67 MBytes
[  5]   4.00-5.00   sec   224 MBytes  1.88 Gbits/sec    0   1.77 MBytes
[  5]   5.00-6.00   sec   233 MBytes  1.96 Gbits/sec    0   1.77 MBytes
[  5]   6.00-7.00   sec   241 MBytes  2.02 Gbits/sec    0   1.77 MBytes
[  5]   7.00-8.00   sec   243 MBytes  2.04 Gbits/sec    0   1.77 MBytes
[  5]   8.00-9.00   sec   243 MBytes  2.04 Gbits/sec    0   1.77 MBytes
[  5]   9.00-10.00  sec   243 MBytes  2.04 Gbits/sec    0   1.77 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  2.57 GBytes  2.21 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  2.57 GBytes  2.21 Gbits/sec                  receiver

iperf Done.

jail to jail on different servers

Code:

Accepted connection from 10.0.12.128, port 25861
[  5] local 10.0.12.127 port 5201 connected to 10.0.12.128 port 18803
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   221 MBytes  1.85 Gbits/sec
[  5]   1.00-2.00   sec   340 MBytes  2.85 Gbits/sec
[  5]   2.00-3.00   sec   344 MBytes  2.89 Gbits/sec
[  5]   3.00-4.00   sec   392 MBytes  3.29 Gbits/sec
[  5]   4.00-5.00   sec   341 MBytes  2.86 Gbits/sec
[  5]   5.00-6.00   sec   262 MBytes  2.20 Gbits/sec
[  5]   6.00-7.00   sec   390 MBytes  3.27 Gbits/sec
[  5]   7.00-8.00   sec   392 MBytes  3.29 Gbits/sec
[  5]   8.00-9.00   sec   397 MBytes  3.33 Gbits/sec
[  5]   9.00-10.00  sec   398 MBytes  3.34 Gbits/sec
[  5]  10.00-10.00  sec  1014 KBytes  4.11 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  3.40 GBytes  2.92 Gbits/sec                  receiver
-----------------------------------------------------------

test-128 ~ # iperf3 -c 10.0.12.127
Connecting to host 10.0.12.127, port 5201
[  5] local 10.0.12.128 port 18803 connected to 10.0.12.127 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   221 MBytes  1.86 Gbits/sec    0    495 KBytes
[  5]   1.00-2.00   sec   340 MBytes  2.85 Gbits/sec    0   1007 KBytes
[  5]   2.00-3.00   sec   344 MBytes  2.89 Gbits/sec    0   1.47 MBytes
[  5]   3.00-4.00   sec   392 MBytes  3.29 Gbits/sec    0   1.77 MBytes
[  5]   4.00-5.00   sec   341 MBytes  2.86 Gbits/sec    0   1.77 MBytes
[  5]   5.00-6.00   sec   262 MBytes  2.20 Gbits/sec    0   1.77 MBytes
[  5]   6.00-7.00   sec   390 MBytes  3.27 Gbits/sec    0   1.77 MBytes
[  5]   7.00-8.00   sec   392 MBytes  3.29 Gbits/sec    0   1.77 MBytes
[  5]   8.00-9.00   sec   397 MBytes  3.33 Gbits/sec    0   1.77 MBytes
[  5]   9.00-10.00  sec   399 MBytes  3.34 Gbits/sec    0   1.77 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  3.40 GBytes  2.92 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  3.40 GBytes  2.92 Gbits/sec                  receiver

iperf Done.

Backport of if_bridge patch for better performance to 12.1

zirias@

Attachments

sahlawat

pos

sahlawat