Backport of if_bridge patch for better performance to 12.1

Zirias

Aspiring Daemon

Reaction score: 356
Messages: 826

Hi all,

here's a recent commit to head (-CURRENT): https://svnweb.freebsd.org/base?view=revision&revision=360345

I thought it might be worthwile to test that with 12.1, so I backported the patch. So far, I only did a quick smoke test with two VNET jails using epair devices connected by a bridge. Using dd and nc, I managed to reach a data rate of 240 MB/s on my old desktop PC, so that doesn't look like an improvement, but at least it's a smoke test that the bridge still works ;) Maybe the epair devices or my aging CPU are limiting performance, I don't know…

So, if you feel advanturous enough to trust my backporting work, please try the patch attached here, maybe it improves some setups :) Of course, no guarantees I didn't break anything.
 

Attachments

sahlawat

New Member

Reaction score: 1
Messages: 1

A 4x network performance patch, I had to try this; really wanted this to work, but ...

1. The patch applies clean and compiles with 12.1-p5 r361429M
2. No system crashes - I have 4 bridges with 5 VLANs and a 2 interface lagg port connecting 45 jails with VNET interfaces.

Performed 2 simple iperf3 tests:
1. Jail (hub) to Jail (.123)
2. External pfsense (fw) to Jail (.123)
Jail 192.168.0.123 was running the iperf3 server in both cases

Performance Before Patch:

hub ~ # iperf3 -c 192.168.0.123
Connecting to host 192.168.0.123, port 5201
[ 5] local 192.168.0.50 port 43309 connected to 192.168.0.123 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 374 MBytes 3.13 Gbits/sec 0 2.00 MBytes
[ 5] 1.00-2.00 sec 384 MBytes 3.22 Gbits/sec 0 2.00 MBytes
[ 5] 2.00-3.00 sec 382 MBytes 3.20 Gbits/sec 0 2.00 MBytes
[ 5] 3.00-4.00 sec 382 MBytes 3.20 Gbits/sec 0 2.00 MBytes
[ 5] 4.00-5.00 sec 385 MBytes 3.23 Gbits/sec 0 2.00 MBytes
[ 5] 5.00-6.00 sec 386 MBytes 3.24 Gbits/sec 0 2.00 MBytes
[ 5] 6.00-7.00 sec 384 MBytes 3.22 Gbits/sec 0 2.00 MBytes
[ 5] 7.00-8.00 sec 389 MBytes 3.26 Gbits/sec 0 2.00 MBytes
[ 5] 8.00-9.00 sec 387 MBytes 3.25 Gbits/sec 0 2.00 MBytes
[ 5] 9.00-10.00 sec 387 MBytes 3.25 Gbits/sec 0 2.00 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 3.75 GBytes 3.22 Gbits/sec 0 sender
[ 5] 0.00-10.00 sec 3.75 GBytes 3.22 Gbits/sec receiver

iperf Done.
hub ~ # iperf3 -c 192.168.0.123 -u
Connecting to host 192.168.0.123, port 5201
[ 5] local 192.168.0.50 port 33443 connected to 192.168.0.123 port 5201
[ ID] Interval Transfer Bitrate Total Datagrams
[ 5] 0.00-1.00 sec 128 KBytes 1.05 Mbits/sec 90
[ 5] 1.00-2.00 sec 128 KBytes 1.05 Mbits/sec 90
[ 5] 2.00-3.00 sec 128 KBytes 1.05 Mbits/sec 90
[ 5] 3.00-4.00 sec 128 KBytes 1.05 Mbits/sec 90
[ 5] 4.00-5.00 sec 127 KBytes 1.04 Mbits/sec 89
[ 5] 5.00-6.00 sec 128 KBytes 1.05 Mbits/sec 90
[ 5] 6.00-7.00 sec 128 KBytes 1.05 Mbits/sec 90
[ 5] 7.00-8.00 sec 128 KBytes 1.05 Mbits/sec 90
[ 5] 8.00-9.00 sec 127 KBytes 1.04 Mbits/sec 89
[ 5] 9.00-10.00 sec 128 KBytes 1.05 Mbits/sec 90
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-10.00 sec 1.25 MBytes 1.05 Mbits/sec 0.000 ms 0/898 (0%) sender
[ 5] 0.00-10.02 sec 1.25 MBytes 1.05 Mbits/sec 0.013 ms 0/898 (0%) receiver

iperf Done.


[2.4.5-RELEASE][admin@fw]/root: iperf3 -c 192.168.0.123
Connecting to host 192.168.0.123, port 5201
[ 5] local 192.168.0.5 port 41841 connected to 192.168.0.123 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 112 MBytes 939 Mbits/sec 0 65.1 KBytes
[ 5] 1.00-2.00 sec 108 MBytes 903 Mbits/sec 0 65.1 KBytes
[ 5] 2.00-3.00 sec 112 MBytes 938 Mbits/sec 0 65.1 KBytes
[ 5] 3.00-4.00 sec 111 MBytes 934 Mbits/sec 0 65.1 KBytes
[ 5] 4.00-5.00 sec 112 MBytes 936 Mbits/sec 0 65.1 KBytes
[ 5] 5.00-6.00 sec 110 MBytes 919 Mbits/sec 0 65.1 KBytes
[ 5] 6.00-7.00 sec 112 MBytes 939 Mbits/sec 0 65.1 KBytes
[ 5] 7.00-8.00 sec 112 MBytes 938 Mbits/sec 0 65.1 KBytes
[ 5] 8.00-9.00 sec 112 MBytes 939 Mbits/sec 0 65.1 KBytes
[ 5] 9.00-10.00 sec 112 MBytes 938 Mbits/sec 0 65.1 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.09 GBytes 932 Mbits/sec 0 sender
[ 5] 0.00-10.01 sec 1.09 GBytes 932 Mbits/sec receiver

iperf Done.
[2.4.5-RELEASE][admin@fw]/root: iperf3 -c 192.168.0.123 -u
Connecting to host 192.168.0.123, port 5201
[ 5] local 192.168.0.5 port 54771 connected to 192.168.0.123 port 5201
[ ID] Interval Transfer Bitrate Total Datagrams
[ 5] 0.00-1.00 sec 128 KBytes 1.05 Mbits/sec 90
[ 5] 1.00-2.00 sec 128 KBytes 1.05 Mbits/sec 90
[ 5] 2.00-3.00 sec 128 KBytes 1.05 Mbits/sec 90
[ 5] 3.00-4.00 sec 128 KBytes 1.05 Mbits/sec 90
[ 5] 4.00-5.00 sec 127 KBytes 1.04 Mbits/sec 89
[ 5] 5.00-6.00 sec 128 KBytes 1.05 Mbits/sec 90
[ 5] 6.00-7.00 sec 128 KBytes 1.05 Mbits/sec 90
[ 5] 7.00-8.00 sec 128 KBytes 1.05 Mbits/sec 90
[ 5] 8.00-9.00 sec 127 KBytes 1.04 Mbits/sec 89
[ 5] 9.00-10.00 sec 128 KBytes 1.05 Mbits/sec 90
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-10.00 sec 1.25 MBytes 1.05 Mbits/sec 0.000 ms 0/898 (0%) sender
[ 5] 0.00-10.00 sec 1.25 MBytes 1.05 Mbits/sec 0.010 ms 0/898 (0%) receiver

iperf Done.




Performance After Patch:

hub ~ # iperf3 -c 192.168.0.123
Connecting to host 192.168.0.123, port 5201
[ 5] local 192.168.0.50 port 44597 connected to 192.168.0.123 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 29.3 MBytes 246 Mbits/sec 0 64.9 KBytes
[ 5] 1.00-2.00 sec 26.1 MBytes 219 Mbits/sec 0 64.9 KBytes
[ 5] 2.00-3.00 sec 32.7 MBytes 274 Mbits/sec 0 64.9 KBytes
[ 5] 3.00-4.00 sec 35.6 MBytes 298 Mbits/sec 0 64.9 KBytes
[ 5] 4.00-5.00 sec 28.3 MBytes 238 Mbits/sec 0 64.9 KBytes
[ 5] 5.00-6.00 sec 35.2 MBytes 296 Mbits/sec 0 64.9 KBytes
[ 5] 6.00-7.00 sec 28.6 MBytes 240 Mbits/sec 0 64.9 KBytes
[ 5] 7.00-8.00 sec 32.4 MBytes 272 Mbits/sec 0 64.9 KBytes
[ 5] 8.00-9.00 sec 25.5 MBytes 214 Mbits/sec 0 64.9 KBytes
[ 5] 9.00-10.00 sec 36.2 MBytes 304 Mbits/sec 0 64.9 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 310 MBytes 260 Mbits/sec 0 sender
[ 5] 0.00-10.00 sec 310 MBytes 260 Mbits/sec receiver

iperf Done.
hub ~ # iperf3 -c 192.168.0.123 -u
Connecting to host 192.168.0.123, port 5201
[ 5] local 192.168.0.50 port 26229 connected to 192.168.0.123 port 5201
[ ID] Interval Transfer Bitrate Total Datagrams
[ 5] 0.00-1.00 sec 128 KBytes 1.05 Mbits/sec 90
[ 5] 1.00-2.00 sec 128 KBytes 1.05 Mbits/sec 90
[ 5] 2.00-3.00 sec 128 KBytes 1.05 Mbits/sec 90
[ 5] 3.00-4.00 sec 128 KBytes 1.05 Mbits/sec 90
[ 5] 4.00-5.00 sec 127 KBytes 1.04 Mbits/sec 89
[ 5] 5.00-6.00 sec 128 KBytes 1.05 Mbits/sec 90
[ 5] 6.00-7.00 sec 128 KBytes 1.05 Mbits/sec 90
[ 5] 7.00-8.00 sec 128 KBytes 1.05 Mbits/sec 90
[ 5] 8.00-9.00 sec 127 KBytes 1.04 Mbits/sec 89
[ 5] 9.00-10.00 sec 128 KBytes 1.05 Mbits/sec 90
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-10.00 sec 1.25 MBytes 1.05 Mbits/sec 0.000 ms 0/898 (0%) sender
[ 5] 0.00-10.02 sec 1.25 MBytes 1.05 Mbits/sec 0.010 ms 0/898 (0%) receiver

iperf Done.


[2.4.5-RELEASE][admin@fw]/root: iperf3 -c 192.168.0.123
Connecting to host 192.168.0.123, port 5201
[ 5] local 192.168.0.5 port 16638 connected to 192.168.0.123 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 112 MBytes 937 Mbits/sec 0 65.1 KBytes
[ 5] 1.00-2.00 sec 112 MBytes 939 Mbits/sec 0 65.1 KBytes
[ 5] 2.00-3.00 sec 112 MBytes 939 Mbits/sec 0 65.1 KBytes
[ 5] 3.00-4.00 sec 112 MBytes 939 Mbits/sec 0 65.1 KBytes
[ 5] 4.00-5.00 sec 109 MBytes 917 Mbits/sec 0 65.1 KBytes
[ 5] 5.00-6.00 sec 112 MBytes 938 Mbits/sec 0 65.1 KBytes
[ 5] 6.00-7.00 sec 112 MBytes 936 Mbits/sec 0 65.1 KBytes
[ 5] 7.00-8.00 sec 112 MBytes 937 Mbits/sec 0 65.1 KBytes
[ 5] 8.00-9.00 sec 112 MBytes 938 Mbits/sec 0 65.1 KBytes
[ 5] 9.00-10.00 sec 112 MBytes 939 Mbits/sec 0 65.1 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.09 GBytes 936 Mbits/sec 0 sender
[ 5] 0.00-10.01 sec 1.09 GBytes 935 Mbits/sec receiver

iperf Done.
[2.4.5-RELEASE][admin@fw]/root: iperf3 -c 192.168.0.123 -u
Connecting to host 192.168.0.123, port 5201
[ 5] local 192.168.0.5 port 25837 connected to 192.168.0.123 port 5201
[ ID] Interval Transfer Bitrate Total Datagrams
[ 5] 0.00-1.00 sec 128 KBytes 1.05 Mbits/sec 90
[ 5] 1.00-2.00 sec 128 KBytes 1.05 Mbits/sec 90
[ 5] 2.00-3.00 sec 128 KBytes 1.05 Mbits/sec 90
[ 5] 3.00-4.00 sec 128 KBytes 1.05 Mbits/sec 90
[ 5] 4.00-5.00 sec 127 KBytes 1.04 Mbits/sec 89
[ 5] 5.00-6.00 sec 128 KBytes 1.05 Mbits/sec 90
[ 5] 6.00-7.00 sec 128 KBytes 1.05 Mbits/sec 90
[ 5] 7.00-8.00 sec 128 KBytes 1.05 Mbits/sec 90
[ 5] 8.00-9.00 sec 127 KBytes 1.04 Mbits/sec 89
[ 5] 9.00-10.00 sec 128 KBytes 1.05 Mbits/sec 90
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-10.00 sec 1.25 MBytes 1.05 Mbits/sec 0.000 ms 0/898 (0%) sender
[ 5] 0.00-10.00 sec 1.25 MBytes 1.05 Mbits/sec 0.012 ms 0/898 (0%) receiver

iperf Done.


Given the inter-jail throughput drop I restored the 12.1 bridge files.

Next - will standup another server with 13-current and see what performance I get on that branch.
 

pos

Member

Reaction score: 11
Messages: 64

I have read of Krystofs great bridge work, and I welcome it *very* much.

The existing bridge code is not super. But works. I do not understand why you see so bad numbers... I have a virtual linux on top of FreeBSD 12.0. The virtual CentOS 7 server is connected over a FreeBSD bridge on Bhyve host. That virtual server gets 7.2 Gbit/s (over 7 routers hops to a target on the internet from my location). Speed test site is "bredbandskollen.se" with CLI client from http://www.bredbandskollen.se/om/mer-om-bbk/bredbandskollen-cli/ (also on git hub). Yes, I have a 10 Gbit internet connection. If not using a FreeBSD bridge I get 8.5 to 9 Gbit. Using a bridge in linux KVM in the same way I do not se any performance drop at all over the bridge. So if the systems are correctly configured the FreeBSD bridges do give half decent performance (but eat resources if you push much traffic).

So my personal opinion is to wait and upgrade to FreeBSD 13 if it is included there. But of course, if you have time available. Why not :)
 
Top