This has been a really difficult case for me to solve, but what I am seeing sounds ... too weird to believe. I really don't know where the fault lies, but any advice would be amazing.
What I see is that when a Linux router is on the path between a FreeBSD server or client, a collapse in bandwidth occurs. I am also seeing situations where Linux as a client performs worse when connected to a FreeBSD server.
But more over, the moment a linux router is involved, the throughput tanks *hard*.
The reason I noticed this is that I have a FreeBSD server in a VPS. Downloading from the FreeBSD machine to another machine in the same DC, I see 480MB/s. From FreeBSD via my internet to my UXG I am able to see 30MB/s - this is maxing out my line rate, and is what I expect.
But from the FreeBSD machine to *anything* behind the UXG (a linux router) I see 5MB/s, and it fluctuates wildly.
On top of this, others who consumer the services of that FreeBSD server reported the same throughput tanking - ubiquiti, tplink, openwrt - all seemed to be affected.
I've spent a lot of time trying out various tunings - the worst was enable RACK which caused these values to sink into 200kb/s kind of numbers. htcp instead of cubic improves things somewhat but not much. Buffer tunigs etc don't seem to have much effect.
To isolate this I have tried to break down a reproducer to a few components as possible. This will be a long post, but I'd rather be thorough than not.
The reproduction environment is as follows:
I am using virtual machines hosted on a FreeBSD 14.3 server which has an Intel Xeon Silver 4310 (24) @ 2.100GHz and 128GB of ram.
The network adapter is an Intel ix 10GBe adapter, which has LRO/TSO disabled. Ifconfig output below.
On top of this are vlan interfaces, that then are connected to bridges. As an example:
I am using bhyve + vm-bhyve to manage the machines. Each machine has 2 cpu and 2GB of ram.
All machines are completely stock - no sysctl tuning or anything like that.
In the examples I will show iperf3 output - keep in mind I am also testing with curl + webservers, and the results are near identical. For brevity as this will already be very long, I will only show the iperf3 results as they align to other tests performed.
As a baseline, I establish localhost throughputs:
:
These values set the "upper limit". I would assume that even though a discrepancy exists here, 10GBit is achievable to both instances. The software is *NOT* the bottleneck.
Within the same bridge interface I see the following throughput:
We can already see a stark difference here: from a FreeBSD server to a Linux client, there is a significant reduction in throughput even when a router is not involved.
Now, to establish the bandwidth between the router and the clients.
The router is a ubiquiti UXG PRO 10GBe connected with 10GBe DAC's to a Ubiquiti Aggregation switch with 10GB SFP's.
Since the path is from the server, via the switch, to the router and then the same return path, there should be no other variables.
These numbers now are *crossing* that router.
So we can see that the "upper" limit is 1.4GBit, but every other combination yields far lower results. The worst being FreeBSD to FreeBSD when crossing the UXG.
There seems to be something fundamental at play here - even in the direct L2 tests, FreeBSD as a server to an OpenSUSE client has nearly 50% of the throughput of any other combination. FreeBSD to FreeBSD also seems to be experiencing some kind of issue.
Again, all the values above are *stock* installs with no tunings.
The most significant evidence I have found is that when wiresharking the connections, from linux to linux, I see a collection of 10 to 16 packets, followed by a single ACK, rinse repeat. Where as in FreeBSD it will received 10 packets and send 3 to 4 acks, and then the connection devolves into single data frame - single ACK.
So there is some interaction here that I do not understand.
What should I look at next? What should I test? Is this a bug?
Any advice would be really appreciated.
What I see is that when a Linux router is on the path between a FreeBSD server or client, a collapse in bandwidth occurs. I am also seeing situations where Linux as a client performs worse when connected to a FreeBSD server.
But more over, the moment a linux router is involved, the throughput tanks *hard*.
The reason I noticed this is that I have a FreeBSD server in a VPS. Downloading from the FreeBSD machine to another machine in the same DC, I see 480MB/s. From FreeBSD via my internet to my UXG I am able to see 30MB/s - this is maxing out my line rate, and is what I expect.
But from the FreeBSD machine to *anything* behind the UXG (a linux router) I see 5MB/s, and it fluctuates wildly.
On top of this, others who consumer the services of that FreeBSD server reported the same throughput tanking - ubiquiti, tplink, openwrt - all seemed to be affected.
I've spent a lot of time trying out various tunings - the worst was enable RACK which caused these values to sink into 200kb/s kind of numbers. htcp instead of cubic improves things somewhat but not much. Buffer tunigs etc don't seem to have much effect.
To isolate this I have tried to break down a reproducer to a few components as possible. This will be a long post, but I'd rather be thorough than not.
The reproduction environment is as follows:
I am using virtual machines hosted on a FreeBSD 14.3 server which has an Intel Xeon Silver 4310 (24) @ 2.100GHz and 128GB of ram.
The network adapter is an Intel ix 10GBe adapter, which has LRO/TSO disabled. Ifconfig output below.
Code:
ix0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
options=4e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
ether 64:9d:99:b1:a4:78
media: Ethernet autoselect (10Gbase-SR <full-duplex,rxpause,txpause>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>On top of this are vlan interfaces, that then are connected to bridges. As an example:
Code:
ix0.20: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
options=4200001<RXCSUM,RXCSUM_IPV6,MEXTPG>
ether 64:9d:99:b1:a4:78
groups: vlan
vlan: 20 vlanproto: 802.1q vlanpcp: 0 parent interface: ix0
media: Ethernet autoselect (10Gbase-SR <full-duplex,rxpause,txpause>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
bridge20: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
options=0
ether 58:9c:fc:00:31:42
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: tap15 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 33 priority 128 path cost 2000000
...
member: ix0.20 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 15 priority 128 path cost 2000
groups: bridge vm-switch viid-8286e@
nd6 options=9<PERFORMNUD,IFDISABLED>I am using bhyve + vm-bhyve to manage the machines. Each machine has 2 cpu and 2GB of ram.
All machines are completely stock - no sysctl tuning or anything like that.
In the examples I will show iperf3 output - keep in mind I am also testing with curl + webservers, and the results are near identical. For brevity as this will already be very long, I will only show the iperf3 results as they align to other tests performed.
As a baseline, I establish localhost throughputs:
Code:
SERVER to CLIENT : THROUGHPUT
OpenSUSE 15.6 to SELF : 53.6 Gbits/sec
FreeBSD 14.3 to SELF : 33.9 Gbits/secThese values set the "upper limit". I would assume that even though a discrepancy exists here, 10GBit is achievable to both instances. The software is *NOT* the bottleneck.
Within the same bridge interface I see the following throughput:
Code:
SERVER to CLIENT : THROUGHPUT
OpenSUSE 15.6 to OpenSUSE 15.6: 3.71 Gbits/sec
OpenSUSE 15.6 to FreeBSD 14.3: 2.80 Gbits/sec
FreeBSD 14.3 to FreeBSD 14.3: 1.73 Gbits/sec
FreeBSD 14.3 to OpenSUSE 15.6: 964 Mbits/secWe can already see a stark difference here: from a FreeBSD server to a Linux client, there is a significant reduction in throughput even when a router is not involved.
Now, to establish the bandwidth between the router and the clients.
The router is a ubiquiti UXG PRO 10GBe connected with 10GBe DAC's to a Ubiquiti Aggregation switch with 10GB SFP's.
Since the path is from the server, via the switch, to the router and then the same return path, there should be no other variables.
These numbers now are *crossing* that router.
Code:
SERVER to CLIENT : THROUGHPUT
OpenSUSE 15.6 to OpenSUSE 15.6: 1.40 Gbits/sec
OpenSUSE 15.6 to FreeBSD 14.3: 706 Mbits/sec
FreeBSD 14.3 to FreeBSD 14.3: 304 Mbits/sec
FreeBSD 14.3 to OpenSUSE 15.6: 534 Mbits/secSo we can see that the "upper" limit is 1.4GBit, but every other combination yields far lower results. The worst being FreeBSD to FreeBSD when crossing the UXG.
There seems to be something fundamental at play here - even in the direct L2 tests, FreeBSD as a server to an OpenSUSE client has nearly 50% of the throughput of any other combination. FreeBSD to FreeBSD also seems to be experiencing some kind of issue.
Again, all the values above are *stock* installs with no tunings.
The most significant evidence I have found is that when wiresharking the connections, from linux to linux, I see a collection of 10 to 16 packets, followed by a single ACK, rinse repeat. Where as in FreeBSD it will received 10 packets and send 3 to 4 acks, and then the connection devolves into single data frame - single ACK.
So there is some interaction here that I do not understand.
What should I look at next? What should I test? Is this a bug?
Any advice would be really appreciated.
Last edited by a moderator:
