Asymmetric Network Performance FreeBSD VM on XCP-NG

D

Daryl Bert

I have a Dell R720 with XCP-NG 8.1 installed on it -- After installing a FreeBSD 11.3 virtual machine I noticed that I was getting asymmetric network transfer speeds between VM machines. I have several Debian and Ubuntu virtual machines that have symmetric network performance to one another. However, when a network test is run between the FreeBSD virtual machine and the Debian/Ubuntu virtual machines the transfer shows asymmetric results. i.e. one direction is much slower than the other. Furthermore, FreeBSD VM (post version 10.4) to FreeBSD VM (post version 10.4) demonstrates symmetric performance but slow in both directions. After identifying the asymmetry I decided to install older and newer versions of FreeBSD and my research shows that FreeBSD 10.4 actually performs at a higher level than my Linux based VMs and is symmetric (or closer to symmetric - both are acceptable performance for me).

FreeBSD 11.0 is the version where the asymmetry appears and it also exists in 11.3 and 12.1. i.e. it is still the case for the most recent production release.

dmesg | grep xn0 (FreeBSD VM)

Output:
xn0: <Virtual Network Interface> at device/vif/0 on xenbusb_front0
xn0: Ethernet address: xx:xx:xx:xx:xx:xx
xn0: backend features:random: unblocking device.
xn0: 2 link states coalesced
xn0: link state changed to UP

Network performance - measured via iperf3

FreeBSD 11.3 (similar results FreeBSD 11.0 and 12.1) (172.16.1.56) <--> Ubuntu 18.04 (172.16.1.4) (Asymmetric performance)

iperf3 -c 172.16.1.4

Output: (poor performance compared to next test going the other direction - notice the large number of retries in the results)
Connecting to host 172.16.1.4, port 5201
[ 5] local 172.16.1.56 port 41608 connected to 172.16.1.4 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 293 MBytes 2.46 Gbits/sec 469 211 KBytes
[ 5] 1.00-2.00 sec 288 MBytes 2.41 Gbits/sec 352 216 KBytes
[ 5] 2.00-3.00 sec 284 MBytes 2.38 Gbits/sec 665 156 KBytes
[ 5] 3.00-4.00 sec 276 MBytes 2.31 Gbits/sec 344 90.7 KBytes
[ 5] 4.00-5.00 sec 308 MBytes 2.59 Gbits/sec 724 154 KBytes
[ 5] 5.00-6.00 sec 304 MBytes 2.55 Gbits/sec 813 200 KBytes
[ 5] 6.00-7.00 sec 311 MBytes 2.61 Gbits/sec 676 109 KBytes
[ 5] 7.00-8.00 sec 312 MBytes 2.61 Gbits/sec 490 149 KBytes
[ 5] 8.00-9.00 sec 301 MBytes 2.52 Gbits/sec 670 175 KBytes
[ 5] 9.00-10.00 sec 299 MBytes 2.51 Gbits/sec 598 190 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 2.90 GBytes 2.50 Gbits/sec 5801 sender
[ 5] 0.00-10.00 sec 2.90 GBytes 2.49 Gbits/sec receiver

iperf Done.


iperf3 -c 172.16.1.4 -R

Output (higher performance - no retries):

Connecting to host 172.16.1.4, port 5201
Reverse mode, remote host 172.16.1.4 is sending
[ 5] local 172.16.1.56 port 50635 connected to 172.16.1.4 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 1.08 GBytes 9.23 Gbits/sec
[ 5] 1.00-2.00 sec 1.05 GBytes 9.04 Gbits/sec
[ 5] 2.00-3.00 sec 1.09 GBytes 9.37 Gbits/sec
[ 5] 3.00-4.00 sec 1.06 GBytes 9.15 Gbits/sec
[ 5] 4.00-5.00 sec 1.10 GBytes 9.42 Gbits/sec
[ 5] 5.00-6.00 sec 1.13 GBytes 9.75 Gbits/sec
[ 5] 6.00-7.00 sec 1.12 GBytes 9.58 Gbits/sec
[ 5] 7.00-8.00 sec 1.03 GBytes 8.84 Gbits/sec
[ 5] 8.00-9.00 sec 1.12 GBytes 9.60 Gbits/sec
[ 5] 9.00-10.00 sec 1.01 GBytes 8.65 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 10.8 GBytes 9.27 Gbits/sec 0 sender
[ 5] 0.00-10.00 sec 10.8 GBytes 9.26 Gbits/sec receiver

iperf Done.

FreeBSD 10.4 (172.16.1.54) <--> Ubuntu 18.04 (172.16.1.4) (Good performance both directions)

iperf3 -c 172.16.1.4

Output: (no retries)
Connecting to host 172.16.1.4, port 5201
[ 5] local 172.16.1.54 port 52160 connected to 172.16.1.4 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 1.62 GBytes 13.9 Gbits/sec 0 3.00 MBytes
[ 5] 1.00-2.00 sec 1.76 GBytes 15.1 Gbits/sec 0 3.00 MBytes
[ 5] 2.00-3.00 sec 1.61 GBytes 13.9 Gbits/sec 0 3.00 MBytes
[ 5] 3.00-4.00 sec 1.57 GBytes 13.5 Gbits/sec 0 3.00 MBytes
[ 5] 4.00-5.00 sec 1.58 GBytes 13.5 Gbits/sec 0 3.00 MBytes
[ 5] 5.00-6.00 sec 1.67 GBytes 14.4 Gbits/sec 0 3.00 MBytes
[ 5] 6.00-7.00 sec 1.70 GBytes 14.6 Gbits/sec 0 3.00 MBytes
[ 5] 7.00-8.00 sec 1.62 GBytes 13.9 Gbits/sec 0 3.00 MBytes
[ 5] 8.00-9.00 sec 1.67 GBytes 14.3 Gbits/sec 0 3.00 MBytes
[ 5] 9.00-10.00 sec 1.68 GBytes 14.5 Gbits/sec 0 3.00 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 16.5 GBytes 14.2 Gbits/sec 0 sender
[ 5] 0.00-10.00 sec 16.5 GBytes 14.2 Gbits/sec receiver

iperf Done.

iperf3 -c 172.16.1.4 -R

Output: (no retries)

Connecting to host 172.16.1.4, port 5201
Reverse mode, remote host 172.16.1.4 is sending
[ 5] local 172.16.1.54 port 17523 connected to 172.16.1.4 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 1.09 GBytes 9.35 Gbits/sec
[ 5] 1.00-2.00 sec 1.07 GBytes 9.16 Gbits/sec
[ 5] 2.00-3.00 sec 1.01 GBytes 8.64 Gbits/sec
[ 5] 3.00-4.00 sec 1.16 GBytes 9.92 Gbits/sec
[ 5] 4.00-5.00 sec 1.01 GBytes 8.70 Gbits/sec
[ 5] 5.00-6.00 sec 1.04 GBytes 8.93 Gbits/sec
[ 5] 6.00-7.00 sec 1.07 GBytes 9.16 Gbits/sec
[ 5] 7.00-8.00 sec 1.08 GBytes 9.24 Gbits/sec
[ 5] 8.00-9.00 sec 1.01 GBytes 8.70 Gbits/sec
[ 5] 9.00-10.00 sec 1.05 GBytes 9.05 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 10.6 GBytes 9.09 Gbits/sec 0 sender
[ 5] 0.00-10.00 sec 10.6 GBytes 9.09 Gbits/sec receiver

iperf Done.

The implication of the asymmetry is that I am getting poor read performance when I am copying files from the FreeBSD 11.3 virtual machine. Based on research in forums I have attempted the three following things independently of one another.

1) Disabling the following on the FreeBSD network interface (rxcsum, txcsum, tso, and lro) using:

ifconfig xn0 -rxcsum -txcsum -tso -lro

This did have some positive impact with the speed moving from around 2.5 Gbits/second (from the first test above) to 4.2 Gbits/second. An improvement, but still well short of the 9 Gbit/sec in the other direction).

2) Disabling these features on the virtual network interface in XCP-NG with similar results. Output below shows the state after making the change in XCP-NG.

xe vif-param-get uuid=b8363445-bab2-dddd-b1234-815fd305cd29 param-name=other-config

Output:
ethtool-rx: off; ethtool-tx: off; ethtool-sg: off; ethtool-tso: off; ethtool-ufo: off; ethtool-gso: off

3) I have also attempted some sysctl type adjustments with no enhancement in network performance.


I captured some tcpdump data and the significant difference between "good" performance and "poor" performance is the size of the packet. See data below...

tcpdump data (good performance - FreeBSD 10.4 VM to Ubuntu 18.04 VM) -- note packet length of 65160

00:36:25.660150 IP (tos 0x0, ttl 64, id 57165, offset 0, flags [DF], proto TCP (6), length 65212)
172.16.1.54.49266 > 172.1.16.4.5201: Flags [.], cksum 0x590a (incorrect -> 0x99c1), seq 2382579110:2382644270, ack 1, win 1040, options [nop,nop,TS val 1177932 ecr 1428452870], length 65160

tcpdump data (poor performance - FreeBSD 12.1 VM to Ubuntu 18.04 VM) -- note packet length of 1448

00:34:23.568384 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 1500)
172.16.1.56.63487 > 172.16.1.4.5201: Flags [.], cksum 0x602b (incorrect -> 0x6791), seq 200409586:200411034, ack 1, win 4, options [nop,nop,TS val 14000446 ecr 3327744086], length 1448

Any ideas of adjustments that can be made in newer FreeBSD versions (11.3 to 12.1) to achieve better network performance in both directions for my VMs?
 
After many searches and reading I am convinced that the packet size is a significant clue.

My hypothesis is that the VM based network interface (virtio/vtnet in my case I believe) or perhaps netmap may be contributing to the asymmetric transfer speeds. i.e. something in FreeBSD 10.4 allowed the packet sizes to grow from the MTU length of 1500 to 65160 to maximize transfer speeds. FreeBSD 11.0 to current versions are locking the packet size to the MTU length of 1500.

A few questions that I have are...
- what kernel component supports the growth in packet sizes? is it something negotiated between the FreeBSD host and hypervisor?
- are there any diagnostic tests that I can run to either support or refute my admittedly broad hypothesis?

Are there any other hypotheses to dive into?
 
i need to revive this thread
any news on this?
im using freebsd as my daily driver and i just got symetric internet service, all other devices on my network do a speedtest at 200kbs up and down... my freebsd laptop however im getting 15kbs down and 3kbs up, could this be wifi driver problem? im using iwlwifi0

EDIT: Fixed it by installing wifibox, im a little sad that i had to do that just to get wifi going at speed.. and more sad that there is no solution like that at least for my bluetooth :(
 
You must log in or register to reply here.
Back
Top