FreeBSD as guest on XCP-ng/Xenserver, networking is very slow on internal private network between VM's

C

chrizz

Hi!

Im running two test VM's with FreeBSD 12.0 as guests on a XCP-ng host. I have a private network on this host and connected one VIF for each FreeBSD 12.0 server to this. Running iperf between these two VM's I get only 40-50Mbit. I have multiple servers on this host, and in other scenarios I can reach 10gbit. This should be way faster, but I'm really stuck now, and don't know how to solve it.

As a note, I have a 10.2-RELEASE VM also, and this does NOT seem to have any networking issues at all. 12.0, 11.1, 11.2 does (these are the versions I've tried so far).

Any ideas on how to proceed? Thanks

edit: I never tried 11.3, I mixed up the numbers
 
What network card are you using on the FreeBSD side? Are you using vtnet(4)? In any case, try switching off TSO and/or LSO.
 
On xen side I've turned basically everything off for the VIF's of these two VM's: other-config (MRW): ethtool-sg: off; ethtool-tso: off; ethtool-ufo: off; ethtool-gso: off; ethtool-rx: off; ethtool-tx: off

In FreeBSD the VIF turns up as xn0. Is that using vtnet(4)? Running as is I get 40-50mbit/s. Turning off TSO with ifconfig xn0 -tso makes no difference, nor turning off lro with ifconfig xn0 -lro.
 
chrizz said:
In FreeBSD the VIF turns up as xn0. Is that using vtnet(4)?
Click to expand...
As far as I know that's part of xen(4) and should provide something similar but specific to XEN. The manual page for xbn(4) has some additional info:
Code: 
     For this reason, it is recommended that if xnb is bridged to a physical
     interface, then transmit checksum offloading should be disabled on the
     netfront.  The Xennet protocol does not have any mechanism for the
     netback to request the netfront to do this; the operator must do it
     manually.
But if I understood correctly you already turned TSO/LSO off on both the host side and the guest.
 
I'm a bit confused, what is the netfront and netback? If I understand correctly, netfront is on the VM side, in FreeBSD? So from the xnb(4) info I should run ifconfig xn0 -txcsum only and be all good regarding to this. If this does not work, the issue must be something else then?
 
I've made some new tests that show me that something happened in 11.0-RELEASE, and even more in 12.0-RELEASE

Results of my tests with 12.0-RELEASE; 11.0-RELEASE and 10.4-RELEASE (all clean installs). All using same type of VIF and the same private network on the same XCP-ng 7.6.0 host:

iperf -s on first server listed below, and iperf -c <ip> -r from the second to test speed back and forth:

12.0 <-> 12.0: 50Mbit as client and server
12.0 <-> 11.0: 800Mbit/s (11.0 as client), and 140Mbit/s (11.0 as server)
12.0 <-> 10.4: 2.76Gbit (10.4 as client), and 1.25Gbit (10.4 as server).
11.0 <-> 11.0: 219Mbit as client, 99Mbit as server
10.4 <-> 10.4: 11.2Gbit as client, 10.9Gbit as server

As I side note, not sure if related, but I've noticed that I can't run iperf with -r flag on 10.4-RELEASE, I then get this error message:

iperf -c 172.31.16.122 -r
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
write failed: Broken pipe
------------------------------------------------------------
Client connecting to 172.31.16.122, TCP port 5001
TCP window size: 32.5 KByte (default)
------------------------------------------------------------
[ 5] local 172.31.16.121 port 19231 connected with 172.31.16.122 port 5001
[ ID] Interval Transfer Bandwidth
[ 5] 0.0- 0.0 sec 0.00 Bytes 0.00 bits/sec

I can run iperf -s fine, and iperf -c <ip> from the other 10.4 VM though:

iperf -c 172.31.16.122
------------------------------------------------------------
Client connecting to 172.31.16.122, TCP port 5001
TCP window size: 32.5 KByte (default)
------------------------------------------------------------
[ 3] local 172.31.16.121 port 22055 connected with 172.31.16.122 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 12.9 GBytes 11.1 Gbits/sec
 
I'd report the issue on the mailing lists. I'm not entirely sure if freebsd-xen only deals with running Xen on FreeBSD or FreeBSD on Xen but I'm sure they'll know more about the intricate details of xen(4). It's been around since 8.1 but there may be some major differences between the 10 and 11 branches. If it's not the correct mailing list I'm also sure they can point you to the correct one.

Your use case is a little beyond the general user support we can give here. You can give it a couple of days, maybe one of our members knows a bit more but I'm afraid most members are not that familiar with the internals.
 
You must log in or register to reply here.
Back
Top