I have a Xen server using CentOS as dom0, and two guests in domU - FreeBSD and Linux. Under dom0, I have set all interfaces on the bridge to an MTU of 20400 (seems to be the highest that works). The external interface has an MTU of 65520.
I can reliably get packets unfragmented up to the 20400 bridge MTU between other computers on the network and either the CentOS dom0 or CentOS domU. I've tested this with both
However, I can only ping the FreeBSD domU with packets up to ~1470 bytes and iperf reports the MTU is 1500 bytes, the default for the Xen interface (and ethernet, of course). This is despite having set the MTU on xn1 manually to 20400. Running tcpdump inside of FreeBSD on the interface shows that the packets arrive unfragmented, but the response cannot make it back without breaking down below 1500 bytes.
In this scenario, I am running into a bandwidth ceiling around 1.5Gb/s, whereas both dom0 and the Linux domU can reliably reach 11Gb/s. Note that between FreeBSD and the Hypervisor, I can get >10Gb/s, so it seems to be an issue of latency combined with the MTU. A solution with either allowing a larger bandwidth delay product or fixing the MTU could probably work...
The current status of the xn1 adapter in FreeBSD:
I can reliably get packets unfragmented up to the 20400 bridge MTU between other computers on the network and either the CentOS dom0 or CentOS domU. I've tested this with both
ping -s 20000 x.y.z.w
and iperf -c x.y.z.w -m
. iperf correctly reports a discovered MTU of 20400.However, I can only ping the FreeBSD domU with packets up to ~1470 bytes and iperf reports the MTU is 1500 bytes, the default for the Xen interface (and ethernet, of course). This is despite having set the MTU on xn1 manually to 20400. Running tcpdump inside of FreeBSD on the interface shows that the packets arrive unfragmented, but the response cannot make it back without breaking down below 1500 bytes.
In this scenario, I am running into a bandwidth ceiling around 1.5Gb/s, whereas both dom0 and the Linux domU can reliably reach 11Gb/s. Note that between FreeBSD and the Hypervisor, I can get >10Gb/s, so it seems to be an issue of latency combined with the MTU. A solution with either allowing a larger bandwidth delay product or fixing the MTU could probably work...
The current status of the xn1 adapter in FreeBSD:
Code:
xn1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 20400
options=503<RXCSUM,TXCSUM,TSO4,LRO>
ether 00:16:3e:be:d0:42
inet 172.16.3.24 netmask 0xffffff00 broadcast 172.16.3.255
inet6 fe80::216:3eff:febe:d042%xn1 prefixlen 64 scopeid 0x3
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet manual
status: active