Solved [Solved] Xen NIC cannot set MTU > 1500 bytes, BW suffers

I have a Xen server using CentOS as dom0, and two guests in domU - FreeBSD and Linux. Under dom0, I have set all interfaces on the bridge to an MTU of 20400 (seems to be the highest that works). The external interface has an MTU of 65520.

I can reliably get packets unfragmented up to the 20400 bridge MTU between other computers on the network and either the CentOS dom0 or CentOS domU. I've tested this with both ping -s 20000 x.y.z.w and iperf -c x.y.z.w -m. iperf correctly reports a discovered MTU of 20400.

However, I can only ping the FreeBSD domU with packets up to ~1470 bytes and iperf reports the MTU is 1500 bytes, the default for the Xen interface (and ethernet, of course). This is despite having set the MTU on xn1 manually to 20400. Running tcpdump inside of FreeBSD on the interface shows that the packets arrive unfragmented, but the response cannot make it back without breaking down below 1500 bytes.

In this scenario, I am running into a bandwidth ceiling around 1.5Gb/s, whereas both dom0 and the Linux domU can reliably reach 11Gb/s. Note that between FreeBSD and the Hypervisor, I can get >10Gb/s, so it seems to be an issue of latency combined with the MTU. A solution with either allowing a larger bandwidth delay product or fixing the MTU could probably work...

The current status of the xn1 adapter in FreeBSD:
Code:
xn1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 20400
	options=503<RXCSUM,TXCSUM,TSO4,LRO>
	ether 00:16:3e:be:d0:42
	inet 172.16.3.24 netmask 0xffffff00 broadcast 172.16.3.255 
	inet6 fe80::216:3eff:febe:d042%xn1 prefixlen 64 scopeid 0x3 
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
	media: Ethernet manual
	status: active
 
Re: Xen NIC cannot set MTU > 1500 bytes, BW suffers

I have found the issue. It stems from a disconnect in my management of Linux interfaces, where I have not specifically associated MTUs with routes. It seems that on FreeBSD, MTUs are regularly and statically assigned to routes.

In particular: I had my existing interface setup in /etc/rc.conf that specified only the adapter's IP and netmask. The interface was brought up each boot with only these tweaked, the MTU left at the default of 1500 bytes. This caused it to create a route for that interface's subnet with the default MTU of 1500. Later modification of the interface with ifconfig xn1 mtu 20400 updated only the interface, not the route associated with it. For testing purposes, I was able to use route change x.y.z.0 -mtu 20400, and then follow this up with the addition of the MTU specification in the interface's on-boot configuration.
 
Back
Top