VLAN + Jumbo Frames

S

sko

Hello,

I'm in the process of setting up a new gateway. The local network will be connected via 2x Intel I350, configured as lagg / LACP.
Another FreeBSD machine, also with 2xI350 configured as lagg/LACP is used for testing.

This is the rc.conf on both machines for the lagg and VLAN configuration:
Code: 
cloned_interfaces="lagg0"
ifconfig_igb0="-lro -tso -vlanhwtag mtu 9000 up"
ifconfig_igb1="-lro -tso -vlanhwtag mtu 9000 up"
ifconfig_lagg0="laggproto lacp laggport igb0 laggport igb1 192.168.10.1/24"
cloned_interfaces+="vlan5"
ifconfig_vlan5="inet 10.50.50.1/24 vlan 5 vlandev lagg0"
(of course the second machine gets another IP address...)

Using the lagg0 interfaces directly (eg. setting an IP address on lagg0 and running iperf with these IPs) works as expected. Iperf3 shows a constant 991-992Mbit/s throughput.

But if I run the benchmarks over the vlan interface I'm stuck with this:
Code: 
# iperf3 -c 10.50.50.100
Connecting to host 10.50.50.100, port 5201
[  4] local 10.50.50.1 port 25257 connected to 10.50.50.100 port 5201
[ ID] Interval  Transfer  Bandwidth  Retr  Cwnd
[  4]  0.00-1.06  sec  43.0 KBytes  331 Kbits/sec  2  76.4 MBytes  
[  4]  1.06-2.06  sec  0.00 Bytes  0.00 bits/sec  1  76.4 MBytes  
[  4]  2.06-3.06  sec  0.00 Bytes  0.00 bits/sec  1  76.4 MBytes  
[  4]  3.06-4.06  sec  0.00 Bytes  0.00 bits/sec  0  76.4 MBytes  
[  4]  4.06-5.06  sec  0.00 Bytes  0.00 bits/sec  1  76.4 MBytes  
[  4]  5.06-6.06  sec  0.00 Bytes  0.00 bits/sec  0  76.4 MBytes  
[  4]  6.06-7.05  sec  0.00 Bytes  0.00 bits/sec  0  76.4 MBytes  
[  4]  7.05-8.04  sec  0.00 Bytes  0.00 bits/sec  0  76.4 MBytes  
[  4]  8.04-9.03  sec  0.00 Bytes  0.00 bits/sec  1  76.4 MBytes  
[  4]  9.03-10.01  sec  0.00 Bytes  0.00 bits/sec  0  76.4 MBytes  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval  Transfer  Bandwidth  Retr
[  4]  0.00-10.01  sec  43.0 KBytes  35.2 Kbits/sec  6  sender
[  4]  0.00-10.01  sec  0.00 Bytes  0.00 bits/sec  receiver
iperf Done.

Neither iperf nor netperf can seem to get any traffic through the VLAN devices. However I can log in on that IP via SSH without any problem (connecting host MTU at 1500 or 9000). Both hosts can also connect via ssh, pinging each other etc.

Trying to transfer a file via scp or even simple nc fails, and on the receiving side tcpdump shows a lot of packages with 'length 0'.

All switches in the network are cisco SG300 series with jumbo frames enabled. Both systems are currently plugged into the same switch with the corresponding ports configured as LAGs /w LACP active. I also temporarily connected the systems directly with 2 patch cables to rule out any glitches on the switch, but still get the same results.

If I revert the igb interfaces back to MTU 1500, transfers and metrics on the vlan device are working and iperf showing ~940MBit/s throughput on lagg0 and vlan5.

I couldn't find any limitations on lagg or vlan with jumbo frames in the manpages or documentation. Searching the forums or the interwebs only revealed some limitations/bugs regarding the MTU with some specific chipsets, but as the lagg0 performs well with a MTU of 9000 I doubt there's any problem on the hardware or driver side...

Are there any knobs for VLAN I have to adjust for jumbo frames?

I'm not terribly dependent on these last 50Mbits, but it would be nice to know what the problem is (and if it can be resolved).
 
There's an error in your rc.conf:
Code: 
cloned_interfaces="lagg0"
{...}
cloned_interfaces="vlan5"
The second cloned_interfaces overrules the first. You need to realize rc.conf is just a plain shell script that gets sourced by the rc(8) scripts.

Code: 
cloned_interfaces="lagg0 vlan5"
Or:
Code: 
cloned_interfaces="lagg0"
cloned_interfaces+="vlan5"
 
No idea, but I'd suggest approaching the problem from two directions:
  1. Configure lagg with only igb0 @ 9k and see if that makes a difference to the vlan driver. (my guess is no; eg: it is a packet size issue not an aggregation issue)
  2. As 1500 works and 9k fails, try to determine at which packet size vlan fails. (watching some tcpdump might be interesting, maybe with ping -s to look for sizes/fragmentation)
 
Thanks. I already tried a MTU of 5000 and 4000 with no result. It's quite time-consuming to test, because I have to reboot both machines in order to change the MTU. ifconfig igbN mtu xxxx always fails with "ifconfig: ioctl SIOCSIFMTU (set mtu): Invalid argument" on both machines - even with the default value of 1500 or (much) lower....

Maximum packetsize ping gets through with MTU9000 on the VLAN is 8968 bytes. At the lagg0 interfaces the maximum size is 8972 bytes.
Setting net.inet.tcp.mssdflt=8968 doesn't seem to affect iperf, as it still fails to get any data through. The -M switch of iperf fails just like trying to set the mtu via ifconfig with an "Invalid argument".
 
I just stumbled across the very same error again and finally resolved the issue (i think).

TL;DR:
Framesize for packets from access ports was too big - MTU discovery seems to exclude the 4k for VLAN tags. Switch adding/stripping VLAN tags from/to access ports also seems to mess up the MTU discovery...



The lagg interface inherits its MTU from the underlying interfaces (igb0+1), which is 9000:
Code: 
# ifconfig igb0
igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
  options=6400ab<RXCSUM,TXCSUM,VLAN_MTU,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
...
# ifconfig igb1
igb1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
....
# ifconfig lagg0
lagg0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
...
        laggproto lacp lagghash l2,l3,l4
        laggport: igb0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
        laggport: igb1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>

Creating a VLAN interface on top of it inherits the same MTU of 9000:
Code: 
# ifconfig vlan666 create vlandev lagg0 vlan 666
# ifconfig vlan666
vlan666: flags=8842<BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
        ether a0:36:9f:10:34:80
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: active
        vlan: 666 vlanpcp: 0 parent interface: lagg0
        groups: vlan

Now, everything that doesn't max out the MTU just works (SSHing into the machine, receiving stuff from external networks which send frames <1500 etc).
I recognized a weird pattern on 2 hosts where only incoming transfers on VLAN interfaces would fail from 2 other FreeBSD hosts, connected to VLAN via access ports, but succeed from e.g. an illumos host or the old debian installation on my laptop, both with tagged VLAN on trunk ports.

I reduced the MTU to 8000 on all directly involved interfaces (vlan interfaces) and recognized, the frames from the affected freeBSD hosts were 4k bigger (=VLAN tag added by switch) than those incoming from the illumos host. So when ramping up to a full MTU of 9000, the frames from those would be too big.
Same thing for anything coming from hosts on access ports.

Reducing the MTU on the VLAN interface by 4 made the frames equal in size to those coming from illumos or debian.
So it seems when negotiating the MTU, the FreeBSD host doesn't take the VLAN tag into account and/or the switch messes up the discovery by silently adding/removing the VLAN tags. This results in incoming packets being too big (+4k by switch), outgoing packets to access ports get their VLAN tag stripped by the switch, leading to the expected framesize on the receiving side.

I ended up setting the MTU on all virtual interfaces to 8600, which leaves enough headroom for any weird switch or socket behavior (or misunderstanding on my side....) and doesn't impose any measurable penalty on throughput (LTO and TSO had the biggest impact...)
 
You must log in or register to reply here.
Back
Top