rc.conf jumbo frame MTU settings not applied for IPv6?

B

Blue|Fusion

Hello,

On my three FreeBSD 15.1 servers I am using Jumbo Frames (9000 bytes) via 40 Gigabit links with LACP, bridges, and VLANs for jails and VMs and all three have the same issue. Pasted below is my rc.conf for one of my servers and the others are identical on construction just with different IPs. The problem I am experiencing is the jumbo frame MTU 9000 is set and visible in ifconfig upon boot-up. However when I try to ping another jumbo frame enabled host it fails with a message too big error when using IPv6 but works with IPv4. To get IPv6 to work with the set MTU, I need to set the MTU of the host's VLAN interface on the bridge to something different and then reset it to 9000. Also seen below is this exact process.

Code: 
#
# Network Configuration
#
hostname="ecorp.mydomain.tld"

# 1GigE interfaces (disabled)
ifconfig_em0="down"
ifconfig_em1="down"

# 40GigE interfaces (MTU 9004 for 4-byte tagged VLAN overhead)
ifconfig_mlxen0="-lro -tso -txcsum -rxcsum mtu 9004 up"
ifconfig_mlxen1="-lro -tso -txcsum -rxcsum mtu 9004 up"

# Build LACP lagg and vlan interface for host
cloned_interfaces="lagg0 bridge0"
ifconfig_lagg0="laggproto lacp lagghash l3,l4 laggport mlxen0 laggport mlxen1 mtu 9004 up"
vlans_bridge0="vlan140"
create_args_vlan140="vlan 140"

# Enable VLAN filtering on bridge and add lagg0 to it
ifconfig_bridge0="vlanfilter addm lagg0 tagged 100,101,110,140,150 mtu 9004 up"

# Configure host interface on VLAN 140
ifconfig_vlan140="inet 172.23.40.10/24 mtu 9000 up"
ifconfig_vlan140_ipv6="inet6 fd33:58bc:59a0:2340::10/64 accept_rtadv"
defaultrouter="172.23.40.1"

The basic setup of the virtual network stack is as follows:
Code: 
     mlxen0/mlxen1 (40Gb NICs)
           |
         lagg0 (LACP)
           |
        bridge0
       /       \
  vlan140   [jail epairs]

Below is the output after a fresh reboot with the above rc.conf settings. You'll see that IPv4 jumbo frames work, IPv6 do not until resetting the MTU of the vlan140 interface.
Code: 
rich@ecorp:~ % uptime
 1:51PM  up 2 mins, 1 user, load averages: 0.77, 0.33, 0.13
rich@ecorp:~ %
rich@ecorp:~ % ifconfig vlan140
vlan140: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
        options=0
        ether 58:9c:fc:10:26:cd
        inet 172.23.40.10 netmask 0xffffff00 broadcast 172.23.40.255
        inet6 fe80::5a9c:fcff:fe10:26cd%vlan140 prefixlen 64 scopeid 0xa
        inet6 fd33:58bc:59a0:2340::10 prefixlen 64
        inet6 2600:2b00:ad50:c340:5a9c:fcff:fe10:26cd prefixlen 64 autoconf pltime 600 vltime 1800
        groups: vlan
        vlan: 140 vlanproto: 802.1q vlanpcp: 0 parent interface: bridge0
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
rich@ecorp:~ %
rich@ecorp:~ % route -n get -6 fd33:58bc:59a0:2340::11
   route to: fd33:58bc:59a0:2340::11
destination: fd33:58bc:59a0:2340::
       mask: ffff:ffff:ffff:ffff::
        fib: 0
  interface: vlan140
      flags: <UP,DONE,PINNED>
 recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0      9000         1         0
rich@ecorp:~ %
rich@ecorp:~ % doas ping -D -s 8972 172.23.40.11
PING 172.23.40.11 (172.23.40.11): 8972 data bytes
8980 bytes from 172.23.40.11: icmp_seq=0 ttl=64 time=0.296 ms
8980 bytes from 172.23.40.11: icmp_seq=1 ttl=64 time=0.168 ms
^C
--- 172.23.40.11 ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.168/0.232/0.296/0.064 ms
rich@ecorp:~ %
rich@ecorp:~ % ping -D -s 8952 fd33:58bc:59a0:2340::11
ping: you need -b to increase socket buffer size
PING(9000=40+8+8952 bytes) fd33:58bc:59a0:2340::10 --> fd33:58bc:59a0:2340::11
ping: sendmsg: Message too long
ping: wrote fd33:58bc:59a0:2340::11 8960 chars, ret=-1
^C
--- fd33:58bc:59a0:2340::11 ping statistics ---
1 packets transmitted, 0 packets received, 100.0% packet loss
rich@ecorp:~ %
rich@ecorp:~ % doas ifconfig vlan140 mtu 8999
rich@ecorp:~ % doas ifconfig vlan140 mtu 9000
rich@ecorp:~ %
rich@ecorp:~ % ping -D -s 8952 fd33:58bc:59a0:2340::11
ping: you need -b to increase socket buffer size
PING(9000=40+8+8952 bytes) fd33:58bc:59a0:2340::10 --> fd33:58bc:59a0:2340::11
8960 bytes from fd33:58bc:59a0:2340::11, icmp_seq=0 hlim=64 time=0.211 ms
8960 bytes from fd33:58bc:59a0:2340::11, icmp_seq=1 hlim=64 time=0.179 ms
^C
--- fd33:58bc:59a0:2340::11 ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.179/0.195/0.211/0.016 ms

FreeBSD uname:
Code: 
rich@ecorp:~ % uname -a 
FreeBSD ecorp.mydomain.tld 15.1-RELEASE-p1 FreeBSD 15.1-RELEASE-p1 #0 -dirty: Tue Jun 30 03:16:39 UTC 2026     root@amd64-builder.daemonology.net:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64
rich@ecorp:~ %

I do not know if this was also the experience on 15.0-RELEASE or earlier as I only discovered this issue last night doing some network performance tests and network interrurpts freezing my system for minutes at a time during iperf3 runs.

Thank you for your help and input and let me know if there's anything else I can provide or try.

EDIT to add:
In case it matters, the system is an old Dell PowerEdge R710 II with dual Xeon X5690 CPUs and a ConnectX-3 Pro 40GbE NIC using the mlx4en driver. The two other machines are Supermicro X8DTN+ with single Xeon X5680 CPUs and the same NIC and driver in use.
 
This looks closely related to a FreeBSD IPv6 ND MTU bug I recently reported:

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=296656

FreeBSD keeps IPv6-specific linkmtu and maxmtu values separately from the interface MTU. Consequently, ifconfig can show MTU 9000 while IPv6 still uses a stale lower value. Your workaround of changing the MTU to 8999 and then back to 9000 likely triggers the missing MTU notification and refreshes the IPv6 value.

Bug 296656 specifically concerns a stale linkmtu. Your VLAN/bridge/LAGG case may instead involve a stale maxmtu caused by interface initialization ordering, so I would not yet call it the same bug.

Could you run this immediately after a fresh reboot, before applying the workaround?

ndp -i vlan140

Then run it again after changing the MTU to 8999 and back to 9000. The before-and-after linkmtu and maxmtu values should show whether this is the same defect or a closely related one.
 
Hello,

I tried as you suggested and it does show an MTU of 1500 with that command until after the bandaid "fix".

Code: 
rich@ecorp:~ % uptime
 8:18PM  up 4 mins, 1 user, load averages: 0.49, 0.47, 0.24
rich@ecorp:~ % ndp -i vlan140
linkmtu=0, maxmtu=1500, curhlim=64, basereachable=30s0ms, reachable=39s, retrans=1s0ms
Flags: nud accept_rtadv auto_linklocal 
rich@ecorp:~ % doas ifconfig vlan140 mtu 8999
rich@ecorp:~ % doas ifconfig vlan140 mtu 9000
rich@ecorp:~ % ndp -i vlan140
linkmtu=0, maxmtu=9000, curhlim=64, basereachable=30s0ms, reachable=39s, retrans=1s0ms
Flags: nud accept_rtadv auto_linklocal
 
Thanks Oleg.

I looked over the bug you reported and it's technicalities are mostly beyond me to tell whether it's the same experience but sounds like it probably is.

Regarding bug 255749, this isn't quite the same. In the last comment, it is mentioned that setting MTU on the vlan interface is necessary to set the desired MTU vs inheriting it from it's parent interface. However, in my rc.conf the MTUis explicitly set on the vlan interface upon IP address configuration, just the IPv6 part doesn't make that switch, opposite of the "works 100% correctly" claim the same comment makes. This makes it sound more like your reported IPv6 bug.
 
My report concerns the nonzero linkmtu bug. The other bug report and the problem you encountered concern a stale cached maxmtu value.
 
You must log in or register to reply here.
Back
Top