Hello,
On my three FreeBSD 15.1 servers I am using Jumbo Frames (9000 bytes) via 40 Gigabit links with LACP, bridges, and VLANs for jails and VMs and all three have the same issue. Pasted below is my rc.conf for one of my servers and the others are identical on construction just with different IPs. The problem I am experiencing is the jumbo frame MTU 9000 is set and visible in ifconfig upon boot-up. However when I try to ping another jumbo frame enabled host it fails with a message too big error when using IPv6 but works with IPv4. To get IPv6 to work with the set MTU, I need to set the MTU of the host's VLAN interface on the bridge to something different and then reset it to 9000. Also seen below is this exact process.
The basic setup of the virtual network stack is as follows:
Below is the output after a fresh reboot with the above rc.conf settings. You'll see that IPv4 jumbo frames work, IPv6 do not until resetting the MTU of the vlan140 interface.
FreeBSD uname:
I do not know if this was also the experience on 15.0-RELEASE or earlier as I only discovered this issue last night doing some network performance tests and network interrurpts freezing my system for minutes at a time during iperf3 runs.
Thank you for your help and input and let me know if there's anything else I can provide or try.
EDIT to add:
In case it matters, the system is an old Dell PowerEdge R710 II with dual Xeon X5690 CPUs and a ConnectX-3 Pro 40GbE NIC using the mlx4en driver. The two other machines are Supermicro X8DTN+ with single Xeon X5680 CPUs and the same NIC and driver in use.
On my three FreeBSD 15.1 servers I am using Jumbo Frames (9000 bytes) via 40 Gigabit links with LACP, bridges, and VLANs for jails and VMs and all three have the same issue. Pasted below is my rc.conf for one of my servers and the others are identical on construction just with different IPs. The problem I am experiencing is the jumbo frame MTU 9000 is set and visible in ifconfig upon boot-up. However when I try to ping another jumbo frame enabled host it fails with a message too big error when using IPv6 but works with IPv4. To get IPv6 to work with the set MTU, I need to set the MTU of the host's VLAN interface on the bridge to something different and then reset it to 9000. Also seen below is this exact process.
Code:
#
# Network Configuration
#
hostname="ecorp.mydomain.tld"
# 1GigE interfaces (disabled)
ifconfig_em0="down"
ifconfig_em1="down"
# 40GigE interfaces (MTU 9004 for 4-byte tagged VLAN overhead)
ifconfig_mlxen0="-lro -tso -txcsum -rxcsum mtu 9004 up"
ifconfig_mlxen1="-lro -tso -txcsum -rxcsum mtu 9004 up"
# Build LACP lagg and vlan interface for host
cloned_interfaces="lagg0 bridge0"
ifconfig_lagg0="laggproto lacp lagghash l3,l4 laggport mlxen0 laggport mlxen1 mtu 9004 up"
vlans_bridge0="vlan140"
create_args_vlan140="vlan 140"
# Enable VLAN filtering on bridge and add lagg0 to it
ifconfig_bridge0="vlanfilter addm lagg0 tagged 100,101,110,140,150 mtu 9004 up"
# Configure host interface on VLAN 140
ifconfig_vlan140="inet 172.23.40.10/24 mtu 9000 up"
ifconfig_vlan140_ipv6="inet6 fd33:58bc:59a0:2340::10/64 accept_rtadv"
defaultrouter="172.23.40.1"The basic setup of the virtual network stack is as follows:
Code:
mlxen0/mlxen1 (40Gb NICs)
|
lagg0 (LACP)
|
bridge0
/ \
vlan140 [jail epairs]Below is the output after a fresh reboot with the above rc.conf settings. You'll see that IPv4 jumbo frames work, IPv6 do not until resetting the MTU of the vlan140 interface.
Code:
rich@ecorp:~ % uptime
1:51PM up 2 mins, 1 user, load averages: 0.77, 0.33, 0.13
rich@ecorp:~ %
rich@ecorp:~ % ifconfig vlan140
vlan140: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
options=0
ether 58:9c:fc:10:26:cd
inet 172.23.40.10 netmask 0xffffff00 broadcast 172.23.40.255
inet6 fe80::5a9c:fcff:fe10:26cd%vlan140 prefixlen 64 scopeid 0xa
inet6 fd33:58bc:59a0:2340::10 prefixlen 64
inet6 2600:2b00:ad50:c340:5a9c:fcff:fe10:26cd prefixlen 64 autoconf pltime 600 vltime 1800
groups: vlan
vlan: 140 vlanproto: 802.1q vlanpcp: 0 parent interface: bridge0
nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
rich@ecorp:~ %
rich@ecorp:~ % route -n get -6 fd33:58bc:59a0:2340::11
route to: fd33:58bc:59a0:2340::11
destination: fd33:58bc:59a0:2340::
mask: ffff:ffff:ffff:ffff::
fib: 0
interface: vlan140
flags: <UP,DONE,PINNED>
recvpipe sendpipe ssthresh rtt,msec mtu weight expire
0 0 0 0 9000 1 0
rich@ecorp:~ %
rich@ecorp:~ % doas ping -D -s 8972 172.23.40.11
PING 172.23.40.11 (172.23.40.11): 8972 data bytes
8980 bytes from 172.23.40.11: icmp_seq=0 ttl=64 time=0.296 ms
8980 bytes from 172.23.40.11: icmp_seq=1 ttl=64 time=0.168 ms
^C
--- 172.23.40.11 ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.168/0.232/0.296/0.064 ms
rich@ecorp:~ %
rich@ecorp:~ % ping -D -s 8952 fd33:58bc:59a0:2340::11
ping: you need -b to increase socket buffer size
PING(9000=40+8+8952 bytes) fd33:58bc:59a0:2340::10 --> fd33:58bc:59a0:2340::11
ping: sendmsg: Message too long
ping: wrote fd33:58bc:59a0:2340::11 8960 chars, ret=-1
^C
--- fd33:58bc:59a0:2340::11 ping statistics ---
1 packets transmitted, 0 packets received, 100.0% packet loss
rich@ecorp:~ %
rich@ecorp:~ % doas ifconfig vlan140 mtu 8999
rich@ecorp:~ % doas ifconfig vlan140 mtu 9000
rich@ecorp:~ %
rich@ecorp:~ % ping -D -s 8952 fd33:58bc:59a0:2340::11
ping: you need -b to increase socket buffer size
PING(9000=40+8+8952 bytes) fd33:58bc:59a0:2340::10 --> fd33:58bc:59a0:2340::11
8960 bytes from fd33:58bc:59a0:2340::11, icmp_seq=0 hlim=64 time=0.211 ms
8960 bytes from fd33:58bc:59a0:2340::11, icmp_seq=1 hlim=64 time=0.179 ms
^C
--- fd33:58bc:59a0:2340::11 ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.179/0.195/0.211/0.016 msFreeBSD uname:
Code:
rich@ecorp:~ % uname -a
FreeBSD ecorp.mydomain.tld 15.1-RELEASE-p1 FreeBSD 15.1-RELEASE-p1 #0 -dirty: Tue Jun 30 03:16:39 UTC 2026 root@amd64-builder.daemonology.net:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64
rich@ecorp:~ %I do not know if this was also the experience on 15.0-RELEASE or earlier as I only discovered this issue last night doing some network performance tests and network interrurpts freezing my system for minutes at a time during iperf3 runs.
Thank you for your help and input and let me know if there's anything else I can provide or try.
EDIT to add:
In case it matters, the system is an old Dell PowerEdge R710 II with dual Xeon X5690 CPUs and a ConnectX-3 Pro 40GbE NIC using the mlx4en driver. The two other machines are Supermicro X8DTN+ with single Xeon X5680 CPUs and the same NIC and driver in use.