Solved VNET jails very slow inbound networking after upgrading host to 14.0-RELEASE

subnetspider

subnetspider

Hello everyone,

after upgrading my jail host from 13.2-RELEASE-p5 to 14.0-RELEASE, I noticed that the services hosted on it became very slow and unresponsive.

This is most noticeable with my Plex and Nextcloud servers, which are either painfully slow or completely unusable.

After some troubleshooting, I noticed that when running speedtest with iperf3, the jails on that host only manage to send about 1-2 Mbps while
receiving full gigabit at ~930 Mbps (dns02 is a jail on another 13.2-RELEASE jail host):

Code: 
admin@dns02:~ % iperf3 -c fd5e:d1c2:c9de:60::6
...
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.97  sec  1.38 MBytes  1.05 Mbits/sec                 sender
-----------------------------------------------------------

Code: 
admin@dns02:~ % iperf3 -c fd5e:d1c2:c9de:60::6 -R
...
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.08 GBytes   927 Mbits/sec    0            receiver
-----------------------------------------------------------

This happens regardless of IPv4 or IPv6, or using 13.2 vs 14.0 jails, but interestingly, when running iperf3 between
two jails on the same host, they don't suffer any slowdowns (same bridge):

Code: 
admin@ns1:~ % iperf3 -c fd5e:d1c2:c9de:60::6
...
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.01  sec  6.66 GBytes  5.72 Gbits/sec                  sender
-----------------------------------------------------------

I tried booting into the old 13.2-RELEASE-p5 boot environment, and the slowness disappeared.

For anyone wondering, here is the network part of my jail host's /etc/rc.conf:

Code: 
ifconfig_igb0="up"
ifconfig_igb1="up"
cloned_interfaces="lagg0 vlan12 vlan60 vlan70 vlan80"
ifconfig_lagg0="laggproto lacp laggport igb0 laggport igb1"
ifconfig_vlan12="vlan 12 vlandev lagg0"
ifconfig_vlan60="vlan 60 vlandev lagg0"
ifconfig_vlan70="vlan 70 vlandev lagg0"
ifconfig_vlan80="vlan 80 vlandev lagg0"
ifconfig_vlan80_ipv6="inet6 fd5e:d1c2:c9de:80::12/64"

ifconfig_em0="inet 10.1.10.7/24"
defaultrouter="10.1.10.1"
ifconfig_em0_ipv6="inet6 accept_rtadv"
ifconfig_em0_alias0="inet6 fd00:10::12/64"
ifconfig_em0_alias1="inet6 fd5e:d1c2:c9de:10::12/64"

VLAN12, VLAN60, and VLAN70 are used for the VNET jails, VLAN80 is for mounting NFS shares on my TrueNAS, em0 is for management of the jail host.

The jail host uses a Intel i340-T4 network card and has a LACP link aggregation to a Brocade ICX6430-24 switch, and a Intel i219-V for the management interface.

Does someone have an Idea what could cause this problem? I really want to upgrade to FreeBSD 14.0, but this makes all my jails unusable. :(
 
junkyalleycat said:
might be related to this other thread? Thread vnet-jails-very-slow-inbound-networking-after-upgrading-host-to-14-0-release.91467
Click to expand...
Hello junkyalleycat,

I assume you mean this thread? slow network speeds with jail (and vnet) on FreeBSD 14.0-RELEASE
uchman said:
Well, I disabled LRO on the physical interface (em0) and now I have good speeds to my jails again. Someone who can explain this?
Click to expand...

I just ran sudo ifconfig igb0 -lro && sudo ifconfig igb1 -lro and everything seems to work again like normal.

To make it permanent, I updated my /etc/rc.conf:

Code: 
ifconfig_igb0="up -lro"
ifconfig_igb1="up -lro"

(See: Disabling Hardware Offloading on FreeBSD)

Now, the jails are fast again and iperf3 confirms it:

Code: 
admin@ns2:~ % iperf3 -c fd5e:d1c2:c9de:60::6
Connecting to host fd5e:d1c2:c9de:60::6, port 5201
[  5] local fd5e:d1c2:c9de:60::5 port 38009 connected to fd5e:d1c2:c9de:60::6 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.01   sec   112 MBytes   930 Mbits/sec    0    492 KBytes
[  5]   1.01-2.01   sec   111 MBytes   926 Mbits/sec    0    492 KBytes
[  5]   2.01-3.00   sec   109 MBytes   926 Mbits/sec    0    492 KBytes
[  5]   3.00-4.00   sec   111 MBytes   928 Mbits/sec    0    738 KBytes
[  5]   4.00-5.01   sec   111 MBytes   927 Mbits/sec    0    738 KBytes
[  5]   5.01-6.02   sec   112 MBytes   926 Mbits/sec    0    738 KBytes
[  5]   6.02-7.02   sec   110 MBytes   926 Mbits/sec    0    738 KBytes
[  5]   7.02-8.02   sec   110 MBytes   926 Mbits/sec    0    738 KBytes
[  5]   8.02-9.02   sec   110 MBytes   926 Mbits/sec    0    738 KBytes
[  5]   9.02-10.02  sec   110 MBytes   926 Mbits/sec    0    738 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.02  sec  1.08 GBytes   927 Mbits/sec    0             sender
[  5]   0.00-10.02  sec  1.08 GBytes   926 Mbits/sec                  receiver

I guess it was really the same thing as in the other thread, no idea why LRO is suddenly a problem with 14.0 when it wasnt until 13.2...
 
You must log in or register to reply here.
Back
Top