Solved LAGG/LACP and Jumbo frames configuration via rc.conf?

[Bobi B.]

Greetings, All!

I'm planning to migrate a handful of office servers to LAGG/LACP and I've been wondering if enabling Jumbo frames will improve performance with Samba/NFS/ISCSI. So, after reading ifconfig(8) and lagg(4), I made up the following network configuration on one of the storage servers (/etc/rc.conf.d/network):

cloned_interfaces="lagg0"
ifconfig_igb0="-lro -tso -vlanhwtag mtu 9000 up"
ifconfig_igb1="-lro -tso -vlanhwtag mtu 9000 up"
ifconfig_lagg0="laggproto lacp laggport igb0 laggport igb1 192.168.1.8/24 description 'mgmt'"

LAGG is also configured on the switch and after executing service netif restart (thankfully servers also provide IPMI) LAGG interface is created and is working fine. However its MTU is not 9000, but the default -- 1500; so is the MTU of both Intel gigabit adapters. After going through /etc/rc.d/netif and /etc/network.subr, it seems that cloned interfaces are created before physical interfaces are being configured (netif_start() calls clone_up() before setting-up physical interfaces), hence before MTU is applied, leading to LAGG running with MTU of 1500 bytes. And once LAGG is up and running MTU of physical interfaces can no-longer be changed.

Destroying the LAGG interface, increasing the MTU of physical interfaces and creating the LAGG interface manually works fine. But can someone suggest how to make this work using rc.conf(5)? Thanks in advance!

 

[Phishfry]

Did you see this from last month? Some similar problems.
https://lists.freebsd.org/pipermail/freebsd-current/2018-November/072178.html

Good post here with some info:
https://unix.stackexchange.com/questions/470915/what-cloned-interfaces-does-to-interfaces-in-rc-conf

 

[Bobi B.]

Did you see this from last month? Some similar problems.
https://lists.freebsd.org/pipermail/freebsd-current/2018-November/072178.html

As far as I understood, their problem is that network configuration runs before wireless drivers being loaded, hence wireless interfaces are not yet available. Whereas my problem is only with the way /etc/rc.d/netif is written.

 

[Phishfry]

What about using a separate line for ifconfig 'up' and your options. Maybe add a NETWAIT for interfaces to come up.
Looking at /etc/rc.d/netif and NETWORKING I see what you'r talking about.
Maybe your options need to be on the LAGG interface instead?

 

[VladiBG]

Strange, i tested your configuration in virtual machine and it's working.

ifconfig_hn0="mtu 9000" cloned_interfaces="lagg0" ifconfig_lagg0="lagproto lacp laggport hn0" reboot
and the lagg0 has mtu of 9000

From the lagg(4) man page:

The MTU of the first interface to be added is used as the lagg MTU. All
additional interfaces are required to have exactly the same value.

 

[Phishfry]

Some more reading:
https://lists.freebsd.org/pipermail/freebsd-stable/2016-September/085670.html

https://lists.freebsd.org/pipermail/freebsd-hackers/2015-February/046986.html

 

[SirDice]

This has been working for me for quite some time:

cloned_interfaces="lagg0"
ifconfig_igb0="inet 192.168.10.180 netmask 255.255.255.0 mtu 9000"
ifconfig_igb1="up mtu 9000 -rxcsum -rxcsum6 -txcsum -txcsum6 -lro -tso -vlanhwtso"
ifconfig_igb2="up mtu 9000 -rxcsum -rxcsum6 -txcsum -txcsum6 -lro -tso -vlanhwtso"
ifconfig_lagg0="laggproto lacp laggport igb1 laggport igb2"
defaultrouter="192.168.10.1"

I had to turn off all the hardware checksums, LRO, TSO, etc. or else the network performance would be abysmal.
My igb0 is used as a management interface. The lagg(4) is tied to a trunk on my switch.

root@hosaka:~ # vm switch list
NAME     TYPE      IFACE       ADDRESS  PRIVATE  MTU   VLAN  PORTS
servers  standard  vm-servers  -        no       9000  11    lagg0
public   standard  vm-public   -        no       9000  10    lagg0

 

[Phishfry]

Right and you have 'ifconfig up' for LAGG interfaces before the options.....

I though about perhaps loading the interface early:
/boot/loader.conf
if_igb_load="YES"

Personally I would put the "cloned_interfaces="lagg0"" line after my real interfaces.
I know it don't help for ordering, it just makes me feel better.

I really like the 'managment interface' idea.

 

[SirDice]

Right and you have 'ifconfig up' for LAGG interfaces before the options.....

Yes, the individual member interfaces need to be set to "up", this is the bare bones config:

cloned_interfaces="lagg0"
ifconfig_int0="up"
ifconfig_int1="up"
ifconfig_lagg0="laggproto lacp laggport int0 laggport int1"

I don't think the order within the ifconfig_* strings matters much. I've had the 'up' first, last and somewhere in between and it has always worked. The whole string is fed to ifconfig(8) almost verbatim, so if the order doesn't matter for ifconfig(8) it won't matter for rc.conf either.

 

[Bobi B.]

You know what? It is working when you reboot the machine. I just did that, because we added another one in the rack. But not working when you do service netif restart. Funny. Anyway. Thanks to everyone!