Solved LACP not aggregating, flags=0

P

P-Chan

I've found lots of other posts about LACP on the forums, but nothing that helps me so far. I'm trying to set up LACP between 2 Intel NICs

Code: 
igb0: <Intel(R) PRO/1000 PCI-Express Network Driver> port 0xa000-0xa01f mem 0xdf300000-0xdf37ffff,0xdf380000-0xdf383fff irq 21 at device 0.0 on pci7
igb1: <Intel(R) PRO/1000 PCI-Express Network Driver> port 0x9000-0x901f mem 0xdf200000-0xdf27ffff,0xdf280000-0xdf283fff irq 23 at device 0.0 on pci8

and an HP 1920-16G switch (Gigabit switch, L2 managed)
I set up the switch with 2 different LAGGs, one for the FreeBSD server and one for my Linux firewall/docker system.

The gist of my problem is that the LAGG interface is only active on 1 NIC, it won't go into the right mode on the second NIC. The switch reports (I had to type this, couldn't copy/paste):
GigabitEthernet1/0/14 Unselected "The port's partner is improper for being attached"
GigabitEthernet1/0/16 Selected --
In LACP summary, 0/16 is active, "Partner Port 2" "Partner Port State: ACDEF, Oper Key 3
0/14, inactive, reason 4(Ports partner is improper for being attached), Partner Port 0, Partner Port State: DEF, Oper Key 3

In the details for partner port, only the 0/16 interface has details that aren't 0x00000,0000*

Code: 
FreeBSD entryplug 12.0-RELEASE FreeBSD 12.0-RELEASE r341666 GENERIC  amd64

ifconfig -a:

Code: 
igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=e527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether d0:50:99:c3:12:3d
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
igb1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=e527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether d0:50:99:c3:12:3d
        hwaddr d0:50:99:c3:12:3e
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=e527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether d0:50:99:c3:12:3d
        inet 10.11.12.192 netmask 0xffffff00 broadcast 10.11.12.255
        laggproto lacp lagghash l2,l3,l4
        laggport: igb0 flags=0<>
        laggport: igb1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
        groups: lagg
        media: Ethernet autoselect
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

and my rc.conf: (the ifconfig igb0 and igb1 options produce errors during startup, that these aren't recognized, but the interfaces do go up and seem ok, from what I can tell):
Code: 
ifconfig_igb0="UP"
ifconfig_igb1="UP"
cloned_interfaces="lagg0"
ifconfig_lagg0="laggproto lacp laggport igb0 laggport igb1 DHCP"

After research and troubleshooting, I found a mention of adding this to sysctl (and did so):
Code: 
net.link.lagg.lacp.default_strict_mode=0
(rather, the mention for was for the specific lagg group, but I set it as default as the specifying the lagg group was producing an error, but that value was listed in sysctl -a)

I could turn on lacp.debug in sysctl, if someone thinks that info might help. I'm hoping it's just something simple (it's my first time setting up LACP, so I wouldn't be surprised if it was just user error).

Thanks for any assistance anyone might have in getting me moving in the right direction with this.
 
How's the HP configured? On my HP 2530 I have this:
Code: 
trunk 11-12 trk1 lacp
In order to assign port 11 and 12 to Trk1 with LACP. My host is connected to those two ports:
Code: 
lagg0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
        options=8120b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER>
        ether 00:25:90:f1:58:39
        laggproto lacp lagghash l2,l3,l4
        laggport: igb1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
        laggport: igb2 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
        groups: lagg
        media: Ethernet autoselect
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
 
SirDice said:
How's the HP configured? On my HP 2530 I have this:
Code: 
trunk 11-12 trk1 lacp
Click to expand...

I don't see that anywhere from the telnet interface, I enabled cmdline and did a "display port trunk" and it was empty, the configurations I did were via web interface (the cmdline on this model had to be enabled through special dev options, this device was intended to be configured via web UI).

My config (I have 2 different LAGGs defined, I'll remove the one I'm not using yet, just to see if that changes anything):
Code: 
#
 version 5.20.99, Release 1120
#
 sysname HPE
#
 domain default enable system
#
 ipv6
#
 telnet server enable
#
 ssl version ssl3.0 disable
#
 password-recovery enable
#
vlan 1
#
domain system
 access-limit disable
 state active
 idle-cut disable
 self-service-url disable
#
user-group system
 group-attribute allow-guest
#
local-user admin
 password cipher 
 authorization-attribute level 3
 service-type lan-access
 service-type ssh telnet terminal
 service-type web
#
 stp mode rstp
 stp enable
#
interface Bridge-Aggregation1
 link-aggregation mode dynamic
#
interface Bridge-Aggregation2
 link-aggregation mode dynamic
#
interface NULL0
#
interface Vlan-interface1
 ip address dhcp-alloc
#               
interface GigabitEthernet1/0/1
 speed auto 1000
 port auto-power-down
 stp edged-port enable
#
(above config repeated through 0/12)
#
interface GigabitEthernet1/0/13
 speed auto 1000
 port auto-power-down
 stp edged-port enable
 port link-aggregation group 1
#
interface GigabitEthernet1/0/14
 speed auto 1000
 port auto-power-down
 stp edged-port enable
 port link-aggregation group 2
#
interface GigabitEthernet1/0/15
 speed auto 1000
 port auto-power-down
 stp edged-port enable
 port link-aggregation group 1
#
interface GigabitEthernet1/0/16
 speed auto 1000
 port auto-power-down
 stp edged-port enable
 port link-aggregation group 2
#
interface GigabitEthernet1/0/17
 stp edged-port enable
#
interface GigabitEthernet1/0/18
 stp edged-port enable
#
interface GigabitEthernet1/0/19
 stp edged-port enable
#
interface GigabitEthernet1/0/20
 stp edged-port enable
#
 undo info-center logfile enable
#
 load xml-configuration
#
user-interface aux 0
 authentication-mode scheme
user-interface vty 0 15
 authentication-mode scheme
#
return

In order to assign port 11 and 12 to Trk1 with LACP. My host is connected to those two ports:
Code: 
lagg0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
        options=8120b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER>
        ether 00:25:90:f1:58:39
        laggproto lacp lagghash l2,l3,l4
        laggport: igb1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
        laggport: igb2 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
        groups: lagg
        media: Ethernet autoselect
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
Click to expand...

And that looks a lot more like what I would expect/hope to see.
 
Different model switch, it has a different interface. But if I look at your configuration your LACP group 1 consists of interfaces 13 and 15. Group 2 is port 14 and 16. Make sure the FreeBSD host is connected to 13 and 15, or 14 and 16. Don't use ports from different groups.

Also check if LACP on the switch is really 802.1AX.
 
SirDice said:
How's the HP configured? On my HP 2530 I have this:
Code: 
trunk 11-12 trk1 lacp
Click to expand...

Ok, so I figured out how to set up trunking on those 2 ports:
<HPE>display port trunk
Interface PVID VLAN passing
BAGG2 1 1,
GE1/0/14 1 1,
GE1/0/16 1 1,


I temporarily disconnected/reconnected the ports, hoping it would cause FreeBSD to change it's flags and renegotiate, but that didn't help, still seeing the same issue. Wouldn't I need to now add the VLAN ID to the LAGG configuration in FreeBSD?
 
No need for a VLAN ID. There's no VLAN configured on the switch so all ports are in the default VLAN. You get untagged traffic on the ports. This in turn means that you don't need a VLAN configured on the FreeBSD side. You only need vlan(4) when you have tagged traffic.

Looking further at your configuration, you might want to turn off STP. It may detect a loop and disable one of the ports.
 
SirDice said:
Different model switch, it has a different interface. But if I look at your configuration your LACP group 1 consists of interfaces 13 and 15. Group 2 is port 14 and 16. Make sure the FreeBSD host is connected to 13 and 15, or 14 and 16. Don't use ports from different groups.

Also check if LACP on the switch is really 802.1AX.
Click to expand...
I removed the other LAGG, only 3 ports are used on the switch currently, the LAGG ports (14/16) and another port (connecting to my existing switch).
Docs say it supports IEEE 802.3ad, which looks like it was the previous name for 802.1ax.
I did upgrade the firmware on it so it has the latest version, released in 2018.
5.20.99 Release 1120
 
One more setting I have but this might just be a default setting, I don't remember setting it specifically.

Code: 
trunk-load-balance L2-based

My HP switch is an enterprise model, so it might have a few extra options compared to yours.

On the FreeBSD I only configured lagg(4), nothing else. No sysctl(8).
Code: 
ifconfig_lagg0="laggproto lacp laggport igb1 laggport igb2"
It worked instantly.

Oh, one more thing. Have you tried a different cable? I've had a few cables that would give an active link but never worked, probably a bad cable with one or more bad wires.
 
SirDice said:
No need for a VLAN ID. There's no VLAN configured on the switch so all ports are in the default VLAN. You get untagged traffic on the ports. This in turn means that you don't need a VLAN configured on the FreeBSD side. You only need vlan(4) when you have tagged traffic.

Looking further at your configuration, you might want to turn off STP. It may detect a loop and disable one of the ports.
Click to expand...

I turned off STP. Also, I have no options for trunk-load-balance L2-based. Here's more current config (I've been saving/rebooting the switch between config changes, since I can in my case):
Code: 
#
version 5.20.99, Release 1120
#
sysname HPE
#
domain default enable system
#
ipv6
#
telnet server enable
#
ssl version ssl3.0 disable
#
password-recovery enable
#
vlan 1
#
domain system
access-limit disable
state active
idle-cut disable
self-service-url disable
#
user-group system
group-attribute allow-guest
#
local-user admin
password cipher
authorization-attribute level 3
service-type lan-access
service-type ssh telnet terminal
service-type web
#
stp mode rstp
#
interface Bridge-Aggregation2
port link-type trunk
port trunk permit vlan 1
link-aggregation mode dynamic
stp disable
#
interface NULL0
#
interface Vlan-interface1
ip address dhcp-alloc
#
interface GigabitEthernet1/0/1
speed auto 1000
port auto-power-down
stp disable
#
interface GigabitEthernet1/0/2
speed auto 1000
port auto-power-down
stp disable
#
interface GigabitEthernet1/0/3
speed auto 1000
port auto-power-down
stp disable
#
interface GigabitEthernet1/0/4
speed auto 1000
port auto-power-down
stp disable
#
interface GigabitEthernet1/0/5
speed auto 1000
port auto-power-down
stp disable  
#
interface GigabitEthernet1/0/6
speed auto 1000
port auto-power-down
stp disable
#
interface GigabitEthernet1/0/7
speed auto 1000
port auto-power-down
stp disable
#
interface GigabitEthernet1/0/8
speed auto 1000
port auto-power-down
stp disable
#
interface GigabitEthernet1/0/9
speed auto 1000
port auto-power-down
stp disable
#
interface GigabitEthernet1/0/10
speed auto 1000
port auto-power-down
stp disable
#
interface GigabitEthernet1/0/11
speed auto 1000
port auto-power-down
stp disable
#
interface GigabitEthernet1/0/12
speed auto 1000
port auto-power-down
stp disable
#
interface GigabitEthernet1/0/13
speed auto 1000
port auto-power-down
stp disable
#
interface GigabitEthernet1/0/14
port link-type trunk
port trunk permit vlan 1
speed auto 1000
port auto-power-down
stp disable
port link-aggregation group 2
#
interface GigabitEthernet1/0/15
speed auto 1000
port auto-power-down
stp disable
#
interface GigabitEthernet1/0/16
port link-type trunk
port trunk permit vlan 1
speed auto 1000
port auto-power-down
stp disable
port link-aggregation group 2
#
interface GigabitEthernet1/0/17
stp disable
#
interface GigabitEthernet1/0/18
stp disable
#
interface GigabitEthernet1/0/19
stp disable
#
interface GigabitEthernet1/0/20
stp disable
#
undo info-center logfile enable
#
load xml-configuration
#
user-interface aux 0
authentication-mode scheme
user-interface vty 0 15
authentication-mode scheme
#
return

VLAN 1 is an untagged vlan with all ports and trunk groups (default setup). As for the cables, both are pre-made and work separately and they are both the same length.

I think I might try configuring this with Linux (my firewall) next, just to try to isolate if it's the switch or the OS config, although it sounds like it should just work on the FreeBSD side, based on the configs I have (and what you've mentioned). I'll need to wait until I'm done with work today (telecommute, the firewall/router is the Linux box I'd be working with).

Thanks again for all the help you've been in working through the possible issues/options with me. If you can think of anything else on the FreeBSD/switch side to try, I can do that, otherwise it'll be about 7 hours before I can try on the Linux side to see if that works.
 
So the good news is, it works right out of the gate on my Ubuntu system, so the LACP setup on the switch is not at fault. However, I still need to figure out how to do this in FreebBSD (my configs are listed in a previous post). Here's the switch info:
Code: 
             LACP     Port                    *Inactive Partner   Partner       Oper 
Unit  Port   State    Priority  State         Reason    Port      Port State    Key  
1     0/14   Enable   32768     Active        None      2         ABCDEF        2
1     0/14   Enable   32768     Active        None      1         ABCDEF        2

And my config on the linux side (Ubuntu, netplan - also I have the bridge for docker to attach to my local network, that's why the 1 interface bridge):
Code: 
network:
  version: 2
  renderer: networkd
  ethernets:
    enp4s0:
      dhcp4: true
      nameservers:
        addresses: [ 10.11.12.1 ]
    enp6s0:
      dhcp4: no
    enp8s0:
      dhcp4: no
    enp9s0:
      dhcp4: no
  bridges:
    lan0:
      addresses: [ 10.11.12.1/24 ]
      interfaces: [ bond0 ]
      nameservers:
        search: [ nerv-02.otakuuniversity.org ]
        addresses: [ 10.11.12.1 ]
  bonds:
    bond0:
      interfaces: [ enp8s0, enp9s0 ]
      parameters:
        transmit-hash-policy: "layer3+4"
        mode: 802.3ad
        lacp-rate: fast
        mii-monitor-interval: 100
      dhcp4: no

Any thoughts? Thanks again for the help.
 
I enabled debug on LACP and am getting this:
Code: 
Mar  6 19:12:19 entryplug kernel: igb0: lacpdu transmit
Mar  6 19:12:19 entryplug kernel: actor=(8000,D0-50-99-C3-12-3D,008B,8000,0001)
Mar  6 19:12:19 entryplug kernel: actor.state=45<ACTIVITY,AGGREGATION,DEFAULTED>
Mar  6 19:12:19 entryplug kernel: partner=(FFFF,00-00-00-00-00-00,0000,FFFF,0000)
Mar  6 19:12:19 entryplug kernel: partner.state=3c<AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
Mar  6 19:12:19 entryplug kernel: maxdelay=0
Mar  6 19:12:31 entryplug kernel: igb1: lacpdu receive
Mar  6 19:12:31 entryplug kernel: actor=(8000,44-31-92-51-FA-01,0003,8000,0010)
Mar  6 19:12:31 entryplug kernel: actor.state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
Mar  6 19:12:31 entryplug kernel: partner=(8000,D0-50-99-C3-12-3D,008B,8000,0002)
Mar  6 19:12:31 entryplug kernel: partner.state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
Mar  6 19:12:31 entryplug kernel: maxdelay=0
Mar  6 19:12:46 entryplug kernel: igb1: lacpdu transmit
Mar  6 19:12:46 entryplug kernel: actor=(8000,D0-50-99-C3-12-3D,008B,8000,0002)
Mar  6 19:12:46 entryplug kernel: actor.state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
Mar  6 19:12:46 entryplug kernel: partner=(8000,44-31-92-51-FA-01,0003,8000,0010)
Mar  6 19:12:46 entryplug kernel: partner.state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
Mar  6 19:12:46 entryplug kernel: maxdelay=0
Mar  6 19:12:50 entryplug kernel: igb0: lacpdu transmit
Mar  6 19:12:50 entryplug kernel: actor=(8000,D0-50-99-C3-12-3D,008B,8000,0001)
Mar  6 19:12:50 entryplug kernel: actor.state=45<ACTIVITY,AGGREGATION,DEFAULTED>
Mar  6 19:12:50 entryplug kernel: partner=(FFFF,00-00-00-00-00-00,0000,FFFF,0000)
Mar  6 19:12:50 entryplug kernel: partner.state=3c<AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
Mar  6 19:12:50 entryplug kernel: maxdelay=0

In the UI for the switch, I see:
partner port 2, active, state ACDEF, Oper Key 3
partner port 0, inactive, state DEF, Oper Key 3
Inactive reason is 4, which is "The Port's partner is improper for being attached" and no partner ID or partner oper key listed
On the active port, I see
PartnerID 0x8000,D050-99C3-123D with partner oper key of 139

I don't know enough about LACP to diagnose this, hopefully someone here does. I appreciate all the help.
 
I'm not sure if this applies to your switch:
Each trunk group must start with a primary port. The primary port is always the lowest number in the port range. For example, on the J4140A 10/100 module:
• Ranges for four-port trunk groups: 1 – 4, 5 – 8, 9 – 15, 16 – 20, 21 – 24
• Ranges for two-port trunk groups: 1 – 2, 3 – 4, 5 – 6, 7 – 8, 9 – 10, 11 – 12, 13 – 14, 15 – 16, 17 – 18, 19 – 20, 21 – 22, 23 – 24
Click to expand...
But try to create a group with 15+16 or 13+14 instead.

ftp://ftp.hp.com/pub/networking/software/59906028_ch07.pdf
 
Thanks for all the help, I finally figured it out. The cables used worked fine on their own and were of the same length/brand, but somehow weren't working for the LAGG. This morning I received some new cables to swap with (the same switch ports worked as a lagg in Linux, but those were 6 inch patch cables that wouldn't reach to the FreeBSD system. I'm not sure why the cables would have caused the issue, all things being the same, but I ran new cables and it just started working. I'll probably now back out some of my debugging and extra things I added for testing and try it with the default settings again. In any case, I appreciate your patience in helping with this, I'm just glad it's all fixed now.
 
You must log in or register to reply here.
Back
Top