Mellanox SFP+ and LACP

T

trumee

Hello,

I have a Ubiquiti ES-16-XG (10Gb) switch and Mellanox ConnectX-2 dual port SFP+ card. I am trying to setup LACP between the two devices.

Unfortunately, LACP isnt working for me with the Mellanox card. The config pasted below has worked for me in the past for 1GB NICs.

The mlxen1 does not participate in the LAGG. If i take the mlxen0 cable out, the connection drops. However, if i take the mlxen1 cable out, the connection keeps up.

My config is as follows:
Code: 
ifconfig_ix0="down"
ifconfig_ix1="down"

ifconfig_em0="-rxcsum -txcsum -vlanmtu -vlanhwtso -tso4 -tso6 -vlanhwtso -vlanhwcsum -vlanhwtag up"

ifconfig_mlxen0="up"
ifconfig_mlxen1="up"


cloned_interfaces="lagg0 vlan100 vlan200 "
ifconfig_lagg0="laggproto lacp laggport mlxen0 laggport mlxen1"

ifconfig_vlan100="inet 192.168.1.5 netmask 255.255.255.0 vlan 100 vlandev lagg0 fib 0"
ifconfig_vlan200="inet 192.168.2.5 netmask 255.255.255.0 vlan 200 vlandev lagg0 fib 1"
ifconfig_lo1="inet 127.0.0.2 netmask 255.255.255.0 fib 1"

defaultrouter="192.168.1.1"

The interfaces are up
Code: 
mlxen0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=6d07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (10Gbase-CX4 <full-duplex,rxpause,txpause>)
        status: active
mlxen1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=6d07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (10Gbase-CX4 <full-duplex,rxpause,txpause>)
        status: active
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=6d07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: active
        groups: lagg
        laggproto lacp lagghash l2,l3,l4
        laggport: mlxen0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
        laggport: mlxen1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
vlan100: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=303<RXCSUM,TXCSUM,TSO4,TSO6>
        inet 192.168.1.5 netmask 0xffffff00 broadcast 192.168.1.255
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: active
        vlan: 100 vlanpcp: 0 parent interface: lagg0
        groups: vlan
vlan200: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=303<RXCSUM,TXCSUM,TSO4,TSO6>
        inet 192.168.2.5 netmask 0xffffff00 broadcast 192.168.2.255
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: active
        fib: 1
        vlan: 200 vlanpcp: 0 parent interface: lagg0

I tried to boot into Arch Linux, and was able to setup LACP (without vlans) successfully.
 
Just out of curiosity.....

Just installed a Mellanox ConnectX-2 card in my DL360 G7 running FBSD 11...
but the card isn't even detected during boot...

How did you manage to have the driver loaded?
 
Ok, it seems the FreeBSD driver may be faulty. I replaced with a Solarflare SFC9020 card and LACP works fine.

davorin specify these in /boot/loader.conf
Code: 
mlx4_load="YES"
mlxen_load="YES"
 
As a thought, if you don't get the Mellanox card working it's probably worth asking on the freebsd-infiniband mailing list. The Mellanox FreeBSD guys are on that mailing list, and they're generally pretty helpful.
 
You must log in or register to reply here.
Back
Top