FreeBSD15 on Dell R750 with 4 port BCM57504 NetXtreme-E Ethernet

R

rojebsd

BCM57504 has 4 physical ports, each split in 4 logical ports ending up with 16 interfaces: bnxt0 - bnxt15
class=0x020000 rev=0x12 hdr=0x00 vendor=0x14e4 device=0x1801 subvendor=0x14e4 subdevice=0x5045

I'm connecting physical port 1 and 2 to Cisco Nexus switches in a lacp bundle, trunking interfaces and allowed vlans. Just as we configure for esxi and other platforms.
bnxt0 is physical port 1, logical port 1
bnxt1 is physical port 2, logical port 1
..
bnxt4 is physical port 1, logical port 2

Code: 
ifconfig bnxt0 up
ifconfig bnxt1 up
Code: 
ifconfig lagg0 create
ifconfig lagg0 up laggproto lacp laggport bnxt0 laggport bnxt1
ifconfig lagg0 up

ifconfig -v lagg0
lagg0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
  options=4e507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
    ether 04:32:01:4e:69:b0
    hwaddr 00:00:00:00:00:00
    laggproto lacp lagghash l2,l3,l4
    lagg options:
        flags=14<USE_NUMA,LACP_STRICT>
        flowid_shift: 16
    lagg statistics:
        active ports: 0
        flapping: 0
    lag id: [(0000,00-00-00-00-00-00,0000,0000,0000),
         (0000,00-00-00-00-00-00,0000,0000,0000)]
    laggport: bnxt0 flags=0<> state=45<ACTIVITY,AGGREGATION,DEFAULTED>
        [(8000,04-32-01-4E-69-B0,0299,8000,0004),
         (FFFF,00-00-00-00-00-00,0000,FFFF,0000)]
    laggport: bnxt1 flags=0<> state=45<ACTIVITY,AGGREGATION,DEFAULTED>
        [(8000,04-32-01-4E-69-B0,0299,8000,0005),
         (FFFF,00-00-00-00-00-00,0000,FFFF,0000)]
    groups: lagg
    media: Ethernet autoselect
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    drivername: lagg0
I don't see my laggports as ACTIVE

ifconfig bnxt0 (or bnxt1) both show status: active

I also tried without lagg and just configure an allowed vlan on bnxt0:
Code: 
ifconfig vlan205 create
ifconfig vlan205 vlan 205 vlandev bnxt0 100.205.0.3 netmask 255.255.255.0
ifconfig vlan205 up

ping to a working host in vlan205 immediately give sendto: Network is down

Does someone have experience with the BCM57504 or similar cards as I assume I'm missing options in my config

Thanks
 
We did miss an item in switch LACP config, and one of DAC cable transceivers was coded wrong.
My network colleagues have also checked that LACP is working between more switches.

Code: 
ifconfig -v lagg0
lagg0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
    options=4e507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
    ether 04:32:01:4e:69:b0
    hwaddr 00:00:00:00:00:00
    laggproto lacp lagghash l2,l3,l4
    lagg options:
        flags=14<USE_NUMA,LACP_STRICT>
        flowid_shift: 16
    lagg statistics:
        active ports: 2
        flapping: 0
    lag id: [(0000,00-00-00-00-00-00,0000,0000,0000),
         (0000,00-00-00-00-00-00,0000,0000,0000)]
    laggport: bnxt0 flags=18<COLLECTING,DISTRIBUTING> state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
        [(8000,04-32-01-4E-69-B0,0299,8000,0004),
         (007F,54-E0-32-99-D6-C0,0001,007F,0001)]
    laggport: bnxt1 flags=18<COLLECTING,DISTRIBUTING> state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
        [(8000,04-32-01-4E-69-B0,0299,8000,0005),
         (007F,54-E0-32-99-D6-C0,0001,007F,0002)]
    groups: lagg
    media: Ethernet autoselect
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    drivername: lagg0

Configured vlan to test connectivity
Code: 
ifconfig -v vlan205
vlan205: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
    options=4600703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,RXCSUM_IPV6,TXCSUM_IPV6,MEXTPG>
    ether 04:32:01:4e:69:b0
    inet 100.205.0.3 netmask 0xffffff00 broadcast 100.126.0.255
    groups: vlan
    vlan: 205 vlanproto: 802.1q vlanpcp: 0 parent interface: lagg0
    media: Ethernet autoselect
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    drivername: vlan205
Code: 
# ping 100.205.0.201
PING 100.205.0.201 (100.205.0.201): 56 data bytes
ping: sendto: Network is down
Network connectivity in vlan have been verified between switches, so switch config is correct.

From switch
Code: 
testlab> show lacp interfaces  
Aggregated interface: ae0
   LACP state:       Role   Exp   Def  Dist  Col  Syn  Aggr  Timeout  Activity
     xe-0/0/1       Actor    No    No   Yes  Yes  Yes   Yes     Fast    Active
     xe-0/0/1     Partner    No    No   Yes  Yes  Yes   Yes     Slow    Active
     xe-0/0/3       Actor    No    No   Yes  Yes  Yes   Yes     Fast    Active
     xe-0/0/3     Partner    No    No   Yes  Yes  Yes   Yes     Slow    Active
   LACP protocol:        Receive State  Transmit State          Mux State  
     xe-0/0/1                  Current   Slow periodic Collecting distributing
xe-0/0/3 Current Slow periodic Collecting distributing

Partner (freebsd15) has timeout role slow, should be fast but I asume its not the root cause
 
You must log in or register to reply here.
Back
Top