BCM5720 lagg

saeed · Mar 21, 2013

I'm trying to aggregate the four interfaces: two on-chip Broadcom BCM5720 A0, ASIC rev and two Intel (R) PRO/1000 Network Connection version - 2.3.4

Code:

ifconfig lagg0 create
ifconfig igb0 up
ifconfig igb1 up
ifconfig bge0 up
ifconfig bge1 up
ifconfig lagg0 up 1.1.1.1/24 laggproto lacp laggport igb0 laggport igb1 laggport bge0 laggport bge1

Code:

ifconfig lagg0
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
        ether 1c:8b:99:36:73:42
        inet 1.1.1.1 netmask 0xffffff00 broadcast 1.1.1.255
        media: Ethernet autoselect
        status: active
        laggproto lacp lagghash l2,l3,l4
        laggport: bge1 flags=18<COLLECTING,DISTRIBUTING>
        laggport: bge0 flags=18<COLLECTING,DISTRIBUTING>
        laggport: igb1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
        laggport: igb0 flags=18<COLLECTING,DISTRIBUTING>

Code:

ifconfig -v lagg0
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
        ether 1c:8b:99:36:73:42
        inet 1.1.1.1 netmask 0xffffff00 broadcast 1.1.1.255
        media: Ethernet autoselect
        status: active
        groups: lagg
        laggproto lacp lagghash l2,l3,l4
        lag id: [(8000,9C-8E-99-26-75-42,01B0,0000,0000),
                 (0001,1C-7E-E5-6C-6B-80,0055,0000,0000)]
        laggport: bge1 flags=18<COLLECTING,DISTRIBUTING> state=7D
                [(8000,9C-8E-99-26-75-42,01B0,8000,0002),
                 (FFFF,00-00-00-00-00-00,0000,FFFF,0000)]
        laggport: bge0 flags=18<COLLECTING,DISTRIBUTING> state=7D
                [(8000,9C-8E-99-26-75-42,01B0,8000,0001),
                 (FFFF,00-00-00-00-00-00,0000,FFFF,0000)]
        laggport: igb1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> state=3D
                [(8000,9C-8E-99-26-75-42,01B0,8000,0004),
                 (0001,1C-7E-E5-6C-6B-80,0055,0001,0056)]
        laggport: igb0 flags=18<COLLECTING,DISTRIBUTING> state=7D
                [(8000,9C-8E-99-26-75-42,01B0,8000,0003),
                 (FFFF,00-00-00-00-00-00,0000,FFFF,0000)]

dmesg to get a lot of messages:

Code:

bge1: 3 link states coalesced
bge1: 3 link states coalesced
bge1: 3 link states coalesced
bge1: 3 link states coalesced
bge1: 3 link states coalesced

as well as many on and off the interfaces:

Code:

bge0: link state changed to UP
bge0: link state changed to DOWN
bge0: link state changed to UP
bge0: link state changed to DOWN
bge0: link state changed to UP
bge0: link state changed to DOWN
bge0: link state changed to UP
bge0: link state changed to DOWN
bge0: link state changed to UP
bge0: link state changed to DOWN
bge0: link state changed to UP
bge0: link state changed to DOWN
bge0: link state changed to UP
bge0: link state changed to DOWN
bge0: link state changed to UP

And according to the traffic on interfaces bge not walk. This whole thing comes from the switch D-Link 3120 with config like this:

Code:

config link_aggregation algorithm mac_source
create link_aggregation group_id 2 type lacp
config link_aggregation group_id 2 master_port 2:21 ports 2:21-2:22,3:23-3:24 state enable
config lacp_port 3:23-3:24 mode active
config lacp_port 2:21-2:22 mode active

If you do only for igb all works and all activity status. Separately for bge aggregation does not work as well or go great loss ...

Code:

hw.bge.allow_asf="0"
hw.pci.enable_msix="0"

does not help

saeed · Mar 22, 2013

Added to /boot/loader.conf:

Code:

hw.bge.allow_asf = 0
hw.pci.enable_msix = 0
dev.bge.0.msi = 0
dev.bge.1.msi = 0

Code:

lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
        ether 1c:8b:99:36:73:42
        inet 1.1.1.1 netmask 0xffffff00 broadcast 1.1.1.255
        media: Ethernet autoselect
        status: active
        laggproto lacp lagghash l2,l3,l4
        laggport: bge0 flags=18<COLLECTING,DISTRIBUTING>
        laggport: bge1 flags=18<COLLECTING,DISTRIBUTING>
        laggport: igb1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
        laggport: igb0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>

Code:

                    /0   /1   /2   /3   /4   /5   /6   /7   /8   /9   /10
     Load Average

      Interface           Traffic               Peak                Total
          lagg0  in      0.000 KB/s          0.000 KB/s           86.959 MB
                 out     0.000 KB/s          0.000 KB/s          209.934 KB

           igb1  in      0.000 KB/s          0.000 KB/s           23.226 MB
                 out     0.000 KB/s          0.000 KB/s            1.764 MB

           igb0  in      0.000 KB/s          0.000 KB/s           63.862 MB
                 out     0.000 KB/s          0.000 KB/s            1.969 MB

           bge1  in      0.000 KB/s          0.000 KB/s            0.121 KB
                 out     0.000 KB/s          0.000 KB/s           61.395 KB

           bge0  in      0.000 KB/s          0.000 KB/s            0.363 KB
                 out     0.000 KB/s          0.000 KB/s           73.262 KB

Traffic on bge does not go, what can you need to put the patches?

BCM5720 lagg

saeed

saeed