LAGGs, VLANs, and STP help needed

B

Blue|Fusion

Hi all, I have two servers now with multiple interfaces at 1GigE and either 10 or 40GigE (soon to both be 40GigE). The 40GigE interfaces are temporarily attached to the same switch as the 1G and 10G links until a new 40GigE switch arrives. I wanted to set up a way to keep these servers online in the event the primary 40Gig links fail (i.e. that switch is offline). I figure it's best to go about this with STP.

The two servers are ecorp and allsafe. Both host Jails while one also hosts bhyve VMs. Jails are assigned an epair on a specific vlan.

Here is the network configuration from ecorp's /etc/rc.conf
Code: 
# 1GigE interfaces
ifconfig_em0="up"
ifconfig_em1="up"

# 10GigE interfaces
ifconfig_mlxen0="up"
ifconfig_mlxen1="up"

# Create member interfaces and rename them as desired
cloned_interfaces="lagg0 lagg1 bridge100 bridge101 bridge140 bridge150"
ifconfig_lagg0_name="lagg10g"
ifconfig_lagg1_name="lagg1g"
ifconfig_bridge100_name="vlan100"
ifconfig_bridge101_name="vlan101"
ifconfig_bridge140_name="vlan140"
ifconfig_bridge150_name="vlan150"

# Add ports to respective laggs and create vlan members
ifconfig_lagg10g="laggproto lacp laggport mlxen0 laggport mlxen1"
ifconfig_lagg1g="laggproto lacp laggport em0 laggport em1"
vlans_lagg10g="100 101 140 150"
vlans_lagg1g="100 101 140 150"

# Bring up all interfaces
ifconfig_lagg10g_100="up"
ifconfig_lagg10g_101="up"
ifconfig_lagg10g_140="up"
ifconfig_lagg10g_150="up"
ifconfig_lagg1g_100="up"
ifconfig_lagg1g_101="up"
ifconfig_lagg1g_140="up"
ifconfig_lagg1g_150="up"

# Add lagg.vlan members to their respective bridges with STP (RSTP)
ifconfig_vlan100="addm lagg10g.100 stp lagg10g.100 ptp lagg10g.100 addm lagg1g.100 stp lagg1g.100 ptp lagg1g.100 up"
ifconfig_vlan101="addm lagg10g.101 stp lagg10g.101 ptp lagg10g.101 addm lagg1g.101 stp lagg1g.101 ptp lagg1g.101 up"
ifconfig_vlan140="addm lagg10g.140 stp lagg10g.140 ptp lagg10g.140 addm lagg1g.140 stp lagg1g.140 ptp lagg1g.140 up"
ifconfig_vlan150="addm lagg10g.150 stp lagg10g.150 ptp lagg10g.150 addm lagg1g.150 stp lagg1g.150 ptp lagg1g.150 up"

# Host (this) machine addresses
ifconfig_vlan140_alias0="inet 172.23.40.10/24"
ifconfig_vlan140_alias1="inet6 fd33:58bc:59a0:2340::10/64 accept_rtadv"
defaultrouter="172.23.40.1"

And here is allsafe's /etc/rc.conf
Code: 
# 1GigE interfaces
ifconfig_igb0="up"
ifconfig_igb1="up"

# 40GigE interfaces
ifconfig_mlxen0="up"
ifconfig_mlxen1="up"

# Create member interfaces and rename them as desired
cloned_interfaces="lagg0 lagg1 bridge101 bridge110 bridge140 bridge150"
ifconfig_lagg0_name="lagg40g"
ifconfig_lagg1_name="lagg1g"
ifconfig_bridge101_name="vlan101"
ifconfig_bridge110_name="vlan110"
ifconfig_bridge140_name="vlan140"
ifconfig_bridge150_name="vlan150"

# Add ports to respective laggs and create vlan members
ifconfig_lagg40g="laggproto lacp laggport mlxen0 laggport mlxen1"
ifconfig_lagg1g="laggproto lacp laggport igb0 laggport igb1"
vlans_lagg40g="101 110 140 150"
vlans_lagg1g="101 110 140 150"

# Bring up all interfaces
ifconfig_lagg40g_101="up"
ifconfig_lagg40g_110="up"
ifconfig_lagg40g_140="up"
ifconfig_lagg40g_150="up"
ifconfig_lagg1g_101="up"
ifconfig_lagg1g_110="up"
ifconfig_lagg1g_140="up"
ifconfig_lagg1g_150="up"

# Add lagg.vlan members to their respective bridges with STP (RSTP)
ifconfig_vlan101="addm lagg40g.101 stp lagg40g.101 ptp lagg40g.101 addm lagg1g.101 stp lagg1g.101 ptp lagg1g.101 up"
ifconfig_vlan110="addm lagg40g.110 stp lagg40g.110 ptp lagg40g.110 addm lagg1g.110 stp lagg1g.110 ptp lagg1g.140 up"
ifconfig_vlan140="addm lagg40g.140 stp lagg40g.140 ptp lagg40g.140 addm lagg1g.140 stp lagg1g.140 ptp lagg1g.150 up"
ifconfig_vlan150="addm lagg40g.150 stp lagg40g.150 ptp lagg40g.150 addm lagg1g.150 stp lagg1g.150 ptp lagg1g.150 up"

# Host (this) machine addresses
ifconfig_vlan140_alias0="inet 172.23.40.11/24"
ifconfig_vlan140_alias1="inet6 fd33:58bc:59a0:2340::11/64 accept_rtadv"
defaultrouter="172.23.40.1"

Problem #1
The ecorp server does not give an appropriate interface cost to it's 10Gig links, and the 1Gig links get prioritized.

Ecorp's ifconfig vlan140
Code: 
vlan140: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=0
ether 58:9c:fc:10:ff:f2
inet 172.23.40.10 netmask 0xffffff00 broadcast 172.23.40.255
inet6 fd33:58bc:59a0:2340::10 prefixlen 64
id 00:02:c9:1c:46:80 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 74:8e:f8:e7:b4:b0 priority 4096 ifcost 2000000 port 18
member: lagg1g.140 flags=147<LEARNING,DISCOVER,STP,AUTOEDGE,AUTOPTP>
        ifmaxaddr 0 port 18 priority 128 path cost 2000000 proto rstp
        role root state forwarding
member: lagg10g.140 flags=1c7<LEARNING,DISCOVER,STP,AUTOEDGE,PTP,AUTOPTP>
        ifmaxaddr 0 port 14 priority 128 path cost 2000000 proto rstp
        role disabled state discarding
groups: bridge
nd6 options=3<PERFORMNUD,ACCEPT_RTADV>

Compare that to allsafe's ifconfig vlan140
Code: 
vlan140: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=0
        ether 58:9c:fc:10:e6:52
        inet 172.23.40.11 netmask 0xffffff00 broadcast 172.23.40.255
        inet6 fd33:58bc:59a0:2340::11 prefixlen 64
        id 00:25:90:18:9a:c0 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 74:8e:f8:e7:b4:b0 priority 4096 ifcost 500 port 14
        member: lagg1g.140 flags=147<LEARNING,DISCOVER,STP,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 18 priority 128 path cost 10000 proto rstp
                role alternate state discarding
        member: lagg40g.140 flags=147<LEARNING,DISCOVER,STP,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 14 priority 128 path cost 500 proto rstp
                role root state forwarding
        groups: bridge
        nd6 options=3<PERFORMNUD,ACCEPT_RTADV>

And here is the switch interface briefing:
Code: 
Port       Link    State   Dupl Speed Trunk Tag Pvid Pri MAC             Name
1/1/7      Up      Forward Full 1G    5     Yes N/A  0   748e.f8e7.b4b0  allsafe-1G-lag
1/1/8      Up      Forward Full 1G    5     Yes N/A  0   748e.f8e7.b4b0  allsafe-1G-lag
1/1/15     Up      Forward Full 1G    8     Yes N/A  0   748e.f8e7.b4b0  ecorp-1G-lag
1/1/16     Up      Forward Full 1G    8     Yes N/A  0   748e.f8e7.b4b0  ecorp-1G-lag
1/2/1      Up      Forward Full 40G   6     Yes N/A  0   748e.f8e7.b4b0  allsafe-40G-lag
1/2/3      Up      Blocked Full 10G   2     Yes N/A  0   748e.f8e7.b4b0  ecorp-10G-lag
1/2/8      Up      Blocked Full 10G   2     Yes N/A  0   748e.f8e7.b4b0  ecorp-10G-lag

And the RSTP information fro the switch:
Code: 
SSH@sw1#show 802-1w vlan 140

--- VLAN 140 [ STP Instance owned by VLAN 140 ] ----------------------------

Bridge IEEE 802.1W Parameters:

Bridge           Bridge Bridge Bridge Force    tx
Identifier       MaxAge Hello  FwdDly Version  Hold
hex              sec    sec    sec             cnt
1000748ef8e7b4b0 20     2      15     Default  3

RootBridge       RootPath  DesignatedBri-   Root   Max Fwd Hel
Identifier       Cost      dge Identifier   Port   Age Dly lo
hex                        hex                     sec sec sec
1000748ef8e7b4b0 0         1000748ef8e7b4b0 Root   20  15  2

Port IEEE 802.1W Parameters:

          <--- Config Params --><-------------- Current state ----------------->
Port      Pri PortPath P2P Edge Role       State       Designa-  Designated  
Num           Cost     Mac Port                        ted cost  bridge      
1/1/7     128 20000    T   F    DESIGNATED FORWARDING  0         1000748ef8e7b4b0
1/1/8     128 20000    T   F    DESIGNATED FORWARDING  0         1000748ef8e7b4b0
1/1/15    128 20000    T   F    DESIGNATED FORWARDING  0         1000748ef8e7b4b0
1/1/16    128 20000    T   F    DESIGNATED FORWARDING  0         1000748ef8e7b4b0
1/2/1     128 1400     T   F    DESIGNATED FORWARDING  0         1000748ef8e7b4b0
1/2/2     128 0        F   F    DISABLED   DISABLED    0         0000000000000000
1/2/3     128 0        F   F    DISABLED   DISABLED    0         0000000000000000    <----- ecorp 10Gig blocked but shouldn't be
1/2/4     128 0        F   F    DISABLED   DISABLED    0         0000000000000000
1/2/5     128 0        F   F    DISABLED   DISABLED    0         0000000000000000
1/2/6     128 0        F   F    DISABLED   DISABLED    0         0000000000000000
1/2/7     128 0        F   F    DISABLED   DISABLED    0         0000000000000000
1/2/8     128 0        F   F    DISABLED   DISABLED    0         0000000000000000    <---- ecorp 10Gig blocked but shouldn't be
1/2/9     128 0        F   F    DISABLED   DISABLED    0         0000000000000000
1/2/10    128 0        F   F    DISABLED   DISABLED    0         0000000000000000

The switch's interface configs for these ports in question (note: the LACP ports only show config from one port as configs are mirrored to the other port):
Code: 
lag allsafe-1G dynamic id 5                                       
 ports ethernet 1/1/7 to 1/1/8 
 primary-port 1/1/7
 lacp-timeout short
 deploy
 port-name allsafe-1G-lag ethernet 1/1/7
 port-name allsafe-1G-lag ethernet 1/1/8

lag allsafe-40G dynamic id 6
 ports ethernet 1/2/1 
 primary-port 1/2/1
 lacp-timeout short
 deploy
 port-name allsafe-40G-lag ethernet 1/2/1

lag ecorp-10G dynamic id 2
 ports ethernet 1/2/3 ethernet 1/2/8 
 primary-port 1/2/3
 lacp-timeout short
 deploy
 port-name ecorp-10G-lag ethernet 1/2/3
 port-name ecorp-10G-lag ethernet 1/2/8

lag ecorp-1G dynamic id 8                                         
 ports ethernet 1/1/15 to 1/1/16 
 primary-port 1/1/15
 lacp-timeout short
 deploy
 port-name ecorp-1G-lag ethernet 1/1/15
 port-name ecorp-1G-lag ethernet 1/1/16

interface ethernet 1/1/7
 port-name allsafe-1G-lag
 spanning-tree root-protect
 spanning-tree 802-1w admin-pt2pt-mac
 no flow-control
 trust dscp
!
interface ethernet 1/1/15
 port-name ecorp-1G-lag
 dhcp snooping trust
 spanning-tree root-protect
 spanning-tree 802-1w admin-pt2pt-mac
 no flow-control
 trust dscp
!
interface ethernet 1/2/1
 port-name allsafe-40G-lag
 spanning-tree root-protect
 spanning-tree 802-1w admin-pt2pt-mac
 no flow-control
 trust dscp
!
interface ethernet 1/2/3
 port-name ecorp-10G-lag                                      
 dhcp snooping trust
 spanning-tree root-protect
 spanning-tree 802-1w admin-pt2pt-mac
 no flow-control
 trust dscp

Problem #2

IPv6 does not work as expected on the bridge interface. It should end up with a ULA statically assigned from rc.conf and a GLA from SLAAC. I can manually add the ULA address, but it is not applied by rc.conf. How can I get these two to work appropriately?
 
Last edited:
Okay so slight changes. I have been doing a bunch of reading, research, and trial-and-error. I now have the following configurations:

allsafe /etc/rc.conf
Code: 
# 1GigE interfaces
ifconfig_igb0="up"
ifconfig_igb1="up"

# 40GigE interfaces
ifconfig_mlxen0="up"
ifconfig_mlxen1="up"

# Create member interfaces and rename them as desired
cloned_interfaces="lagg0 lagg1 bridge0 bridge1 bridge2 bridge3"
ifconfig_lagg0_name="lagg40g"
ifconfig_lagg1_name="lagg1g"
ifconfig_bridge0_name="vlan101"
ifconfig_bridge1_name="vlan110"
ifconfig_bridge2_name="vlan140"
ifconfig_bridge3_name="vlan150"

# Add ports to respective laggs and create vlan members
ifconfig_lagg40g="laggproto lacp laggport mlxen0 laggport mlxen1"
ifconfig_lagg1g="laggproto lacp laggport igb0 laggport igb1"
vlans_lagg40g="101 110 140 150"
vlans_lagg1g="101 110 140 150"

# Bring up all interfaces
ifconfig_lagg40g_101="up"
ifconfig_lagg40g_110="up"
ifconfig_lagg40g_140="up"
ifconfig_lagg40g_150="up"
ifconfig_lagg1g_101="up"
ifconfig_lagg1g_110="up"
ifconfig_lagg1g_140="up"
ifconfig_lagg1g_150="up"

# Add lagg.vlan members to their respective bridges with STP (RSTP)
ifconfig_vlan101="addm lagg40g.101 stp lagg40g.101 ifpathcost lagg40g.101 500 addm lagg1g.101 stp lagg1g.101 ifpathcost lagg1g.101 20000"
ifconfig_vlan110="addm lagg40g.110 stp lagg40g.110 ifpathcost lagg40g.110 500 addm lagg1g.110 stp lagg1g.110 ifpathcost lagg1g.110 20000"
ifconfig_vlan140="addm lagg40g.140 stp lagg40g.140 ifpathcost lagg40g.140 500 addm lagg1g.140 stp lagg1g.140 ifpathcost lagg1g.140 20000"
ifconfig_vlan150="addm lagg40g.150 stp lagg40g.150 ifpathcost lagg40g.150 500 addm lagg1g.150 stp lagg1g.150 ifpathcost lagg1g.150 20000"

# Host (this) machine addresses
ifconfig_vlan140_alias0="inet 172.23.40.11/24"
ifconfig_vlan140_ipv6="inet6 fd33:58bc:59a0:2340::11/64 accept_rtadv"
defaultrouter="172.23.40.1"

ecorp /etc/rc.conf
Code: 
# 1GigE interfaces
ifconfig_em0="up"
ifconfig_em1="up"

# 10GigE interfaces
ifconfig_mlxen0="up"
ifconfig_mlxen1="up"

# Create member interfaces and rename them as desired
cloned_interfaces="lagg0 lagg1 bridge0 bridge1 bridge2 bridge3"
ifconfig_lagg0_name="lagg10g"
ifconfig_lagg1_name="lagg1g"
ifconfig_bridge0_name="vlan100"
ifconfig_bridge1_name="vlan101"
ifconfig_bridge2_name="vlan140"
ifconfig_bridge3_name="vlan150"

# Add ports to respective laggs and create vlan members
ifconfig_lagg10g="laggproto lacp laggport mlxen0 laggport mlxen1"
ifconfig_lagg1g="laggproto lacp laggport em0 laggport em1"
vlans_lagg10g="100 101 140 150"
vlans_lagg1g="100 101 140 150"

# Bring up all interfaces
ifconfig_lagg10g_100="up"
ifconfig_lagg10g_101="up"
ifconfig_lagg10g_140="up"
ifconfig_lagg10g_150="up"
ifconfig_lagg1g_100="up"
ifconfig_lagg1g_101="up"
ifconfig_lagg1g_140="up"
ifconfig_lagg1g_150="up"

# Add lagg.vlan members to their respective bridges with STP (RSTP)
ifconfig_vlan100="addm lagg10g.100 stp lagg10g.100 ifpathcost lagg10g.100 2000 addm lagg1g.100 stp lagg1g.100 ifpathcost lagg1g.100 20000"
ifconfig_vlan101="addm lagg10g.101 stp lagg10g.101 ifpathcost lagg10g.101 2000 addm lagg1g.101 stp lagg1g.101 ifpathcost lagg1g.101 20000"
ifconfig_vlan140="addm lagg10g.140 stp lagg10g.140 ifpathcost lagg10g.140 2000 addm lagg1g.140 stp lagg1g.140 ifpathcost lagg1g.140 20000"
ifconfig_vlan150="addm lagg10g.150 stp lagg10g.150 ifpathcost lagg10g.150 2000 addm lagg1g.150 stp lagg1g.150 ifpathcost lagg1g.150 20000"

# Host (this) machine addresses
ifconfig_vlan140_alias0="inet 172.23.40.10/24"
ifconfig_vlan140_ipv6="inet6 fd33:58bc:59a0:2340::10/64 accept_rtadv"
defaultrouter="172.23.40.1"

I discovered that because of the LAGG, the interface speed is not carried over from the physical interfaces, hence the cost always defaulting to 2000000. I manually entered costs now which performs as expected (kind of....).

I noticed that sometimes the ptp flag would be set and other times it would not be set despite being manually set in the configurations. By my understanding of the man pages, the ptp iface flag should make RSTP bring the interfaces up faster knowing its a Point-to-Point link and treat it as such versus the autoptp/autoedge detection. I just removed this flag anyway and let the autoptp/autoedge do their thing.

My major problems now.....
1) On my switch, all 3 links on allsafe show as forwarding. Is this correct as only the 40Gig link should be used?
2) The LACP LAGG on ecorp is entirely blocked by my switch. It has the same switch config that it has had for years giving no trouble but now the 10G LAGG is completely not working.
3) Bridge IPv6 SLAAC addressing does not work, and perhaps it's mby design but is there a way to get it working another way? This is rather low priority for now as I have configured a GUA manually in addition to the ULA.
 
You must log in or register to reply here.
Back
Top