multiple ethernet subinterfaces, multiple bridges?

spork

spork

Before going too far with this just want to verify if this is possible in FreeBSD:

- Single ethernet port connected to a trunk port on a managed switch
- 3 VLANs tagged, plus the untagged traffic, so 1 physical interface, 3 vlan (sub)interfaces, 4 bridges
- Bridges to be later shared with bhyve vtnet interfaces so that VMs can, if configured in all VLANs, communicate with all 3 VLANs and the untagged/native VLAN

This is on 15.1 and using what I guess is now the "old" (ie: kind of documented) syntax.

What I'm basically seeing is that if I can bring any ONE VLAN up (or the native/untagged VLAN), things work fine. If I bring additional VLANs up by adding them to their respective bridges, the first bridge stops passing traffic. Observed on a remote host that also sees all 4 VLANs that when things stop working, the remote host still sees traffic FROM the problematic host, with the proper VLAN tag, and it replies back on the right VLAN, but that simply vanishes in the bridge.

I'm thinking that while this feels like a very standard config, perhaps there's some limitation here that I'm not aware of (like the number of bridges and VLANs that can be configured off of one PHYSICAL interface).

Ideas?
 
Hi spork,

This is a classic trap with FreeBSD if_bridge when mixing raw parent physical interfaces and vlan(4) subinterfaces.

What's happening under the hood: when you add the parent physical interface (say em0 or igb0) directly to bridge0 for untagged/native traffic, if_bridge puts the physical NIC into promiscuous mode. In this mode, incoming tagged frames can get swallowed by bridge0's packet filter or MAC learning table before the kernel demuxes them to the respective vlan10, vlan20 interfaces, causing the traffic to silently vanish on return paths.

A couple of ways to fix this:
1. Don't put the raw physical parent interface directly in a bridge if it has VLAN children. Instead, configure a dedicated VLAN for native/untagged traffic on your switch, create vlan1 (or whatever PVID) in FreeBSD, and attach only the vlan interfaces to their respective bridges (vlan1 -> bridge0, vlan10 -> bridge1, etc.).
2. Check your packet filtering sysctls:
sysctl net.link.bridge.pfil_member=0
sysctl net.link.bridge.pfil_bridge=0
sysctl net.link.bridge.pfil_onlyip=0
If you have pf or ipfw active, by default bridge member filtering might be dropping cross-bridge frames.

Which physical NIC driver are you using (igb, ixgbe, em, etc.), and do you have packet filtering (pf/ipfw) enabled in rc.conf?
 
Ugh… so I spent a few days on this and then realized that whatever combo of old/new vlan/bridge/interface config styles I was using in rc.conf were just not working. Not sure why yet, which is common these days for me as I'm trying to dig in and actually see what's been changing in the past few major releases (I do a LOT of work on outdated hosts), and I find it hard to figure out which doc/support resource can actually address things I'm running into most of the time, so I mostly rely on random google results. :)

Anyhow, whatever combo of rc.conf settings I had for bridges, VLANs, and interfaces was not leaving me with enabled subinterfaces. Sure, int0.101, int0.102 and so on *existed*, were added to the right bridges, but in that mass of ifconfig output I missed that the interface didn't have the "UP" tag. No idea why, but rebuilding this from scratch with a remote IP-KVM on a running host where I'd manually deleted all the bridges, vlans and subinterfaces works. I manually created the bridges, the VLANs, added the VLANs to each bridge and it all works. I've already swapped switches in case that was an issue, and also the old clunker has four "bge" interfaces on a card as well as an "em" device (to answer your question on hardware), and I've tried both before getting to this point. I also am not running pf, but I did double check that it was inactive and the same with ipfw.

So now it's just a matter of seeing if a) I actually have the right config syntax after more googling b) if it seems I do, then hopefully enabling the rc debug stuff will give me some hints as to what's going wrong or c) if it seems I don't perhaps I can pitch some doc changes - it's wild to me that neither the vlan(4) nor if_bridge(4) really get around to any of the caveats that are in play when using them in combination.

The other thing that was interesting to me is that even with the vlan subinterfaces "down", outbound traffic still made it out with the proper tags. That bit I think is what really put me on the wrong track - I'd never assume a downed interface would pass traffic. But I'm also probably leaning too hard on my cisco brain (which is where yank the term "subinterface" from, as it really is a nice description of a VLAN interface that's a child to an ethernet interface). I've got a lot of coffee in me, so free cisco config snippet from memory here of setting up a few 802.1q vlans:

Code: 
! "parent" interface and native vlan
interface gigabitethernet 1/1
 ip address 10.0.0.1 255.255.255.0
 
! vlan 101
interface gigabitethernet 1/1.101
 encapsulation dot1q 101
 ip address 10.101.0.1 255.255.255.0
 
! vlan 102
interface gigabitethernet 1/1.102
 encapsulation dot1q 102
 ip address 10.102.0.1 255.255.255.0

! and so on...

I only bring this up because sometimes all this flipping between environments gets me a bit lost and loaded up on bad assumptions. :)
 
spork said:
Ugh… so I spent a few days on this and then realized that whatever combo of old/new vlan/bridge/interface config styles I was using in rc.conf were just not working
Click to expand...
My working "new style" config works in such way:

rc.conf:
Code: 
# example for 3 igc* interfaces and 2 vlans: untagged "lan" and tagged "guest"

cloned_interfaces="bridge0 bridge0.1 bridge0.2"
create_args_bridge0="ether aa:bb:cc:dd:ee:ff"

ifconfig_bridge0="vlanfilter \
          addm igc1 untagged 1 tagged 2 \
          addm igc2 untagged 1 tagged 2 \
          addm igc3 untagged 1 tagged 2 \
          up"

ifconfig_bridge0_1_name="lan"
ifconfig_bridge0_2_name="guest"

ifconfig_lan="inet 192.168.1.1/24"
ifconfig_lan_ipv6="inet6 fd... prefixlen 64"

ifconfig_guest="inet 192.168.2.1/24"
ifconfig_guest_ipv6="inet6 fd... prefixlen 64"
 
Interesting. Is this documented anywhere?

I see you're kind of doing the opposite (I have a trunk port where I want each VLAN to attach to a unique bridge)...
 
spork said:
Interesting. Is this documented anywhere? I see you're kind of doing the opposite (I have a trunk port where I want each VLAN to attach to a unique bridge)...
Click to expand...
spork said:
Interesting. Is this documented anywhere? I see you're kind of doing the opposite (I have a trunk port where I want each VLAN to attach to a unique bridge)...
Click to expand...

https://people.freebsd.org/~ivy/bridge_vlan_filtering.txt

In other posts on different forums/blogs people often mix old-new style and talking about their specific jibs and scripts.
 
You must log in or register to reply here.
Back
Top