multiple ethernet subinterfaces, multiple bridges?

[spork]

Before going too far with this just want to verify if this is possible in FreeBSD:

- Single ethernet port connected to a trunk port on a managed switch
- 3 VLANs tagged, plus the untagged traffic, so 1 physical interface, 3 vlan (sub)interfaces, 4 bridges
- Bridges to be later shared with bhyve vtnet interfaces so that VMs can, if configured in all VLANs, communicate with all 3 VLANs and the untagged/native VLAN

This is on 15.1 and using what I guess is now the "old" (ie: kind of documented) syntax.

What I'm basically seeing is that if I can bring any ONE VLAN up (or the native/untagged VLAN), things work fine. If I bring additional VLANs up by adding them to their respective bridges, the first bridge stops passing traffic. Observed on a remote host that also sees all 4 VLANs that when things stop working, the remote host still sees traffic FROM the problematic host, with the proper VLAN tag, and it replies back on the right VLAN, but that simply vanishes in the bridge.

I'm thinking that while this feels like a very standard config, perhaps there's some limitation here that I'm not aware of (like the number of bridges and VLANs that can be configured off of one PHYSICAL interface).

Ideas?

 

[ali-tek]

Hi spork,

This is a classic trap with FreeBSD if_bridge when mixing raw parent physical interfaces and vlan(4) subinterfaces.

What's happening under the hood: when you add the parent physical interface (say em0 or igb0) directly to bridge0 for untagged/native traffic, if_bridge puts the physical NIC into promiscuous mode. In this mode, incoming tagged frames can get swallowed by bridge0's packet filter or MAC learning table before the kernel demuxes them to the respective vlan10, vlan20 interfaces, causing the traffic to silently vanish on return paths.

A couple of ways to fix this:
1. Don't put the raw physical parent interface directly in a bridge if it has VLAN children. Instead, configure a dedicated VLAN for native/untagged traffic on your switch, create vlan1 (or whatever PVID) in FreeBSD, and attach only the vlan interfaces to their respective bridges (vlan1 -> bridge0, vlan10 -> bridge1, etc.).
2. Check your packet filtering sysctls:
sysctl net.link.bridge.pfil_member=0
sysctl net.link.bridge.pfil_bridge=0
sysctl net.link.bridge.pfil_onlyip=0
If you have pf or ipfw active, by default bridge member filtering might be dropping cross-bridge frames.

Which physical NIC driver are you using (igb, ixgbe, em, etc.), and do you have packet filtering (pf/ipfw) enabled in rc.conf?