VLAN trunk bridging with bhyve / tap and unexpected frame forwarding

nebulo · Jan 8, 2024

Hi all
I've recently got into FreeBSD and use it as fileserver with zfs and hypervisor (bhyve) for nearly all my stuff. I had to physically move the system which spawned the need to change my network configuration to the following.

Cisco switch with 10G interface -> (ix0)* FreeBSD box (re0)* -> 1G interface with TP-Link switch -> several other devices next to the freebsd server

*Both configured as trunks with all vlans I use in my home network. (No native vlan/pvid on ports only tagged vlans)

To implement this I used bridge interfaces in FreeBSD to bridge the two trunks.

Code:

bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 58:9c:fc:10:ff:c8
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: re0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 3 priority 128 path cost 55
    member: ix0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 1 priority 128 path cost 2000
    groups: bridge
    nd6 options=9<PERFORMNUD,IFDISABLED>

ix0: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=4a538bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,NOMAP>
    ether 90:1b:0e:0c:5c:d0
    media: Ethernet autoselect (10Gbase-SR <full-duplex,rxpause,txpause>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    
re0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=8209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
    ether 04:7c:16:4b:7e:e4
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

After settings this up and tried to configure the ip address of the host itself I realized I need vlan interfaces aswell. Forwarding of frames between the two switches just stopped when I was adding vlan 128 so I created the other vlans aswell. Because I want to use some of those vlans for virtual machines on the host I created them as members of bridges and added a tap interface for the first virtual machine. Which leads to the following output of ifconfig

Code:

ix0: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=4a538bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,NOMAP>
    ether 90:1b:0e:0c:5c:d0
    media: Ethernet autoselect (10Gbase-SR <full-duplex,rxpause,txpause>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

re0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=8209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
    ether 04:7c:16:4b:7e:e4
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 58:9c:fc:10:ff:c8
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: re0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 3 priority 128 path cost 55
    member: ix0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 1 priority 128 path cost 2000
    groups: bridge
    nd6 options=9<PERFORMNUD,IFDISABLED>
    
bridge128: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 58:9c:fc:00:3e:6b
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: ix0.128 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 12 priority 128 path cost 2000
    groups: bridge
    nd6 options=9<PERFORMNUD,IFDISABLED>
    
bridge144: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 58:9c:fc:10:ff:e3
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 11 priority 128 path cost 2000000
    member: ix0.144 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 13 priority 128 path cost 2000
    groups: bridge
    nd6 options=9<PERFORMNUD,IFDISABLED>
    
bridge160: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 58:9c:fc:10:ff:b6
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: ix0.160 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 14 priority 128 path cost 2000
    groups: bridge
    nd6 options=9<PERFORMNUD,IFDISABLED>
    
bridge176: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 58:9c:fc:10:ff:e5
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: ix0.176 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 15 priority 128 path cost 2000
    groups: bridge
    nd6 options=9<PERFORMNUD,IFDISABLED>
    
bridge666: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 58:9c:fc:10:ff:d8
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: ix0.666 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 16 priority 128 path cost 2000
    groups: bridge
    nd6 options=9<PERFORMNUD,IFDISABLED>
    
tap0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: vmnet/onyxia/0/custom
    options=80000<LINKSTATE>
    ether 58:9c:fc:10:7a:23
    groups: tap vm-port
    media: Ethernet autoselect
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    Opened by PID 30862
    
ix0.128: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=4200003<RXCSUM,TXCSUM,RXCSUM_IPV6,NOMAP>
    ether 90:1b:0e:0c:5c:d0
    inet 192.168.1.130 netmask 0xfffffff0 broadcast 192.168.1.143
    inet6 fe80::921b:eff:fe0c:5cd0%ix0.128 prefixlen 64 scopeid 0xc
    inet6 <REDACTED> prefixlen 64
    groups: vlan
    vlan: 128 vlanproto: 802.1q vlanpcp: 0 parent interface: ix0
    media: Ethernet autoselect (10Gbase-SR <full-duplex,rxpause,txpause>)
    status: active
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

ix0.144: flags=8942<BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=4200001<RXCSUM,RXCSUM_IPV6,NOMAP>
    ether 90:1b:0e:0c:5c:d0
    groups: vlan
    vlan: 144 vlanproto: 802.1q vlanpcp: 0 parent interface: ix0
    media: Ethernet autoselect (10Gbase-SR <full-duplex,rxpause,txpause>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    
ix0.160: flags=8942<BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=4200003<RXCSUM,TXCSUM,RXCSUM_IPV6,NOMAP>
    ether 90:1b:0e:0c:5c:d0
    groups: vlan
    vlan: 160 vlanproto: 802.1q vlanpcp: 0 parent interface: ix0
    media: Ethernet autoselect (10Gbase-SR <full-duplex,rxpause,txpause>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    
ix0.176: flags=8942<BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=4200003<RXCSUM,TXCSUM,RXCSUM_IPV6,NOMAP>
    ether 90:1b:0e:0c:5c:d0
    groups: vlan
    vlan: 176 vlanproto: 802.1q vlanpcp: 0 parent interface: ix0
    media: Ethernet autoselect (10Gbase-SR <full-duplex,rxpause,txpause>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    
ix0.666: flags=8942<BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=4200003<RXCSUM,TXCSUM,RXCSUM_IPV6,NOMAP>
    ether 90:1b:0e:0c:5c:d0
    groups: vlan
    vlan: 666 vlanproto: 802.1q vlanpcp: 0 parent interface: ix0
    media: Ethernet autoselect (10Gbase-SR <full-duplex,rxpause,txpause>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

When the vm boots it sends a dhcp request which can be seen on the following interfaces
* tap0
* bridge144
* ix0.144
* ix0

I can see the request on my dhcp server. The dhcp server is also sending an answer but the answer is never reaching tap0 / the vm. Instead the frame of the request is showing up on the following interfaces
* ix0
* re0
* bridge0

The answer is not even reaching ix0.144 despite being tagged as vlan with id 144 (verified with tcpdump) and is of course not reaching tap0.

The network card which receives the frame is ix0: <Intel(R) X520 82599ES (SFI/SFP+)>
FreeBSD is version 13.2

rc.conf network part looks like this

Code:

cloned_interfaces="bridge0 bridge128 bridge144 bridge160 bridge176 bridge666 tap0"
ifconfig_bridge0="addm ix0 addm re0 up"
ifconfig_ix0="up"
ifconfig_re0="up"
vlans_ix0="128 144 160 176 666"
ifconfig_ix0_144="up"
ifconfig_ix0_176="up"
ifconfig_bridge128="addm ix0.128 up"
ifconfig_bridge144="addm ix0.144 addm tap0 up"
ifconfig_bridge160="addm ix0.160 up"
ifconfig_bridge176="addm ix0.176 up"
ifconfig_bridge666="addm ix0.666 up"
ifconfig_ix0_128="inet 192.168.1.130/28"
defaultrouter="192.168.1.129"

What am I missing here? I would expect frames with vlan tag arrive at vlan interfaces as with ix0.128?

VladiBG · Jan 8, 2024

On your DHCP server from which MAC address you get the request?

nebulo · Jan 8, 2024

I get the request from '58:9c:fc:03:e8:2c' which is the same MAC address I can see inside the vm.

VladiBG · Jan 8, 2024

Remove the bridge0.

Edit:
maybe you are hitting this bug:

240106 – VNET issue with ARP and routing sockets in jails

bugs.freebsd.org

nebulo · Jan 8, 2024

Yes removing the bridge causes the vm to get it's packages but sadly kills all other devices behind re0.

Edit:
Ah okay. Indeed looks like the same behavior. Thanks for pointing that out!

SirDice · Jan 8, 2024

Why is this FreeBSD host in between the 10G and 1G switch?

nebulo · Jan 8, 2024

At the cisco switch there are several devices some are 10G, some are 1G which read and write large files to the server. Sometimes at the same time. So I need/want the 10G connection from that devices to the FreeBSD host. On the tp-link switch are devices which need to communicate with devices behind the cisco switch (e.g. uplink to internet). Sadly the tp-link switch is not capable of doing 10G. So the FreeBSD host is connecting the two switches without loosing 10G capability.

SirDice · Jan 8, 2024

nebulo said:
So I need/want the 10G connection from that devices to the FreeBSD host.

Yes, that's understandable.

But you can just plug the 1Gbit switch directly in the 10G switch. There's no reason to bridge it with a host.

nebulo said:
Sadly the tp-link switch is not capable of doing 10G.

Sure, but the 10G switch can do 1Gbit.

nebulo said:
So the FreeBSD host is connecting the two switches without loosing 10G capability.

No need for this. You still have the 10G connection, the 1Gbit switch won't suddenly have a 10Gb upstream. It's still limited to 1 Gbit.

nebulo · Jan 8, 2024

Yes I know but sadly I can't because the switches are not in the same room and there is only one fiber connecting the rooms. :-( Which is used for the 10G uplink of the FreeBSD host.

SirDice · Jan 8, 2024

nebulo said:
I would expect frames with vlan tag arrive at vlan interfaces as with ix0.128?

Traffic on ix0.128 is untagged. It's tagged on ix0. Or am I misunderstanding what you mean here?

nebulo · Jan 8, 2024

SirDice said:
Traffic on ix0.128 is untagged. It's tagged on ix0. Or am I misunderstanding what you mean here?

What I mean is, contrary to the frames answering those sent from the tap0 interface the packets receiving at the 10G interface with vlan tag 128 are available on ix0.128 and untagged. (Which is the expected behavior) Packets/frames sent to MAC address of tap0 with vlan tag 144 are not available untagged at ix0.144 only tagged on ix0 Edit: (and re0 and bridge0)

VLAN trunk bridging with bhyve / tap and unexpected frame forwarding

nebulo

VladiBG

nebulo

VladiBG

240106 – VNET issue with ARP and routing sockets in jails

nebulo

SirDice

Administrator

nebulo

SirDice

Administrator

nebulo

SirDice

Administrator

nebulo