So I have a few VMs via churchers vm-bhyve, and trying to put a new VM on with a VLAN. The Unifi switch port is configured as a trunk port. So starting out, here's the interfaces I have.
Everything works just fine. So I then create the bridge (virtual-switch in vm-bhyve parlance), for the VLAN, and boot up the VM. I end up with these interfaces.
After the VM finishes starting up, nothing, the VM can't get a DHCP from the VLAN or ping anything. Doing tcpdump on both the router I see the DHCP request and response on the correct VLAN (30), and on the host igb.30, I see the DHCP request, but no response.
From https://github.com/churchers/vm-bhyve/issues/390#issuecomment-785537662 someone else mentions when the bridge on the untagged interface, suddenly the bridge on the VLAN interface starts passing traffic fine. I try the same, shut down the VMs on the public (untagged) bridge, still nothing, and then I destroy the bridge. Suddenly the DHCP works fine, as well as pinging. I then create the bridge on the untagged interface again, and neither ping nor DHCP works correctly anymore.
I then destroy the untagged bridge, and it's back to working again.
I then try and create another VLAN bridge (id 40), and while the bridge on VLAN 30 VM still works, another VM up on the VLAN 40 bridge, and both VLAN bridges (30 and 40) work just fine, no problems with DHCP or ping.
If I create the untagged bridge again, both VMs on VLAN 30 and 40 stop working.
It's as if the untagged bridge swallows all traffic intended for the VLAN bridges or interfaces.
I'm running v12.2-RELEASE.
Code:
igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=a520b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6>
ether d0:50:99:dd:49:c9
inet 192.168.2.5 netmask 0xffffff00 broadcast 192.168.2.255
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
vm-public: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 92:35:c4:3c:e8:d0
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: tap1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 10 priority 128 path cost 2000000
member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 9 priority 128 path cost 2000000
member: igb0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 1 priority 128 path cost 20000
groups: bridge vm-switch viid-4c918@
nd6 options=1<PERFORMNUD>
tap0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: vmnet-pihole-0-public
options=80000<LINKSTATE>
ether 58:9c:fc:10:4a:1a
groups: tap vm-port
media: Ethernet autoselect
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
Opened by PID 83050
tap1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: vmnet-unifi-controller-0-public
options=80000<LINKSTATE>
ether 58:9c:fc:10:ff:ba
inet6 fe80::5a9c:fcff:fe10:ffba%tap2 prefixlen 64 tentative scopeid 0xa
groups: tap vm-port
media: Ethernet autoselect
status: no carrier
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>Everything works just fine. So I then create the bridge (virtual-switch in vm-bhyve parlance), for the VLAN, and boot up the VM. I end up with these interfaces.
Code:
igb0.30: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: vm-vlan-iot-igb0.30
options=200001<RXCSUM,RXCSUM_IPV6>
ether d0:50:99:dd:49:c9
inet6 fe80::d250:99ff:fedd:49c9%igb0.30 prefixlen 64 scopeid 0x7
groups: vlan vm-vlan viid-45833@
vlan: 30 vlanpcp: 0 parent interface: igb0
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
vm-iot: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 2a:2d:aa:58:d8:1b
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: tap2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 8 priority 128 path cost 2000000
member: igb0.30 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 7 priority 128 path cost 55
groups: bridge vm-switch viid-97c20@
nd6 options=1<PERFORMNUD>
tap2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: vmnet-home-assistant-0-iot
options=80000<LINKSTATE>
ether 58:9c:fc:10:ff:b1
inet6 fe80::5a9c:fcff:fe10:ffb1%tap0 prefixlen 64 tentative scopeid 0x8
groups: tap vm-port
media: Ethernet autoselect
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
Opened by PID 4573After the VM finishes starting up, nothing, the VM can't get a DHCP from the VLAN or ping anything. Doing tcpdump on both the router I see the DHCP request and response on the correct VLAN (30), and on the host igb.30, I see the DHCP request, but no response.
From https://github.com/churchers/vm-bhyve/issues/390#issuecomment-785537662 someone else mentions when the bridge on the untagged interface, suddenly the bridge on the VLAN interface starts passing traffic fine. I try the same, shut down the VMs on the public (untagged) bridge, still nothing, and then I destroy the bridge. Suddenly the DHCP works fine, as well as pinging. I then create the bridge on the untagged interface again, and neither ping nor DHCP works correctly anymore.
I then destroy the untagged bridge, and it's back to working again.
I then try and create another VLAN bridge (id 40), and while the bridge on VLAN 30 VM still works, another VM up on the VLAN 40 bridge, and both VLAN bridges (30 and 40) work just fine, no problems with DHCP or ping.
If I create the untagged bridge again, both VMs on VLAN 30 and 40 stop working.
It's as if the untagged bridge swallows all traffic intended for the VLAN bridges or interfaces.
I'm running v12.2-RELEASE.