bhyve vm-bhyve and VLAN

I have read a couple of threads where the poster had seemingly the same problem as I had, but their solutions either didn't work, or I had already tried it. I'm hoping there is something I missed and that I've made a simple mistake.

I have a FreeBSD 13.1 server with its main Ethernet interface connected to igc0. I then have two VLAN's for separate networks. Both of these have the same problem, but for the sake of this discussion I'm using the one called "ext" which is outside the firewall (it's my DMZ network).

I'm using both IPv4 and IPv6. For all my tests, the behaviour are the same regardless of whether I'm doing tests via IPv4 or IPv6. I'm not really considering my IPv6 network secret, but I'm still hiding it in this post in order to reduce potential spam and network probes. In my logs I have replaced the network name by NNNN:NNNN:NNNN.

Here's the output of ifconfig -a for igc0 and ext:

Code:
igc0: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4a520b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,NOMAP>
        ether 50:eb:f6:50:63:09
        inet 192.168.1.11 netmask 0xffffff00 broadcast 192.168.1.255
        inet6 fe80::52eb:f6ff:fe50:6309%igc0 prefixlen 64 scopeid 0x1
        inet6 NNNN:NNNN:NNNN::11 prefixlen 64
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ext: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4200001<RXCSUM,RXCSUM_IPV6,NOMAP>
        ether 50:eb:f6:50:63:09
        groups: vlan
        vlan: 100 vlanproto: 802.1q vlanpcp: 0 parent interface: igc0
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

Note that ext does not have an IP address assigned. This is intentional as I don't want to expose my server to the external network. However, I have tried to temporarily assign an address to the interface and confirmed that everything worked (I can ping my the router's IP on the DMZ network).

With this configuration, I originally used Virtualbox to create an Ubuntu VM with the network assigned to ext. Once I did this, everything worked and I was able to get both IPv4 and IPv6 to work on the DMZ network.

Because of a lot of problems with Virtualbox I decided to move to bhyve, and I've used vm-byhve to set everything up. I used the ubuntu cloud image, but I also tried to run the Ubuntu installer and I couldn't find any network in the installer.

I created the network as follows:

Code:
vm switch create -i ext dmz

This is the output from vm switch info dmz:

Code:
local: _netgraph,: bad variable name
------------------------
Virtual Switch: dmz
------------------------
  type: standard
  ident: vm-dmz
  vlan: -
  physical-ports: ext
  bytes-in: 0 (0.000B)
  bytes-out: 0 (0.000B)

I then use vm configure to set the network0_switch to dmz.

When I start the vm, I can see the following interfaces:

Code:
vm-dmz: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 46:25:2c:87:aa:42
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: tap1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 9 priority 128 path cost 2000000
        member: ext flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 3 priority 128 path cost 20000
        groups: bridge vm-switch viid-6ffdb@
        nd6 options=9<PERFORMNUD,IFDISABLED>
tap1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: vmnet/ext/0/dmz
        options=80000<LINKSTATE>
        ether 58:9c:fc:00:78:4b
        groups: tap vm-port
        media: Ethernet autoselect
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        Opened by PID 17379

As best as I can tell, this should be enough to get everything to work. However, after I boot, and configure the IP addresses correctly (as best as I can tell exactly the same as I did on the VM I was running in Virtualbox) I simply cannot get any connectivity at all. I can ping my own IP address, but I can't even reach the router. It's as if nothing is connected to the interface at all.

I have even tried to configure a bridge to configure everything manually, but the result was the same.
 
Did you pass the vlans on the switch port? It needs to be trunked (i.e. pass vlan information).

Code:
# show config interface Trk1

Startup configuration: 38

interface Trk1
   tagged vlan 10-11
   spanning-tree priority 4
   exit
Trk1 is an LACP of two switch ports. Note that I'm passing tagged vlan information on that port.
 
Did you pass the vlans on the switch port? It needs to be trunked (i.e. pass vlan information).

Code:
# show config interface Trk1

Startup configuration: 38

interface Trk1
   tagged vlan 10-11
   spanning-tree priority 4
   exit
Trk1 is an LACP of two switch ports. Note that I'm passing tagged vlan information on that port.
I don't think so. I mean, I only let vm-bhyve configure the interface with the default settings.

I wasn't aware that the bridge needed to know about the vlans, since ext ext presents the vlan interface itself.

In any case, I do intend to test what you suggest, but your example isn't clear to me. What is the 'show' command? Also, why do I need to use link aggregation? There is only one link.
 
I wasn't aware that the bridge needed to know about the vlans, since ext ext presents the vlan interface itself.
Bridge doesn't need to know. It gets the untagged traffic there, which is fine. But your ext vlan(4) interface encapsulates the traffic with a VLAN header before sending it out igc0. Your switch needs to be aware of this.

What is the 'show' command?
It was the command line of my HP Procurve switch. An example of how my switch is configured for VLAN traffic.

Also, why do I need to use link aggregation?
You don't. I do.
Code:
dice@hosaka:~ % ifconfig lagg0
lagg0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
        options=48120b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,NOMAP>
        ether 00:25:90:f1:58:39
        laggproto lacp lagghash l2,l3,l4
        laggport: igb1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
        laggport: igb2 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
        groups: lagg
        media: Ethernet autoselect
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

dice@hosaka:~ % ifconfig lagg0.11
lagg0.11: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
        description: vm-vlan/servers/lagg0.11
        options=4000000<NOMAP>
        ether 00:25:90:f1:58:39
        groups: vlan vm-vlan viid-8bf4d@
        vlan: 11 vlanproto: 802.1q vlanpcp: 0 parent interface: lagg0
        media: Ethernet autoselect
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

dice@hosaka:~ % ifconfig vm-servers
vm-servers: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
        ether 12:74:b5:6c:fc:4d
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: tap12 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 23 priority 128 path cost 2000000
        member: tap10 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 21 priority 128 path cost 2000000
        member: tap9 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 20 priority 128 path cost 2000000
        member: tap8 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 19 priority 128 path cost 2000000
        member: tap3 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 14 priority 128 path cost 2000000
        member: tap2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 13 priority 128 path cost 2000000
        member: tap1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 12 priority 128 path cost 2000000
        member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 11 priority 128 path cost 2000000
        member: lagg0.11 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 8 priority 128 path cost 2000000
        groups: bridge vm-switch viid-d5539@
        nd6 options=9<PERFORMNUD,IFDISABLED>

root@hosaka:~ # vm switch list
NAME     TYPE      IFACE       ADDRESS  PRIVATE  MTU   VLAN  PORTS
servers  standard  vm-servers  -        no       9000  11    lagg0
public   standard  vm-public   -        no       9000  10    lagg0
 
Thank you. When you say switch you're referring to the switch to which the FreeBSD server is connected.

Yes, this switch (in my case it's unifi) is configured like that. I know it works, because as I mentioned it works when I use the ext interface from FreeBSD itself, and also when I connect it to a vm using Virtualbox.

The problem I'm seeing is only happening from bhyve VM's.
 
An update: I decided to try with a FreeBSD guest as well, and I have the same problem there.

From this I draw the conclusion that the problem must be in the vm-bhyve switch configuration, since the actual ext interface clearly works (tested natively as well as via virtualbox) and the problem can be seen with two different guest OS'es.
 
Here's the output:

Code:
# vm switch list       
NAME    TYPE      IFACE      ADDRESS  PRIVATE  MTU  VLAN  PORTS
public  standard  vm-public  -        no       -    -     igc0
dmz     standard  vm-dmz     -        no       -    -     ext
 
In my case on the freebsd vm-guest.
Code:
dhclient vtnet0
Show only trying to discover...
Code:
DHCPDISCOVER on vtnet0 to 255.255.255.255 port 67 interval 3
I stopped ipfw but no change.
 
My test results are conclusive. "vm switch" is buggy. So i must use something else then the churchers.
When i launch the virtual-machine the IP-interface goes down till next reboot.
 
My test results are conclusive. "vm switch" is buggy. So i must use something else then the churchers.
When i launch the virtual-machine the IP-interface goes down till next reboot.
Which interface? The VLAN one (in my examples, the ext interface)? Because I was able to use that interface from the host by just adding an IP address to it. So clearly the interface itself works.
 
You could run
Code:
netstat -rn
Code:
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            192.168.3.2        UGS      vtnet0
127.0.0.1          link#2             UH          lo0
192.168.3.0/24     link#1             U        vtnet0
192.168.3.31       link#1             UHS         lo0

Internet6:
Destination                       Gateway                       Flags     Netif Expire
::/96                             ::1                           UGRS        lo0
default                           NNNN:NNNN:NNNN:2::1           UGS      vtnet0
::1                               link#2                        UHS         lo0
::ffff:0.0.0.0/96                 ::1                           UGRS        lo0
NNNN:NNNN:NNNN:2::/64             link#1                        U        vtnet0
NNNN:NNNN:NNNN:2::30              link#1                        UHS         lo0
fe80::/10                         ::1                           UGRS        lo0
fe80::%vtnet0/64                  link#1                        U        vtnet0
fe80::5a9c:fcff:fe08:52c2%vtnet0  link#1                        UHS         lo0
fe80::%lo0/64                     link#2                        U           lo0
fe80::1%lo0                       link#2                        UHS         lo0
ff02::/16                         ::1                           UGRS        lo0
On the host and the guest to verify IP-routing is ok.
And test with disabling the firewall.
On the host:
Code:
# netstat -rn
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            192.168.1.2        UGS        igc0
127.0.0.1          link#2             UH          lo0
192.168.1.0/24     link#1             U          igc0
192.168.1.11       link#1             UHS         lo0

Internet6:
Destination                       Gateway                       Flags     Netif Expire
::/96                             ::1                           UGRS        lo0
default                           NNNN:NNNN:NNNN::1             UGS        igc0
::1                               link#2                        UHS         lo0
::ffff:0.0.0.0/96                 ::1                           UGRS        lo0
NNNN:NNNN:NNNN::/64               link#1                        U          igc0
NNNN:NNNN:NNNN::11                link#1                        UHS         lo0
fe80::/10                         ::1                           UGRS        lo0
fe80::%igc0/64                    link#1                        U          igc0
fe80::52eb:f6ff:fe50:6309%igc0    link#1                        UHS         lo0
fe80::%lo0/64                     link#2                        U           lo0
fe80::1%lo0                       link#2                        UHS         lo0
ff02::/16                         ::1                           UGRS        lo0


On the guest:
Code:
# netstat-rn
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            192.168.3.2        UGS      vtnet0
127.0.0.1          link#2             UH          lo0
192.168.3.0/24     link#1             U        vtnet0
192.168.3.31       link#1             UHS         lo0

Internet6:
Destination                       Gateway                       Flags     Netif Expire
::/96                             ::1                           UGRS        lo0
default                           NNNN:NNNN:NNNN:2::1           UGS      vtnet0
::1                               link#2                        UHS         lo0
::ffff:0.0.0.0/96                 ::1                           UGRS        lo0
NNNN:NNNN:NNNN:2::/64             link#1                        U        vtnet0
NNNN:NNNN:NNNN:2::30              link#1                        UHS         lo0
fe80::/10                         ::1                           UGRS        lo0
fe80::%vtnet0/64                  link#1                        U        vtnet0
fe80::5a9c:fcff:fe08:52c2%vtnet0  link#1                        UHS         lo0
fe80::%lo0/64                     link#2                        U           lo0
fe80::1%lo0                       link#2                        UHS         lo0
ff02::/16                         ::1                           UGRS        lo0

# ifconfig -a
vtnet0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
        ether 58:9c:fc:08:52:c2
        inet 192.168.3.31 netmask 0xffffff00 broadcast 192.168.3.255
        inet6 fe80::5a9c:fcff:fe08:52c2%vtnet0 prefixlen 64 scopeid 0x1
        inet6 NNNN:NNNN:NNNN:2::30 prefixlen 64
        media: Ethernet autoselect (10Gbase-T <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

On the guest, this is what I see when I try to ping the router:
Code:
# ping 192.168.3.2
PING 192.168.3.2 (192.168.3.2): 56 data bytes
ping: sendto: Host is down
ping: sendto: Host is down
 
Also had trouble with setting the VLANs in VM-Bhyve.
For some reason you cannot combine DHCP for the VMs and VLANs the same host network interface.
If you want the VMs to use VLANs you have to add a second interface to the Bhyve host. That solved it for me.
 
Odd, as I have several VMs doing DHCP just fine. And I have different VLANs. But the VLANs are split on the host, not on the VMs themselves.

vm switch isn't buggy. All it does is automatically create the correct vlan(4) and bridge(4) interfaces. sysutils/vm-bhyve is "just" a collection of shell scripts that invoke the same commands you would use when setting it up by hand.
 
I'll make it more concrete. When I spin-up a Virtual-machine-FreebSD-Guest, the Interface on the host goes down.
So it is certainly buggy on my computer. Offcourse other configurations, other persons might not see the this specific problem i see.
Maybe a quantum-bug. It's for me there but not for other observers.
 
Odd, as I have several VMs doing DHCP just fine. And I have different VLANs. But the VLANs are split on the host, not on the VMs themselves.

vm switch isn't buggy. All it does is automatically create the correct vlan(4) and bridge(4) interfaces. sysutils/vm-bhyve is "just" a collection of shell scripts that invoke the same commands you would use when setting it up by hand.
This sounds similar to what I'm trying to do. Can you tell if there is something significantly different that I'm doing compared to your configuration?
 
I decided to use tcpdump to see if the packaget actually go anywhere, and I discovered that indeed they reach the destination, however, the reply is never relayed back to the VM.

If I log in to the router itself (which has IP address 192.168.3.2) and do a tcpdump on it while trying to ping the router from the VM what I'm having problems with I see the following:

On the VM:

Code:
# ping 192.168.3.2
PING 192.168.3.2 (192.168.3.2): 56 data bytes
ping: sendto: Host is down
ping: sendto: Host is down
ping: sendto: Host is down
ping: sendto: Host is down

On the router:

Code:
$ sudo tcpdump -i eth1.100
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1.100, link-type EN10MB (Ethernet), capture size 262144 bytes
I can also confirm that pfil is disabled for the bridge:




[CODE]# sysctl net.link.bridge   
net.link.bridge.ipfw: 0
net.link.bridge.allow_llz_overlap: 0
net.link.bridge.inherit_mac: 0
net.link.bridge.log_stp: 0
net.link.bridge.pfil_local_phys: 0
net.link.bridge.pfil_member: 0
net.link.bridge.ipfw_arp: 0
net.link.bridge.pfil_bridge: 0
net.link.bridge.pfil_onlyip: 1
17:34:03.055594 ARP, Request who-has 192.168.3.2 tell 192.168.3.31, length 42
17:34:03.055700 ARP, Reply 192.168.3.2 is-at dc:9f:db:16:63:c6 (oui Unknown), length 28
17:34:04.076961 ARP, Request who-has 192.168.3.2 tell 192.168.3.31, length 42
17:34:04.077066 ARP, Reply 192.168.3.2 is-at dc:9f:db:16:63:c6 (oui Unknown), length 28
17:34:05.154645 ARP, Request who-has 192.168.3.2 tell 192.168.3.31, length 42
17:34:05.154751 ARP, Reply 192.168.3.2 is-at dc:9f:db:16:63:c6 (oui Unknown), length 28
17:34:06.234428 ARP, Request who-has 192.168.3.2 tell 192.168.3.31, length 42
17:34:06.234529 ARP, Reply 192.168.3.2 is-at dc:9f:db:16:63:c6 (oui Unknown), length 28[/CODE]

As we can see, the ARP request arrives on the router, and a reply is sent.

However, if I run tcpdump on the freebsd host, listening on the ext interface, I don't see the reply:

Code:
# tcpdump -i ext
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ext, link-type EN10MB (Ethernet), capture size 262144 bytes
17:35:24.615078 ARP, Request who-has 192.168.3.2 tell 192.168.3.31, length 28
17:35:25.670845 ARP, Request who-has 192.168.3.2 tell 192.168.3.31, length 28
17:35:26.751709 ARP, Request who-has 192.168.3.2 tell 192.168.3.31, length 28

What's really strange about this is that the ext interface works fine when I use it directly, so the interface itself should be configured correctly. What is weird is that I don't see the reply when the request comes from a bhyve vm. I really don't know where I should continue investigating.

I also tried to manually add the ARP entry to the VM, and when I do I can see the ping request on the router, as well as the reply. However, the reply is not seen on the ext interface. In other words, I don't see any replies to the ext interface.

The above does sound like there is some filtering on the interfaces, but it's disabled on the bridge:

Code:
# sysctl net.link.bridge    
net.link.bridge.ipfw: 0
net.link.bridge.allow_llz_overlap: 0
net.link.bridge.inherit_mac: 0
net.link.bridge.log_stp: 0
net.link.bridge.pfil_local_phys: 0
net.link.bridge.pfil_member: 0
net.link.bridge.ipfw_arp: 0
net.link.bridge.pfil_bridge: 0
net.link.bridge.pfil_onlyip: 1
 
Some more information. I can see the problem even without any VM's. If I set an IP address to the ext interface, then I can ping the router. However, if I set an IP address on the bridge interface, then I can no longer poing it, and the behaviour is identical as when I try to ping from the VM. In other words: I can see the packet going out, but the resonse is not forwarded to the bridge.
 
You can not use vm-bhyve and do everything bare.
Thank you for your comment, and yes, I've tried this, with the same results. In its simplest form, the issue can be summarised as follows:

Create bridge0 that contains the main interface (igc0)
Create a vlan interface (vlan0) with vlan 100 and vlandev igc0.
Create bridge1 that contains vlan0
Set an IP address on bridge0 on the vlan 100 network

At this point, traffic from bridge0 reaches its destination, but no replies reach vlan0.

I've learned a lot about FreeBSD networking during this experimentation, and I'm starting to suspect there is a bug somewhere that prevents this from working. I believe this not simply because I have failed to get this to work, but because I've seen similar reports from other people and none of them seem to have been about to actually solve the issue.
 
I had installed Alpine Linux in VirtualBox two months ago. It worked perfectly and always had good network connectivity.

Today I realized I forgot my Alpine Linux password so I couldn't login anymore. I therefore wanted to reinstall Alpine in VirtualBox, but I am not getting network connectivity. I've tried all possible settings, I can't get connectivity. When I select Bridged it simply always crashes directly during startup, although in my memory this was my old setting.

I think the problem may not be limited to bhyve but also applies to VirtualBox. Or can you get network connectivity in the standard version of Alpine Linux currently?

Edit
I didn't have vboxnet_enable="YES" in rc.conf
This was the issue..
VirtualBox works well again for me:
Screenshot_2022-10-23_11-17-38.png
 
Back
Top