bhyve and VLANs

T

tcn

Hi,

I am struggling with VLANs when using bhyve.

I want to achieve a simple vlanned network where my virtual machine will communicate only through this VLAN; I do not want it to get out of it. Simple enough I thought....

On the bhyve server, the main interface (ix0) is already part of the public bridge.

I have setup my switches and router to accept the given VLAN. When I create an inteface (ix0.1) on the bhyve server, traffic comes and goes as it should; this validates my network path.

Now I try to connect my virtual machine to the VLAN. I simply create an isolated bridge and put the ix0.1 interface in it.

What I have tried:
  • Add a tap0 interface to the bridge.
    • This makes the virtual machine see the interface and uses it. ARP requests reach destination; replies are sent back but never received by the interface. (tcpdump does not see the ARP replies at destination)
  • Add a vlanned tap0.1 interface to the bridge.
    • This is tricky as the virtual interface's MTU needs to be increased to 1504 in order to have tap0.1 keep the same MTU as the uplink interface. Creating a VLAN on a virtual adapter drops the MTU to 1496 and thus the new interface cannot be added to the bridge.
    • After doing so, the virtual machine cannot see the interface as connected. It is a Windows machine and gives an unconnected status to the interface.
  • I can't assign a VLAN to a bridge interface. I get a protocol unsupported error.
I am out of ideas from here. I have tried brigning down the MTU to 1496 on the uplink ix0.1 and get the bridge down to 1496 as well but I get the same results.

Can someone guide me into achieving my goal?


Regards,
tcn
 
MTU doesn't change if you add a VLAN tag to a packet, the frame gets bigger (1518->1522).

Code: 
dice@maelcum:~ % ifconfig em0
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 6000
        options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
        ether 90:e2:ba:53:fd:2f
        hwaddr 90:e2:ba:53:fd:2f
        inet6 fe80::92e2:baff:fe53:fd2f%em0 prefixlen 64 scopeid 0x2
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
dice@maelcum:~ % ifconfig vlan10
vlan10: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 6000
        options=3<RXCSUM,TXCSUM>
        ether 90:e2:ba:53:fd:2f
        inet 192.168.10.1 netmask 0xffffff00 broadcast 192.168.10.255
        inet6 fe80::92e2:baff:fe53:fd2f%vlan10 prefixlen 64 scopeid 0x7
        inet6 2001:470:1f15:bcd::1 prefixlen 64
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        vlan: 10 vlanpcp: 0 parent interface: em0
        groups: vlan
dice@maelcum:~ % ifconfig vlan20
vlan20: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 6000
        options=3<RXCSUM,TXCSUM>
        ether 90:e2:ba:53:fd:2f
        inet 10.0.1.1 netmask 0xffffff00 broadcast 10.0.1.255
        inet6 fe80::92e2:baff:fe53:fd2f%vlan20 prefixlen 64 scopeid 0x8
        inet6 2001:470:7989:20::1 prefixlen 64
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        vlan: 20 vlanpcp: 0 parent interface: em0
        groups: vlan
vlan1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 6000
        options=3<RXCSUM,TXCSUM>
        ether 90:e2:ba:53:fd:2f
        inet 10.0.0.1 netmask 0xffffff00 broadcast 10.0.0.255
        inet6 fe80::92e2:baff:fe53:fd2f%vlan1 prefixlen 64 scopeid 0x6
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        vlan: 1 vlanpcp: 0 parent interface: em0
        groups: vlan
Note how all VLAN interfaces have the same MTU as the "trunked" em0 interface.
 
Your VLANs are on a physical interface; try it on a virtual interface.

Code: 
tap100: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80000<LINKSTATE>
        ether 00:bd:47:7a:bd:64
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: no carrier
        groups: tap
tap100.1: flags=8842<BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1496
        ether 00:bd:47:7a:bd:64
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: no carrier
        vlan: 1 vlanpcp: 0 parent interface: tap100
        groups: vlan
 
It looks like you have some extra tap(4) interfaces that really aren't needed. What I think is happening here is that ix0 is 'trunked' and ix0.1 is the VLAN inside that trunk. If you bridge tap0 and ix0.1 traffic on tap0 is already tagged through ix0.1. Adding another VLAN tag as tap0.1 would basically "double-tag" the packet. Once for tap0.1->tap0 and again on xi0.1->xi0.

This is probably much easier to set up if you use sysutils/vm-bhyve: https://github.com/churchers/vm-bhyve/wiki/Virtual-Switches#user-content-assigning-a-vlan-number
 
You are right but I had to try as I am running out of options.... The first bullet describes what you are saying. And the link you sent me doesn't work for the same reason I expressed earlier with the MTU issue.

I think I am starting to undertand what is happening. I think since the only interface that is exposed to the network is ix0.11, traffic does not get to the bridge interface since the IP of the uplink isn't the IP of the virtual machine. The problem isn't the VLAN but rather an issue in routing.

Although I am puzzled as the public interface works very well.....
 
Have you tried tying the whole trunk to the VM and creating the VLAN interface inside the VM itself?

I'm thinking about two solutions which should work (can't test it at the moment though):
  • trunk interface bridged to the VM directly, VLAN tagging inside the VM (vnet0.1 for example).
    ix0 <-bridge-> tap0[VM: vnet0.1]
  • create a VLAN interface on the host and bridge the untagged traffic to the VM
    ix0->ix0.1 <-bridge-> tap0
 
Not sure I am following.
The first option is not possible, Windows virtual machine cannot have a VLAN; it is not available in the advanced options of the interface I'm afraid.
The second option looks like my first bullet which I already tried....
 
tcn said:
The first option is not possible, Windows virtual machine cannot have a VLAN; it is not available in the advanced options of the interface I'm afraid.
Click to expand...
Ah, I'm not sure what the Windows driver allows or not. I typically have Intel cards in my Windows machines and those do allow VLAN tagging/untagging. It's definitely possible the virtualized interface on Windows doesn't allow it.
tcn said:
The second option looks like my first bullet which I already tried....
Click to expand...
Correct. That would be the right way to set it up though, I'm wondering why it's not working for you. I'm unable to test it at the moment, hopefully I'll be able to try a few things when I get home tonight. I have various bhyve VMs and I can create VLANs but I've never actually tied a VM to a VLAN through a trunk (my VMs are all tied to an untagged interface).
 
Still haven't found out why. I just can't figure out why ARP replies won't reach the VM when using a VLAN.....
Outside of the VM, the VLAN works great......
 
Got a little further.
From the switch directly attached to the port, I tried to ping. The ARP request and reply does happen but after this, the switch tries to communicate with the MAC address of the VM and this is where it fails. The traffic does not reach my VLAN interface at all (according to TCP dump).
The host does not recognize the MAC address of the VM. Furthermore, the VM can ping the host's interface within the bridge but the host can't ping the VM...
I tried giving the bridge the MAC of the physical interface but whitout success.
I am really puzzled as of how MACs are being recognized on a VLAN vs a non VLAN interface....
 
The only way I've been able to get the traffic to go is to give the VM the hardware address of my physical interface.
I do not think that the bridge is doing its work when VLAN is in use (i.e. broken). I have seen some thread from DragonFly developers who had to modify the bridge driver.....
 
You must log in or register to reply here.
Back
Top