Windows Virtual machines bridged on a trunk VLAN interface

T

tcn

Hi,
I have been struggling with this issue for a while. I am using vm-bhyve.
I have a few Windows VMs that are bridged and communicate with the exterior world on a VLAN through a trunk port. The only way I have been able to do this is by doing network address translation which kills performance.
Normally, IPerf gives me about 3Gbps (if I perform the test using the bridged interface) but with the NAT, it drops down to 100Mbps.

Has anyone been able to bridge Windows VMs with a VLAN?
I am I did try to put the VLAN on the switch (using vm switch vlan vnet 5. This adds VLANs to the TAP interfaces. The Windows machine will grab the interface but the interface never comes up (no carrier).
I can't simply bridge the physical vlanned network interface because the bridge will not use the VLAN at all. This causes weird issues like ARP messages being sent fine but received not on the VLAN (seen in my post back in 2017 (bhyve and VLANs).

I wish to get more performance on the interfaces because I really need the bandwidth now...... If I could put the Windows VMs on the VLAN directly without NAT, I think performance would be much better.....

tcn
 
tcn said:
Has anyone been able to bridge Windows VMs with a VLAN?
Click to expand...
No Windows VMs but I do use bridge(4), vlan(4) and lagg(4):
Code: 
root@hosaka:~ # vm switch list
NAME     TYPE      IFACE       ADDRESS  PRIVATE  MTU   VLAN  PORTS
servers  standard  vm-servers  -        no       9000  11    lagg0
public   standard  vm-public   -        no       9000  10    lagg0
Code: 
root@hosaka:~ # ifconfig vm-servers
vm-servers: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
        ether ee:1f:04:ce:b3:ee
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: tap7 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 18 priority 128 path cost 2000000
        member: tap2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 13 priority 128 path cost 2000000
        member: tap1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 12 priority 128 path cost 2000000
        member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 11 priority 128 path cost 2000000
        member: lagg0.11 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 8 priority 128 path cost 2000000
        groups: bridge vm-switch viid-d5539@
        nd6 options=1<PERFORMNUD>
Code: 
root@hosaka:~ # ifconfig lagg0.11
lagg0.11: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
        description: vm-vlan-servers-lagg0.11
        ether 00:25:90:f1:58:39
        inet6 fe80::225:90ff:fef1:5839%lagg0.11 prefixlen 64 scopeid 0x8
        groups: vlan vm-vlan viid-8bf4d@
        vlan: 11 vlanpcp: 0 parent interface: lagg0
        media: Ethernet autoselect
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
All I can suggest is that you don't "mix" things on the physical interface. Use a separate interface to bind your VMs to and do administration through a different interface.
 
This does work if your VM can put traffic on a VLAN. Windows Virtual NIC driver do not support VLAN (at least, not the version I am using).
The host must insert the tags (best case senario, the physical interface should do it).
 
tcn said:
Windows Virtual NIC driver do not support VLAN
Click to expand...
It doesn't need to.

tcn said:
The host must insert the tags (best case senario, the physical interface should do it).
Click to expand...
The host does the tagging through vlan(4). In my case it's pass(4) -> vlan(4) -> bridge(4) -> tap(4) (tied to the VM). Which is what happens when you set a VLAN on the vm(8) switch. The VM gets the untagged traffic so there's no need to set the VLAN in the VM itself.
 
tcn said:
Has anyone been able to bridge Windows VMs with a VLAN?
Click to expand...
We are using this exact configuration at work. Two Windows Server VMs in bhyve, with VLANs on top of lagg.

However, my config looks very different from SirDice's:
Code: 
# vm switch list
NAME   TYPE      IFACE     ADDRESS  PRIVATE  MTU  VLAN  PORTS
voice  standard  vm-voice  -        no       -    -     vlan224
data   standard  vm-data   -        no       -    -     vlan254

# vm switch info voice
------------------------
Virtual Switch: voice
------------------------
  type: standard
  ident: vm-voice
  vlan: -
  physical-ports: vlan224
  bytes-in: 17056820423 (15.885G)
  bytes-out: 9467323522 (8.817G)

  virtual-port
    device: tap0
    vm: voicemail

# vm switch info data
------------------------
Virtual Switch: data
------------------------
  type: standard
  ident: vm-data
  vlan: -
  physical-ports: vlan254
  bytes-in: 158644461612 (147.749G)
  bytes-out: 156372027206 (145.632G)

  virtual-port
    device: tap1
    vm: ps01

# ifconfig vlan224
vlan224: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=401<RXCSUM,LRO>
        ether 00:25:90:27:5d:30
        groups: vlan
        vlan: 224 vlanpcp: 5 parent interface: lagg0
        media: Ethernet autoselect
        status: active
I tie the virtual interfaces directly to the vlan interfaces, so no tagging/VLAN configuration needs to be done on the clients or within bhyve.
 
yuripv said:
I really wonder how pass(4) fits here :)
Click to expand...
It fits right into the idea of TCP/IP - "pass_all" and let your router, switches, firewalls and shitload of host's network TCP/IP security utilities, frameworks and servers deal with it - hehe

The switch and a router with no firewall :D
www.flickr.com

Old-fashioned PBX switchboard at NE Life Insurance Co., Copley Square, Back Bay

File name: 11_07_000129 Title: Old-fashioned PBX switchboard at NE Life Insurance Co., Copley Square, Back Bay Creator/Contributor: Grant, Spencer, 1944- (photographer) Date created: 1972 Physical description: 1 negative : film, black & white ; 35 mm. Genre: Film negatives Subjects...
www.flickr.com www.flickr.com
 
Orum: Last time I tried (FreeBSD 10) I had issues with this configuration. I will try again then. Do you have an IP address on the bridge?
Oh and how come you have virtual ports.... My switch looks like this:


vm switch info isolated
------------------------
Virtual Switch: isolated
------------------------
type: standard
ident: vm-isolated
vlan: -
physical-ports: tap10 tap11 tap12 ix0.11
bytes-in: 128 (128.000B)
bytes-out: 0 (0.000B)



`Orum said:
We are using this exact configuration at work. Two Windows Server VMs in bhyve, with VLANs on top of lagg.

However, my config looks very different from SirDice's:
Code: 
# vm switch list
NAME   TYPE      IFACE     ADDRESS  PRIVATE  MTU  VLAN  PORTS
voice  standard  vm-voice  -        no       -    -     vlan224
data   standard  vm-data   -        no       -    -     vlan254

# vm switch info voice
------------------------
Virtual Switch: voice
------------------------
  type: standard
  ident: vm-voice
  vlan: -
  physical-ports: vlan224
  bytes-in: 17056820423 (15.885G)
  bytes-out: 9467323522 (8.817G)

  virtual-port
    device: tap0
    vm: voicemail

# vm switch info data
------------------------
Virtual Switch: data
------------------------
  type: standard
  ident: vm-data
  vlan: -
  physical-ports: vlan254
  bytes-in: 158644461612 (147.749G)
  bytes-out: 156372027206 (145.632G)

  virtual-port
    device: tap1
    vm: ps01

# ifconfig vlan224
vlan224: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=401<RXCSUM,LRO>
        ether 00:25:90:27:5d:30
        groups: vlan
        vlan: 224 vlanpcp: 5 parent interface: lagg0
        media: Ethernet autoselect
        status: active
I tie the virtual interfaces directly to the vlan interfaces, so no tagging/VLAN configuration needs to be done on the clients or within bhyve.
Click to expand...
 
tcn said:
Do you have an IP address on the bridge?
Click to expand...
No. No IP address anywhere. I would recommend reserving an interface specifically for tying to your VMs and do management of the machine itself via a separate interface. There are some weird interactions if you try to do both on the same interface and bridges.

On my host I have igb0 for managing the OS itself, igb1 and igb2 are bundled with lagg(4) and used exclusively for VMs.
 
I just tried and ARP requests are seen on my vlan interface ix0.11 but nothing gets to the bridge interface. There is something that I am missing.
My vlan interface is on the same fib as the bridge and tap devices which is not the default fib. I don't think this would be the issue as my initial configuration back then didn't have multiple fibs.
 
tcn said:
Do you have an IP address on the bridge?
Click to expand...
Yes and no. There is an IP address on the vlan254 interface, which is what I use to connect to the host (and of course the guest has an IP there too). I don't have an IP address on the vlan224 interface (for the host) as it doesn't need one on that VLAN.
tcn said:
Oh and how come you have virtual ports....
Click to expand...
I'm guessing you weren't running any VMs when you typed the command? I think they'll only show up if guests are connected to the virtual switch and up and running.

Edit: Oh, wait, you have a ton of physical ports instead of virtual ports. Odd, I'm not sure how you did that. What commands did you use to set up the switch and what do your VM conf files look like?
 
I think the difference is that I create the TAP interfaces before starting the VMs. I don't let bhyve create the interfaces. That would probably be why. I need to create them in order to put them in the proper fib.
 
I wonder if all of these problems aren't due to the multiple FIBs. As my machine isn't technically multi-homed (as the host only has an IP on one of the two vlan interfaces), I just use the default, single FIB.

Is there a reason you need the host to be multi-homed?

Edit: Also why do tap interfaces have anything to do with the FIB? They're layer 2 (i.e. bridged, not routed), so shouldn't touch the FIB at all. If you were using tun (though I'm not sure if that's possible with bhyve) I'd understand.
 
Hmmm didn’t think about the TAP being level 2. Makes sense to drop the FIB. The bridge on the other hand is routed (it has the gateway address for the fib 1). So I indeed could drop the fib from the tap devices.
I will test this tomorrow morning but not sure how traffic is going top travel. Windows does put an address on the tap interface making it level 3...... I a bit confused with the outer vs inner VM stuff.

I need the multi homed to prevent the Windows VM to be routed on the other network. I do not trust this internal network as I have a few administrators other than the ones working in the company. The Window VMs host software that requires professional installation; the kind that when you have an issue, only the company can fix. They required administrator access. I need to isolate this network.

Right now, I am doing double nat which I want to remove. I need it only to route my internal network to the VM’s because I can’t communicate with the VLAN interface directly. Keeping the multi-homed should be enough.

I will try again to put the VLAN interface in the bridge and removing the TAP interface creation so they become virtual. I failed this morning; traffic was going out of the VMs but was not going in. I must have done something wrong but had to put the system back to normal for the day.
 
I managed to get some things right.
The VM tries to pings an outside machine. ARP requests are sent by the VM, received and replied by the outside machine on the proper VLAN. The reply comes at the interface but doesn't get to the VLAN interface......
I can see ARP replies tagged with the right VLAN on the physical interface but nothing is transferred to the VLAN interface as if it is dropped when the tag is removed to be passed to the VLAN interface...
Not sure what to think of this..... Weird stuff.
 
I think I misunderstood your original post, as re-reading it now it seems like you wanted 802.1Q headers to flow through to the VM, thereby handing the guest a trunk port. The guests are then multi-homed, which is not fun to do within Windows but I've done it in the past on physical hardware. I've not done this in bhyve (as none of our guests need trunk ports), so I can only make a best guess as to how to achieve what you want. But, what I would try is bridging directly with the lagg interface, and not configure any vlan interfaces on the host unless you need to assign an IP to them for management of the host. All tagged traffic should appear on the lagg interface even though if one were to send traffic directly out that interface it would be on the 'native' (untagged) VLAN. You can verify this by running tcpdump(1) on the lagg interface and looking for tagged traffic.

This has a few downsides as every guest would would be able to access every VLAN your physical switch provides (including the native VLAN), which may present security problems. If that's a concern, there's probably a way to configure a virtual switch within vm-bhyve to only permit traffic from certain VLANs to flow through, but I've not investigated this.

Regardless, unless you really need the host multi-homed, you should only have one FIB and one IP on the host, and FIBs shouldn't affect the guests when bridging. As for the guests, be aware that if you give Windows multiple default gateways, which can happen easily if you're assigning IPs via DHCP, you're asking for trouble. I strongly recommend not doing this. Instead, either only have a default gateway on one interface (VLAN), manually configure the entire routing table, or some combination of the two (a default route on one interface with a few manually configured static routes on the others).
 
No. Windows guests cannot have VLANs; the driver doesn't support it. The tag should be added by the physical intetrface which is what it is doing.
The problem is that when the packet is seen at the physical interface, it is dropped when the tag is removed to be passed to the physical VLAN interface. I sniffed my interface on both non VLAN and VLAN identities and saw the ARP replies coming back with the proper tag on the non VLAN identity ( tcpdump -i ix0 -vv -e vlan) but the packet is not seen on the VLAN identity ( tcpdump -i ix0.1 -vv -e).

So if I can describe this....
VM-> ARP request -> VLAN interface(ix0.1) -> physical interface (ix0) -> NET works just fine
NET -> physical interface (ix0) -> DROP!!! -> VLAN interface (ix0.1)

I have no clue as of why the packets are dropped in between.....
 
tcn said:
No. Windows guests cannot have VLANs; the driver doesn't support it.
Click to expand...
Are you sure about that? I see loads of 802.1Q related options on our Windows guests, but then again, I'm using Red Hat's VirtIO driver:

5b1jba.png


Or did you mean they aren't supported on the host/tap side of things?

Anyway, it sounds like you want an access port then on the Windows guests, which is exactly what we have. What's odd to me is the way your vlan interfaces are named, but maybe that's just the new covention? What does your rc.conf look like?

Here's the relevant parts on our bhyve server with the guests I mentioned earlier:
Code: 
ifconfig_em0="up"
ifconfig_em1="up"

cloned_interfaces="lagg0 vlan224 vlan254"
ifconfig_lagg0="laggproto lacp laggport em0 laggport em1"

ifconfig_vlan224="vlan 224 vlanpcp 5 vlandev lagg0"
ifconfig_vlan254="inet 172.16.88.220/24 vlan 254 vlandev lagg0"

defaultrouter="172.16.88.254"

vm_enable="YES"
vm_dir="zfs:zr/vm"
vm_list="voicemail ps01"
vm_delay="20"
And I've already posted what our virtual switches look like, which bridges the vlan interfaces directly with the virtual switches.

Edit:
tcn said:
NET -> physical interface (ix0) -> DROP!!! -> VLAN interface (ix0.1)
Click to expand...
This looks like something is wrong on the FreeBSD network configuration or within the virtual switch. Do you know exactly what commands you typed to configure the virtual switch? Alternatively I think you can look at your .config/system.conf file in the vm_dir. Here's mine:
Code: 
switch_list="voice data"
ports_voice="vlan224"
ports_data="vlan254"
type_voice="standard"
type_data="standard"
 
I am using then new rc.d way of things (separated files instead of a big rc.conf). Here is part of my network config:

Code: 
vlans_ix0="1"
ifconfig_ix0_1="inet 192.168.11.2/24 fib 1"
ifconfig_ix0_1_alias0="inet 192.168.11.9/24 fib 1"
ifconfig_ix0_1_alias1="inet 192.168.11.100/24 fib 1"

It comes down to the same thing but instead, I let the script handle the vlandev part.

I just checked and indeed, VLAN is supported in the VM.... weird.... I was sure this was not supported back when I first built the system. I couldn't find the 802.11Q configuration. I might try this instead.
The tag interface does support VLANs. But if this works, I do not need the fib and I don't even need the address on the vlan interface.
 
Hrm, that's a new way of configuring vlans that seems less explicit. As long as things look right in ifconfig though I suppose it's not a problem.

Also, why have so many aliases? Are you putting the IP addresses of the guests in the interface configuration on FreeBSD? If so, that's a no-no, as you're effectively creating an IP address conflict on the network, as the IP would be assigned to both the host and the guest, which might explain why the guest isn't seeing the traffic. Oh, and you should not use /24 on an alias in the same subnet. Use a /32 mask for those IPs.
 
Less explicit.... I like it better. I have always named my vlans interface.tag; I find it natural as I know to which interface the VLAN is assigned to just by looking at the name.
I am doing double NAT. Each address is translated to the proper internal address on the bridge. And I agree, /24 is not totally right although still works (if it wasn't, I wouldn't be able to communicate with my VMs). The main interface remains the first address configured. I will change it though as it is not right.
 
If the goal is faster performance, you'll have to remove NAT to test things. You might not see the traffic in tcpdump on the vlan interfaces unless they pass it out to another device tied to the same interface (e.g. a switch). This becomes apparent when you want to pass traffic between jails on a single host.

For example, if I have two jails on one host, that are using IPs assigned to a vlan (or any non-loopback interface), and they send traffic to one another, the traffic itself will flow over the loopback (typically lo0) and not the interface that actually holds the IPs. This is really important when you're trying to firewall the traffic with, for example, pf.

Either way, until your configuration is changed to not use NAT and then tested, it's impossible to tell if there's even still a problem.
 
This is the goal. Remove the NAT.
VLAN is not supported on the virtual interface in Windows. You saw 802.3Q things but there is nowhere to set the VLAN ID. So I have to find a way to make the bridge work.
I tested again and found out that the return ARP reply was done using the Windows MAC address. The interface ix0 drops the packet instead of transferring it to the VLAN interface ix0.1. This is what is happening. ix0 is not part of the bridge and drops the packets.
I wonder now how you guys have VLAN working on your setup..... The MAC should be different from the one on the NIC and the NIC should therefore drop it.......
 
You must log in or register to reply here.
Back
Top