Bhyve switch does not work with wifi

D

dilyanb

I have a working setup with bhyve on FreeBSD13.1, with OpenBSD, Debian and Windows 10 VMs. With wired connectivity thru em0, everything is up and running thru their bridge (named public below):

# ifconfig
Code: 
em0: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=4812099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,NOMAP>
    ether 18:03:73:1e:b2:eb
    inet 192.168.3.115 netmask 0xffffff00 broadcast 192.168.3.255
    inet 192.168.3.116 netmask 0xffffff00 broadcast 192.168.3.255
    inet 192.168.3.117 netmask 0xffffff00 broadcast 192.168.3.255
    inet 192.168.3.118 netmask 0xffffff00 broadcast 192.168.3.255
    inet 192.168.3.119 netmask 0xffffff00 broadcast 192.168.3.255
    inet 192.168.3.120 netmask 0xffffff00 broadcast 192.168.3.255
    inet 192.168.3.121 netmask 0xffffff00 broadcast 192.168.3.255
    inet 192.168.3.122 netmask 0xffffff00 broadcast 192.168.3.255
    inet 192.168.3.123 netmask 0xffffff00 broadcast 192.168.3.255
    inet 192.168.3.124 netmask 0xffffff00 broadcast 192.168.3.255
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
    inet 127.0.0.1 netmask 0xff000000
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
wlan0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 5c:a6:e6:c3:2c:38
    inet 192.168.3.116 netmask 0xffffff00 broadcast 192.168.3.255
    groups: wlan
    ssid 2DidoB channel 6 (2437 MHz 11g ht/20) bssid e0:63:da:3d:7c:4d
    regdomain ETSI country BG authmode WPA2/802.11i privacy ON
    deftxkey UNDEF AES-CCM 2:128-bit txpower 30 bmiss 7 scanvalid 60
    protmode CTS ht20 ampdulimit 64k ampdudensity 8 shortgi -stbc -ldpc
    -uapsd wme roaming MANUAL
    parent interface: rtwn0
    media: IEEE 802.11 Wireless Ethernet MCS mode 11ng
    status: associated
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
vm-public: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 5e:82:0c:b6:69:e4
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: tap2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 7 priority 128 path cost 2000000
    member: tap1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 6 priority 128 path cost 2000000
    member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 5 priority 128 path cost 2000000
    member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 1 priority 128 path cost 20000
    groups: bridge vm-switch viid-4c918@
    nd6 options=9<PERFORMNUD,IFDISABLED>
tap0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: vmnet/obsd-bastion/0/public
    options=80000<LINKSTATE>
    ether 58:9c:fc:10:d6:1d
    groups: tap vm-port
    media: Ethernet autoselect
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    Opened by PID 1617
tap1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: vmnet/openbsd72/0/public
    options=80000<LINKSTATE>
    ether 58:9c:fc:10:19:26
    groups: tap vm-port
    media: Ethernet autoselect
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    Opened by PID 1879
tap2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: vmnet/windows10/0/public
    options=80000<LINKSTATE>
    ether 58:9c:fc:10:ff:86
    groups: tap vm-port
    media: Ethernet autoselect
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    Opened by PID 4345

I had to move the PC away from the wired setup and use wifi, so changed the settings of the bhyve bridge to use wlan0 instwed of em0. First I decided to make it with adding wlan0 next to em0 -> vm switch add public wlan0 - not working.
Then I removed the whole "public" bridge and created it from scratch with just the wlan0 on ports - not working. The virtual machines can ping their host machine IP, but not the gateway on .1 thru the wireless. Also no internet connectivity from any of the VMs.

Then I followed https://forums.freebsd.org/threads/vm-bhyve-share-wlan0-with-archlinux.73926/ and added the network address instead of the interface as suggested in the post - no luck.

Moved back the PC on cable and added back the em0 to the bridge - all up and running.

Code: 
vm switch list
NAME    TYPE      IFACE      ADDRESS  PRIVATE  MTU  VLAN  PORTS
public  standard  vm-public    -               no            -          -         em0

Tried to assign IP address on tap0 so I can make a NAT thru pf.conf: nat on wlan0 from tap0 to any -> wlan0 - not working. Tried to nat the vm-public bridge with the -a 192.168.3.0/24 and without it, with wlan0 - also not working.
Has anyone made it work with wifi, and how?
I wonder what's breaking with the wifi so it is not working at all?
 
It doesn't work because you have two interfaces in the same subnet, namely em0 and wlan0. The 192.168.3.0/24 network is directly connected on both interfaces. That makes routing ambiguous.

If you want to do this you will have to move all the other 192.168.3.0/24 addresses (jails?) to wlan0 too.
 
I disabled em0 and all the jails. Left wlan0 only, even changed its IP to the same as the em0 (.115). Still no luck.
Has anyone managed to have bhyve switch with wifi at all?
 
kr0m said:
I am in the same situation, any workaround tip would be appreciated.
Click to expand...
dilyanb said:
I disabled em0 and all the jails. Left wlan0 only, even changed its IP to the same as the em0 (.115). Still no luck.
Has anyone managed to have bhyve switch with wifi at all?
Click to expand...

zirias@ said:
First thing standing out, you have
Code: 
firewall_enable="YES"
pf_enable="yes"
Enabling ipfw and pf at the same time certainly is begging for (networking) trouble.

Then, I'm pretty sceptical doing nat would EVER work when you're not routing. Even if so, you seem to use some dynamic table for it, is it ever filled? Can't see that above.

Why don't you try the straight-forward thing, just add the jail to the bridge and configure the interface inside the jail for DHCP as well, so it would get some valid address from the same subnet as your host (and have the correct routes automatically)? (that works, but probably only when using the LAN interface)

Edit: just thought about how Wifi authenticates stations (based on MAC addresses) and googled a bit, confirming the issue: Well, the bridge with your lagg device will certainly work. But even when you configure it correctly, as soon as the active interface in the lagg is the wifi interface, the access point receiving the package will reject it because it has the wrong source MAC.

The simple solution if you want to share a wifi connection is routing. That's doable with VMs and VNET jails as well of course.

What could also work is using the wds mode for your Wifi, but both AP and station must have it enabled, and I never tried that. It offers one more MAC field, so it can differentiate between the sender of the package (could be your jail) and the sending wifi station (your host's wifi interface).
Click to expand...
 
I have the same problem, but could the problem be with the wireless adapter not being capable of sending the MAC address of the VM / tagged packets?
 
nbari a "wrong" MAC certainly is the problem, see my post where I explained it (after being on the wrong track myself) earlier, that BobSlacker thankfully already quoted right above.

Short of things that might or might not work, the straight-forward workaround is to not ever use a bridge with wifi but instead route your traffic.
 
Any tips, docs about how to route, I am currently using vm-bhyve, but if can't use a bridge wondering how to doit configure the VM's
 
If you also need NAT, here's a guide: https://github.com/churchers/vm-bhyve/wiki/NAT-Configuration -- this is probably the "common case" if the network you're connected to via Wifi shouldn't get any extra routes.

Otherwise it's even simpler, setup the "virtual switch" of vm-bhyve (which is a bridge) for a *different* subnet and don't add your real interface to that bridge, instead setup the necessary routes and enable forwarding on the host.
 
zirias@ said:
If you also need NAT, here's a guide: https://github.com/churchers/vm-bhyve/wiki/NAT-Configuration -- this is probably the "common case" if the network you're connected to via Wifi shouldn't get any extra routes.

Otherwise it's even simpler, setup the "virtual switch" of vm-bhyve (which is a bridge) for a *different* subnet and don't add your real interface to that bridge, instead setup the necessary routes and enable forwarding on the host.
Click to expand...
Hi, can you please help me to better understand how to do the second option, this is my current setup and I think you mean something like this:

Code: 
router opnsense           ---> ~~wifi~~ <--- PC (FreeBSD with bhyve)
LAN (192.168.0.0/24)                         host in lan (192.168.0.11/24)
BHYVE (10.0.0.0/24) VLAN 20                  vm-test (switch in 10.0.0.0/24) <---- (you mean like this)?
     |
     |
     |  (ethernet works fine)
     |
Wired PC with Bhyve
LAN (192.168.0.10/24)
vm-public (10.0.0.0/24) VLAN 20

How to configure the routes so that I can forward them from the PC using wifi on vm-test 10.0.0.0 to the router and send packets back?
 
If I interpret your diagram correctly, I don't think this can work. You can't use the same network in different places, how would you ever configure the routes... the easiest solution for you is probably to indeed use NAT (so your whole vm subnet on your wifi pc hides behind its one address, here 192.168.0.11).

But, if you use an entirely different network for your "vm-test", like e.g. 192.168.1.0/24, you could add a route to it on your router with a gateway of 192.168.0.11.
 
Facing the same problem. And some things seem quite interesting:
  • the switch does receive DHCP-discovery requests' packets, but they are not forwarded (I see them from the host, but I don't see them from another computer in the same network);
  • the same configuration works fine if wlan0 is replaced with em0.
Here is the listing of the DHCP request that I captured on the host

Code: 
    0.0.0.0.bootpc > 255.255.255.255.bootps: [udp sum ok] BOOTP/DHCP, Request from 58:9c:fc:08:ce:d6 (oui Unknown), length 300, xid 0x60b5fe5c, secs 1989, Flags [none] (0x0000)
      Client-Ethernet-Address 58:9c:fc:08:ce:d6 (oui Unknown)
      Vendor-rfc1048 Extensions
        Magic Cookie 0x63825363
        DHCP-Message (53), length 1: Discover
        MSZ (57), length 2: 576
        Parameter-Request (55), length 7:
          Subnet-Mask (1), Default-Gateway (3), Domain-Name-Server (6), Hostname (12)
          Domain-Name (15), BR (28), NTP (42)
        Hostname (12), length 12: "docker.hp800"
        Vendor-Class (60), length 12: "udhcp 1.36.1"
        Client-ID (61), length 7: ether 58:9c:fc:08:ce:d6
        END (255), length 0
        PAD (0), length 0, occurs 6
 
Check your AP if it can be configured in mixed mode so it can act as wifi bridge + standard AP for end clients also known as WDS. Then you can try to connect your end device and check if it allow more than one MAC address via the wifi. Some vendors named this mode as wireless repeater bridge.
 
You must log in or register to reply here.
Back
Top