bhyve with wlan0 NAT for my guests

Y

yha

I'm trying to wrap my head around how virtual bridges/switches/tap interfaces work under bhyve. I guess I'm doing something fundamentally stupid here (or my understanding of how this works is fundamentally wrong), but please bear with me...

I do use bhyve on a few of my machines, but on those I use wired connections like USB or Ethernet for internet access. Now on this machine, a laptop, I connect to the internet through a mifi router (NETGEAR Nighthawk M5, if it matters), so bridging doesn't work in this case, and I have to use NAT.

Before I describe my setup, I have a couple of general questions that have been bothering me:
  1. I see lots of documentation about assigning a bridge or virtual switch an IP address. How is that possible? My understanding is that switches, for example, simply "route" (not in the layer 3 sense) frames to the correct ports on the switch. Same things with bridges: it's a device that bridges two different networks, and doesn't have a NIC for it to be assigned an IP address.
  2. A tap interface, which is a virtual NIC (correct me if I'm wrong), can be assigned an IP address, and is usually "attached" to a guest machine. The problem is it is also available for configuration on the host with something like ifconfig, so how does that work? Do I configure it from the host? The guest? What happens if I say "ifconfig tap0 192.168.1.5" on the host and give it a different address in the guest?
Now back to my actual setup. I'm not getting internet in my Windows 11 guest. My mifi is configured to use a class A network (10.5.5.0/8, with the router itself being 10.5.5.1). I want to give my bhyve guests addresses in the 10.5.10.x range. The host machine is configured to use DHCP for the wlan0 interface, and it usually gets something like 10.5.5.77.

These are the steps I followed to create/configure my guest:

  1. vm switch create public and vm switch add public wlan0.
  2. Use virtio-net in my vm-bhyve config ( network0_type="virtio-net" and network0_switch="public") and install the virtio driver under Windows.
  3. sysctl net.inet.ip.forwarding=1
  4. Add nat on wlan0 from 10.5.10.0/8} to any -> (wlan0) to pf.conf and enable pf (I don't do it in rc.conf, but I do run service pf onestart).
With this config (and DHCP in Windows), I'm getting no internet in the guest (and I can't ping the host from the guest or vice versa). So I tried static addresses:

  1. Guest IP 10.5.10.2/8, default gateway 10.5.5.1 or 10.5.10.1 (tried both).
  2. Assign vm-public an IP of 10.5.10.1.
Again, the host and the guest can't see each other, let alone getting internet in the guest.

The tap and wlan0 interfaces are attached to the virtual switch:
Code: 
t14s# vm switch info public
local: _netgraph,: bad variable name
------------------------
Virtual Switch: public
------------------------
  type: standard
  ident: vm-public
  vlan: -
  physical-ports: wlan0
  bytes-in: 47056 (45.953K)
  bytes-out: 3487760 (3.326M)

  virtual-port
    device: tap0
    vm: windows11
 
First off, I'm just in the experimental stages with bhyve. So I don't know that I'll have great answers, but hopefully I can help you along a bit. Also, we'll be talking about networking stuff, and I'm not great at that either. So there's the caveat :)

yha said:
I see lots of documentation about assigning a bridge or virtual switch an IP address. How is that possible? My understanding is that switches, for example, simply "route" (not in the layer 3 sense) frames to the correct ports on the switch. Same things with bridges: it's a device that bridges two different networks, and doesn't have a NIC for it to be assigned an IP address.
Click to expand...

I do this, because as far as I can tell, it makes networking really easy. You can assign an IP to a bridge interface in FreeBSD. This automatically sets up the correct routing on the host. I set it to e.g. 192.168.3.1/24 and now I just have to plug tap interfaces into it, and give each of the bhyves an IP on that same subnet, set the defaultgateway in the bhyve to 192.168.3.1 and it works.

yha said:
A tap interface, which is a virtual NIC (correct me if I'm wrong), can be assigned an IP address, and is usually "attached" to a guest machine. The problem is it is also available for configuration on the host with something like ifconfig, so how does that work? Do I configure it from the host? The guest?
Click to expand...

I add the host tap to the bridge, no other host configuration needed. I configure the interface in the guest, where it's called vtnet0.

yha said:
My mifi is configured to use a class A network (10.5.5.0/8, with the router itself being 10.5.5.1). I want to give my bhyve guests addresses in the 10.5.10.x range. The host machine is configured to use DHCP for the wlan0 interface, and it usually gets something like 10.5.5.77.

These are the steps I followed to create/configure my guest:

  1. vm switch create public and vm switch add public wlan0.
  2. Use virtio-net in my vm-bhyve config ( network0_type="virtio-net" and network0_switch="public") and install the virtio driver under Windows.
  3. sysctl net.inet.ip.forwarding=1
  4. Add nat on wlan0 from 10.5.10.0/8} to any -> (wlan0) to pf.conf and enable pf (I don't do it in rc.conf, but I do run service pf onestart).
Click to expand...

Okay a few things to check:

1. You need to NAT (which you're doing)
2. The host needs to have sysctl net.inet.ip.forwarding=1
3. Guest needs a default route. In my case, the default route is to the bridge IP.

Now for the big one: I'm not positive, but one potential issue is that the subnets you're describing overlap, and I don't think you can correctly configure networking and NAT with them.

You say your router is 10.5.5.0/8 and you want the bhyves to be 10.5.10.x - but that is part of 10.5.5.0/8. Use this CIDR calculator and you'll see that /8 is 10.0.0.0 to 10.255.255.255 - the whole 10.x.x.x range. So when you say nat on wlan0 from 10.5.10.0/8} to any -> (wlan0), that doesn't make sense to me - it's like it's NATing to the same network.

So here's how I'd proceed if I were in your shoes:

1. Temporarily choose an obviously non-overlapping subnet: 172.16.0.0/24 for example.
2. Install a FreeBSD guest to figure out the networking config.
3. Get it working... my link above shows the main parts of a working config, the rest I've described here.

Once you have that config working, you can start to change it to get closer to what you want. I think you need to make your router and bhyve networks have non-overlapping ranges. Then you can set up a Windows guest with the same networking configuration as the working FreeBSD one.
 
I did fix it. Actually my issue was that vm bhyve does not manage wifi connection out of the box. If i use ethernet cable then does work with the quickstart guide on the github page of vm-bhyve. Just create the switch and assign it to the real interface. However that interface must be ethernet not wifi. But because the guide did not mention it I was using my wifi interface as that is my default to connect to the wweb. So i did destroy vm switch public and was creating a new one and linked to my ethernet interface. Now does work.
 
Yeah, you can use bridge network with ethernet, so the jails can get IP addresses directly from the DHCP server. The problem is on wifi, you can't have multiple IP addresses coming from the same NIC, so you have to use a NAT approach as yha is discussing. The wiki has an entry that simply says you can't use bhyve with a wireless network.
 
Thanks a ton patmaddox! The bad subnetting was indeed the issue. I configured my router to use 10.5.5.0/24 instead of /8, and the guest machine to use 10.5.10.0/24, and everything just worked.
 
You must log in or register to reply here.
Back
Top