Having difficulty with getting network connectivity to a bhyve instance

BeeEsDee

New Member


Messages: 4

I'm currently playing around with bhyve for the first time and am trying to setup networking. My objective is to have this bhyve instance live in its own network concealed within my machine and use the host for reaching the outside world.

This is what I have so far:

Code:
# Create the bridge.

ifconfig bridge create
ifconfig bridge0 name vswitch
iconfig vswitch up
ifconfig vswitch addm em0
ifconfig vswitch inet 172.77.15.1/24

# Create the tap interface.

ifconfig tap0 create
ifconfig vswitch addm tap0
I have configured my pf.conf as follows:

Code:
vms_network="172.77.15.1/24"
...
nat on em0 from $vms_network to any -> (em0)
...

And then I start my vm with as follows:

Code:
sudo bhyve -c 2 -m 4G -w -H \
    -s 0,hostbridge \
    -s 3,ahci-cd,/somewhere/on/disk/archlinux-2021.07.01-x86_64.iso \
    -s 4,virtio-blk,/dev/zvol/pool-1/bhyve/vms/arch \
    -s 5,virtio-net,tap0 \
    -s 29,fbuf,tcp=0.0.0.0:5900,w=1920,h=1080,wait \
    -s 30,xhci,tablet \
    -s 31,lpc -l com1,stdio \
    -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
    arch

And then this is about where I get lost. I can connect to the VM through VNC and it boots successfully, but then it has no network connection. My intention was to set everything statically... which I think I'm supposed to do within the VM's OS? Trying to do so doesn't fix anything (by which I mean setting a static IP of 172.77.15.2 with 172.77.15.1 as the default gateway). I feel like I'm probably missing a lot.

Could someone help me out by walking me through the configuration I should be using to set up what I want? Thanks!
 

cmoerz

Member

Reaction score: 37
Messages: 98

Have you tried pinging your guest from your host? Does that work? Can your guest ping your host? Lets first try to get guest/host communications working, before moving to NAT.
You might still want to check your rc.conf or sysctl.conf whether you enabled packet forwarding. If that isn't turned on, NAT won't work.
 

PMc

Daemon

Reaction score: 676
Messages: 1,358

The most simple way I managed to do it is to leave away the bridge and just create a tap device. Then put an IP adress on that tap device (on the host), put the neighbour IP address on the vtnet0 interface (in the guest), both a netmask of 0xfffffffc, and then that works.
The disadvantage is that now all guest traffic must be routed through the host (layer3, IP-stack), which is a performance penalty. The more effective way is to use a bridge, so the guest can directly access the outbound interface and outside world.

to have this bhyve instance live in its own network concealed within my machine and use the host for reaching the outside world.
If that is the case and the guest uses a separate network that is not visible (or route-able) on the outside, then this must be someway nat'ed on the host. Then the traffic must be routed through the host anyway, and it appears to me somehow pointless to bridge it to the outbound interface. (At least I can't figure how that might work.)

I am doing just that, and I am using a bridge only to connect multiple guests together (and to the host). The outside physical netif is then a different matter, and that is where the nat works (and rewrites all packets to the host's own IP). But then I am using netgraph bridges, and ipfw - because, honestly, I don't understand the other stuff. YMMV.
 
OP
B

BeeEsDee

New Member


Messages: 4

Have you tried pinging your guest from your host? Does that work? Can your guest ping your host? Lets first try to get guest/host communications working, before moving to NAT.
You might still want to check your rc.conf or sysctl.conf whether you enabled packet forwarding. If that isn't turned on, NAT won't work.
Ah, I hadn't enabled packet forwarding! I thought I had as I have jails using shared IPs on lo1 currently NATed through em0. Not sure how that's been working all this time o_O. Anyway, now everything works! Thanks!

Then the traffic must be routed through the host anyway, and it appears to me somehow pointless to bridge it to the outbound interface.

I was wondering about this. I started without adding em0 to the bridge, but added it later when nothing was working. Do I not need to do that at all? Is the NAT enough to somehow carry the data between the bridge and the outside world without "physically connecting" the bridge to em0?
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 12,325
Messages: 38,841

I was wondering about this. I started without adding em0 to the bridge, but added it later when nothing was working. Do I not need to do that at all? Is the NAT enough to somehow carry the data between the bridge and the outside world without "physically connecting" the bridge to em0?
If you connect em0 to the bridge you have a direct connection to the 'outside' world. If you don't add em0 then NAT translates all traffic to the address of the host. It's just a different network setup. If you have a 'direct' connection you're going to need to fix your routing, so other hosts on your network can find the network that's used for the VMs. This is just basic TCP/IP networking.
 

dlo

New Member

Reaction score: 1
Messages: 8

Or use vm-bhyve it does networking for you.
This is no longer the case for NAT-based configurations.

Bash:
$ sudo vm switch nat public on
/usr/local/sbin/vm: WARNING: internal nat support is currently disabled
/usr/local/sbin/vm: WARNING: please add an address to the virtual switch and configure your firewall for NAT manually
 
Top