Problem with vm-bhyve and NAT networking

Follpvosten · May 25, 2018

Hello!

For context, I'm currently trying to get rid of VirtualBox on my server, replacing it with bhyve, using the sysutils/vm-bhyve wrapper.

I created a VM locally, using the automatic NAT provided by vm-bhyve ( vm switch create public, vm switch nat public on), and configured it to my liking, with the static IP 172.16.0.4. On my local machine, pretty much everything worked without any problems, including running the VM and an iocage jail at the same time, as well as installing tons of packages inside the VM (I basically made it ready-to-deploy).

Then, I migrated it to my remote server by taking a snapshot and using zfs send and zfs recv to transfer my whole "vm-bhyve" dataset, including all of my configuration, so I didn't have to set up the "public" switch again.
I added the lines

Code:

# Include vm-bhyve configuration
include "/vm-bhyve/.config/pf-nat.conf"

to my server's pf.conf file (that had been done automatically on my local machine, but not here since I didn't run vm switch nat public on on the remote server). I also changed the interface in that "pf-nat.conf" file from "wlan0" to "em0", which is my server's primary network interface.

After commenting out everything in my rc.conf that had to do with VirtualBox, I rebooted the server to cleanly get rid of the VBox kernel modules, then I ran # sysrc vm_enable="YES" and # sysrc vm_dir="zfs:<the dataset>", followed by # vm init, all of which worked fine. I proceeded to start the vm using # vm start <vm>, which also worked fine.

Now onto the problem - I can SSH into the VM from the host without problems, but the VM can only connect to the internet sometimes. When I try to ping google.com, it doesn't work for 2-3 times; then I get a response from the nameserver, so it tries to ping an IP, which doesn't work. Pinging the nameserver directly (I'm using 1.1.1.1) also works sometimes - when I try to to that, it doesn't work for 2-3 tries, then it works until I stop the program; after that, it doesn't work for the next 5-6 times, and repeat.

The content of some related config files:

Code:

# Public IP addresses
IP_PUB="178.32.223.6"
IP_NC="178.32.109.82"
IP_MAIL="188.165.173.129"
WEB_PORTS="{ 80, 443 }"
MAIL_PORTS="{ 25, 465, 587, 143, 993, 110, 995, 80, 443 }"

# Packet normalization
scrub in all

# Allow outbound connections from within jails
nat on em0 from lo1:network to any -> (em0)

# Include vm-bhyve configuration
include "/vm-bhyve/.config/pf-nat.conf"

# Forward all mailserver traffic
#rdr on em0 proto tcp from any to $IP_MAIL port $MAIL_PORTS -> 172.16.0.4

# Redirect nextcloud IP (no longer needed)
rdr on em0 proto tcp from any to $IP_NC port $WEB_PORTS -> 192.168.0.2

# Redirect HTTP(S) traffic
rdr on em0 proto tcp from any to $IP_PUB port $WEB_PORTS -> 192.168.0.2

Note the commented mail traffic redirection line - that's supposed to be activated when the mail server VM is running.

Code:

# vm-bhyve nat
nat on em0 from {172.16.0.0/24} to any -> (em0)

Note: These are only the parts related to vm-bhyve and networking.

Code:

### This block was added by OVH
# Network configuration (IPv4)
ifconfig_em0="inet 178.32.223.6 netmask 255.255.255.0 broadcast 178.32.223.255"
defaultrouter="178.32.223.254"

# Network configuration (IPv6)
ifconfig_em0_ipv6="inet6 2001:41d0:0008:e206:: prefixlen 64 accept_rtadv no_radr"
ipv6_network_interfaces="em0"
ipv6_default_interface="em0"
ipv6_defaultrouter="2001:41d0:0008:e2ff:ff:ff:ff:ff"
ipv6_route_ovhgw="2001:41d0:0008:e2ff:ff:ff:ff:ff -prefixlen 128 -interface em0"
ipv6_static_routes="ovhgw"

# Daemons
ntpd_enable="YES"
sshd_enable="YES"
local_unbound_enable="YES"

### The following options were added by me
# vm-bhyve
vm_enable="YES"
vm_dir="zfs:zroot/vm-bhyve"

# Jails
cloned_interfaces="lo1"
ipv4_addrs_lo1="192.168.0.1-14/28"
iocage_enable="YES"

pf_enable="YES"

# IP aliases:
# Old Nextcloud IP
ifconfig_em0_alias0="inet 178.32.109.82 netmask 255.255.255.255 broadcast 178.32.109.82"
# Mailserver IP
#ifconfig_em0_alias1="inet 188.165.173.129 netmask 255.255.255.255 broadcast 188.165.173.129"

Code:

bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: vm-public
        ether 02:6b:49:ce:a0:00
        inet 172.16.0.1 netmask 0xffffff00 broadcast 172.16.0.255 
        nd6 options=1<PERFORMNUD>
        groups: bridge 
        id 00:00:00:00:00:00 priority 0 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 0 ifcost 0 port 0
        member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 5 priority 128 path cost 2000000
tap0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: vmnet-debian-mailcow-0-public
        options=80000<LINKSTATE>
        ether 00:bd:d3:8a:f8:00
        hwaddr 00:bd:d3:8a:f8:00
        nd6 options=4b<PERFORMNUD,ACCEPT_RTADV,IFDISABLED,NO_RADR>
        media: Ethernet autoselect
        status: active
        groups: tap 
        Opened by PID 2352

sysctl net.inet.ip.forwarding outputs "net.inet.ip.forwarding: 1", and internet in all of the running Jails works without any problems.

If anyone has any idea or tips on why this could be happening, please tell me! Any help would be appreciated.
Thanks!

Problem with vm-bhyve and NAT networking

Follpvosten