Solved Networking no longer working for guest OS -- FreeBSD Host running Bhyve + Windows 10 and FreeBSD guests

Z

Zuna Mae

Hello,

I am running FreeBSD (13.0-RELEASE-p10) with two guest operating systems (Windows 10 and FreeBSD). I am using the 'vm-bhyve-1.4.2' package for the setup and configuration of the virtual machines.

I've referenced the following guides for setup and configuration:
docs.freebsd.org

Chapter 24. Virtualization

Virtualization software allows multiple operating systems to run simultaneously on the same computer
docs.freebsd.org docs.freebsd.org
https://github.com/churchers/vm-bhyve/wiki

Unfortunately, there is something that I'm missing and I cannot get the guest operating systems to respond on the bridged network. I've tried using both the "vm switch" setup as well as manually setting up a bridge and attaching my ethernet interface / tap interfaces, but to no avail.

Please find below the configuration using the "vm switch" method -- let me know if there's any useful information I am missing.

The ultimate result of the below configuration is no response to pings to the static IP (10.0.100.33) configured on the guest operating system. And obviously, the guest operating system has no outbound network connectivity either.

output from 'kldstat' on the FreeBSD host showing the vmm, nmdm, if_bridge, etc kernel modules loaded:

Code: 
# kldstat
Id Refs Address                Size Name
 1   50 0xffffffff80200000  1f11f90 kernel
 2    1 0xffffffff82113000   6800a0 zfs.ko
 3    1 0xffffffff82794000     ae38 cryptodev.ko
 4    1 0xffffffff82920000     3250 ichsmb.ko
 5    1 0xffffffff82924000     2180 smbus.ko
 6    1 0xffffffff82927000     2340 uhid.ko
 7    1 0xffffffff8292a000     4350 ums.ko
 8    1 0xffffffff8292f000     3380 usbhid.ko
 9    1 0xffffffff82933000     31f8 hidbus.ko
10    1 0xffffffff82a00000   53a420 vmm.ko
11    1 0xffffffff82937000     21cc nmdm.ko
12    1 0xffffffff8293a000     7638 if_bridge.ko
13    1 0xffffffff82942000     50d8 bridgestp.ko
14    1 0xffffffff82948000     2a08 mac_ntpd.ko

Creation of virtual switch on the FreeBSD host:

Code: 
# vm switch create switch0
# vm switch add switch0 bge0
# vm switch list
NAME     TYPE      IFACE       ADDRESS  PRIVATE  MTU  VLAN  PORTS
switch0  standard  vm-switch0  -        no       -    -     bge0

Configuration of virtual machine with mapping to "switch0" bridge device:

Code: 
loader="bhyveload"
cpu=2
memory=2G
network0_type="virtio-net"
network0_switch="switch0"
disk0_type="virtio-blk"
disk0_name="disk0"
disk0_dev="disk0.img"
uuid="0b2e192b-aa3d-11ec-897b-6c2b59c008a7"
network0_mac="58:9c:fc:0d:ca:3b"

ifconfig output on FreeBSD host showing bridge device and bge0 / tap0 as members:

Code: 
# ifconfig -a
bge0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=c0099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,VLAN_HWTSO,LINKSTATE>
        ether 6c:2b:59:c0:08:a7
        inet 10.0.100.30 netmask 0xffffff00 broadcast 10.0.100.255
        inet 10.0.100.6 netmask 0xffffffff broadcast 10.0.100.6
        inet 10.0.100.7 netmask 0xffffffff broadcast 10.0.100.7
        inet 10.0.100.18 netmask 0xffffffff broadcast 10.0.100.18
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
vm-switch0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 32:08:5c:14:48:a3
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 5 priority 128 path cost 2000000
        member: bge0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 200000
        groups: bridge vm-switch viid-a7d0a@
        nd6 options=9<PERFORMNUD,IFDISABLED>
tap0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: vmnet-amsdev-0-switch0
        options=80000<LINKSTATE>
        ether 58:9c:fc:10:c1:10
        groups: tap vm-port
        media: Ethernet autoselect
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        Opened by PID 20571

ifconfig output on the FreeBSD guest showing static IP / interface configuration:

Code: 
 # ifconfig -a
vtnet0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
        ether 58:9c:fc:0d:ca:3b
        inet 10.0.100.33 netmask 0xffffff00 broadcast 10.0.100.255
        media: Ethernet autoselect (10Gbase-T <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
 
As is always the case, as SOON as I post and reach out for help I managed to figure it out -- turns out the issue was outside of this system altogether:

I observed that the arp entries for 10.0.100.33 were pointing to my another device on my network.
Turns out, that device had an address pool for mobile VPNs which contained the 10.0.100.33 network.

Modified the pool on the firewall, deleted the arp entry, now everything is working as expected.

My apologies for the false trouble report.
 
Zuna Mae said:
As is always the case, as SOON as I post and reach out for help I managed to figure it out
Click to expand...
That happens more often than you'd think. Trying to explain a problem to somebody else forces you to gather information, ordering it logically and suddenly spotting the issue.

en.wikipedia.org

Rubber duck debugging - Wikipedia

en.wikipedia.org en.wikipedia.org
In describing what the code is supposed to do and observing what it actually does, any incongruity between these two becomes apparent.
Click to expand...
 
You must log in or register to reply here.
Back
Top