Solved bhyve networking

A

almostunix

Running into an issue where Bhyve Guests are not receiving internet from the bridged ethernet "bhyve0" bridge. Taps auto-ad as expected to the bridge, and both the taps and the Ethernet connection show active on ifconfig.
I believe this is some sort of DHCP issue, if someone could point me to the right area to look.

The issue persists across guests, FreeBSD, RH Linux and Win11 (which runs much better within Bhyve than on baremetal. )

My current workaround is to passthrough a network adapter via PCI passthrough at boot. (that works well but would like to nail down why the bridge wont work as expected. )

I probably misconfigured (or forgot to configure) something. but it seems like its a DHCP issue with the bridge. (all other devices on the network are receiving an IP from the PF sense router (seperate box) without issue, including the host.

Any help much appreciated.

Thanks,
 
tried to remove hardware identifiers where possible. happy to send in a PM if needed, just didnt want it out there scrapable.
Code: 
[B]em0:[/B] flags=1008902<BROADCAST,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=4e500bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
 ether (edit here)
 media: Ethernet autoselect (1000baseT <full-duplex>)
 status: active
 nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
[B]lo0:[/B] flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
 options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
 inet 127.0.0.1 netmask 0xff000000
 inet6 ::1 prefixlen 128
 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
 groups: lo
 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
[B]wlan0:[/B] flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
 options=0
ether [B](edit here)[/B]
inet ([B]IP here.77)[/B] netmask 0xffffff00 broadcast [B](IP here[/B])
 inet6 [B](edit here) [/B]prefixlen 64 scopeid 0x3
inet6 [B](edit here)[/B] prefixlen 64 autoconf pltime 1800 vltime 1800
 groups: wlan
ssid "[B]network name here[/B]" channel 36 (5180 MHz 11a vht/80+) bssid (edited [B]here[/B])
 regdomain FCC country US authmode WPA2/802.11i privacy ON
 deftxkey UNDEF AES-CCM 2:128-bit AES-CCM ucast:128-bit txpower 17
 bmiss 7 mcastrate 6 mgmtrate 6 scanvalid 60 -ampdutx ampdurx
 ampdulimit 64k -amsdutx amsdurx shortgi -ldpctx ldpcrx -uapsd vht
 vht40 vht80 vht160 -vht80p80 wme roaming MANUAL
 parent interface: iwlwifi0
 media: IEEE 802.11 Wireless Ethernet VHT mode 11ac
 status: associated
 nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
[B]bhyve0:[/B] flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
 description: bhyve manager bridge
 options=10<VLAN_HWTAGGING>
ether [B](ID here)[/B]
 id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
 member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
         ifmaxaddr 0 port 1 priority 128 path cost 20000
 groups: bridge
 nd6 options=9<PERFORMNUD,IFDISABLED>
[B]tailscale0:[/B] flags=1008003<UP,BROADCAST,MULTICAST,LOWER_UP> metric 0 mtu 1280
 options=4080000<LINKSTATE,MEXTPG>
 groups: tun
 nd6 options=109<PERFORMNUD,IFDISABLED,NO_DAD>
Opened by PID 99693
 
here is the updated one with tap attached. in theory, once this works for 1 tap on the bridge, any number of VMs supported by the hardware should work as well.

em0: flags=1008902<BROADCAST,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
options=4a500b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,HWSTATS,MEXTPG>
ether (Redacted)
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet 127.0.0.1 netmask 0xff000000
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
wlan0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=0
ether (Redacted)
inet (Redacted) netmask 0xffffff00 broadcast 54.56.5.255
inet6 (Redacted) prefixlen 64 scopeid 0x3
inet6 (Redacted) prefixlen 64 autoconf pltime 1800 vltime 1800
groups: wlan
ssid "(Redacted)" channel 36 (5180 MHz 11a vht/80+) bssid (Redacted)
regdomain FCC country US authmode WPA2/802.11i privacy ON
deftxkey UNDEF AES-CCM 3:128-bit AES-CCM ucast:128-bit txpower 17
bmiss 7 mcastrate 6 mgmtrate 6 scanvalid 60 -ampdutx ampdurx
ampdulimit 64k -amsdutx amsdurx shortgi -ldpctx ldpcrx -uapsd vht
vht40 vht80 vht160 -vht80p80 wme roaming MANUAL
parent interface: iwlwifi0
media: IEEE 802.11 Wireless Ethernet VHT mode 11ac
status: associated
nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
bhyve0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: bhyve manager bridge
options=10<VLAN_HWTAGGING>
ether (Redacted)
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 6 priority 128 path cost 2000000
member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 1 priority 128 path cost 2000000
groups: bridge
nd6 options=9<PERFORMNUD,IFDISABLED>
tailscale0: flags=1008003<UP,BROADCAST,MULTICAST,LOWER_UP> metric 0 mtu 1280
options=4080000<LINKSTATE,MEXTPG>
groups: tun
nd6 options=109<PERFORMNUD,IFDISABLED,NO_DAD>
Opened by PID 21
tap0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: "win11 VM"
options=4080000<LINKSTATE,MEXTPG>
ether (Redacted)
groups: tap
media: Ethernet 1000baseT <full-duplex>
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
Opened by PID 58963
 
Yes, that looks much better. Have you tried running tcpdump(1) on the bhyve0 bridge and/or the em0 'uplink' interface to see if you can actually spot the DHCP DISCOVER/REQUEST etc. being sent from the VMs? And hopefully the DHCP OFFER being sent back from your DHCP service? Actually looking at the packets with tcpdump(1) can be super enlightening.

Dynamic Host Configuration Protocol - Wikipedia

en.wikipedia.org en.wikipedia.org
 
I can take a look. I temporarily tried putting dhcp up via ifconfig (in rc.conf). On bootup that showed the dhcp search but it timed out after like 5 tries.
 
almostunix said:
On bootup that showed the dhcp search but it timed out after like 5 tries.
Click to expand...
That's DHCP on one of the VMs attached to the bhyve0 bridge? You should, at the very least, be able to see that DHCP DISCOVER appear on the bhyve0 interface. Then you basically "follow" the packets, tcpdump(1) em0, then have a look on the pfSense box (I presume it does DHCP for your network) and check to see if it's coming in. If you're methodical about this you can pin down exactly where it's failing.

If it's a FreeBSD VM with ifconfig_someint="DHCP" you can trigger the DHCP request again, after the system booted up, by running service dhclient restart <someint>. You don't have to restart the entire VM.
 
I went through and cleaned up a few things in rc.conf, added a line to enable DHCP on the bridge correctly, disabled dnsmasq for bhyve guests, and the bridge is now working as expected for all clients, BSD, Linux, and Windows, and they are all receiving IPs via DHCP (from PFsense box). marked as solved.
 
AndAlooongCameFreeBSD said:
almostunix would you mind to post the relevant parts of your known-good configuration?
Click to expand...
As an FYI, I often use Bhyvemgr as it neatly organizes the VMs and adds in the vm tap on demand. Assuming you have automatic tap adding for the vm... the below rc.conf config works well for me. I also have WiFi configured and working and use tailscale but those configs are not included below and didnt seem to impact bhyve bridge.

If using Windows VMs remember to install the RHEL ethernet driver on the guest.

cloned_interfaces="bridge0"
ifconfig_bridge0="DHCP"
ifconfig_bridge0_name="bhyve0"
ifconfig_bhyve0="addm em0 up"
ifconfig_bhyve0_descr="bhyve manager bridge"
 
You must log in or register to reply here.
Back
Top