Solved Cannot get bhyve guest network working

fnoyanisi

fnoyanisi

Hi there,

I have a VM running FreeBSD-CURRENT in bhyve but I could not get the network working properly.

I created tap0 and bridge0 interfaces as described in the relevant chapter of the handbook. I have only a wireless NIC, so I followed the advice in bhyve wiki page and created proper pf(4) rules (the suggested rule-set is not correct, already filed a bug report for that, and if somebody will look at it, hopefully the pf rules in the wiki page will be updated).

This is my ifconfig(8)
Code: 
em0: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
   options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>
   ether b8:6b:23:1c:44:17
   nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
   media: Ethernet autoselect
   status: no carrier
iwn0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 2290
   ether 88:53:2e:14:73:dc
   nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
   media: IEEE 802.11 Wireless Ethernet autoselect mode 11ng
   status: associated
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
   options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
   inet6 ::1 prefixlen 128
   inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
   inet 127.0.0.1 netmask 0xff000000
   nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
   ether 02:b2:10:28:d0:00
   nd6 options=9<PERFORMNUD,IFDISABLED>
   id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
   maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
   root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
   member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
    ifmaxaddr 0 port 5 priority 128 path cost 2000000
   member: wlan0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
    ifmaxaddr 0 port 6 priority 128 path cost 33333
tap0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
   options=80000<LINKSTATE>
   ether 00:bd:3b:15:00:00
   nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
   media: Ethernet autoselect
   status: active
   Opened by PID 1473
wlan0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
   ether 88:53:2e:14:73:dc
   inet 192.168.1.67 netmask 0xffffff00 broadcast 192.168.1.255
   nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
   media: IEEE 802.11 Wireless Ethernet MCS mode 11ng
   status: associated
   ssid TELECOM-U5QA29 channel 8 (2447 MHz 11g ht/20) bssid 9c:c1:72:0d:bd:cc
   country US authmode WPA2/802.11i privacy ON deftxkey UNDEF
   TKIP 2:128-bit powersavemode CAM powersavesleep 100 txpower 15
   bmiss 10 scanvalid 60 bgscan bgscanintvl 300 bgscanidle 250
   roam:rssi 7 roam:rate 64 protmode CTS ampdulimit 64k ampdudensity 8
   -amsdutx amsdurx shortgi wme roaming MANUAL
ue0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
   ether 02:80:37:ec:02:00
   nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
So, tap0 and bridge0 are created...

I start the VM as follows
Code: 
./vmrun.sh -f FreeBSD-CURRENT.img freebsd-current
and run
Code: 
dhclient vtnet0
inside the VM. None of DHCP requests get a reply....

When I use tcpdump to monitor wlan0 and tap0, I can see that DHCP requests are coming from tap0 and passed to wlan0, but I get no replies...

tcpdump for wlan0
Code: 
root@fbsd:/home/fnoyanisi/extra/byhve # tcpdump -i wlan0 port 67 or port 68
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on wlan0, link-type EN10MB (Ethernet), capture size 65535 bytes
00:25:08.519149 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 00:a0:98:bb:af:1f (oui Unknown), length 300
00:25:11.612258 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 00:a0:98:bb:af:1f (oui Unknown), length 300
00:25:18.635172 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 00:a0:98:bb:af:1f (oui Unknown), length 300
00:25:25.736099 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 00:a0:98:bb:af:1f (oui Unknown), length 300
^C
4 packets captured
26 packets received by filter
0 packets dropped by kernel
tcpdump for tap0
Code: 
root@fbsd:/home/fnoyanisi # tcpdump -i tap0
tcpdump: WARNING: tap0: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tap0, link-type EN10MB (Ethernet), capture size 65535 bytes
00:25:07.880578 ARP, Reply 192.168.1.66 is-at 00:ee:bd:a2:f8:cf (oui Unknown), length 28
00:25:08.519103 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 00:a0:98:bb:af:1f (oui Unknown), length 300
00:25:11.612214 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 00:a0:98:bb:af:1f (oui Unknown), length 300
00:25:18.635130 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 00:a0:98:bb:af:1f (oui Unknown), length 300
00:25:25.736063 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 00:a0:98:bb:af:1f (oui Unknown), length 300
^C
5 packets captured
5 packets received by filter
0 packets dropped by kernel

So, why the VM cannot get any DHCP replies from the AP?

Thanks
 
Well, a wlan cannot have more than one MAC address, i.e. generally speaking, you cannot use a bridge with wifi (there exist some workarounds though). The easiest way is setting up NAT - what I do in the exactly same setup. The relevant part of my /etc/rc.conf is the following:
Code: 
firewall_enable="YES"
firewall_type="open"
firewall_quiet="NO"
natd_enable="YES"
natd_interface="wlan0"
natd_flags="-u -m -dynamic"
tcp_extensions="NO"
network_interfaces="tap0 lo0"
defaultrouter="NO"
static_routes=""
gateway_enable="YES"
router_enable="NO"
mrouted_enable="NO"
ipxgateway_enable="NO"
ipxrouted_enable="NO"
arpproxy_all=""
forward_sourceroute="NO"
accept_sourceroute="NO"
 
I am no networking expert, so there is a bit of struggle for me....

I added NAT rules into /etc/pf.conf as suggested by the bhyve wiki (note that wiki page has bridge0:network, whereas the correct form is (bridge0:network) )
Code: 
nat on wlan0 from (bridge0:network) to any -> (bridge0)
pass out all
pass in all

I have applied the suggested changes in Section 21.7.7 of FreeBSD Handbook to get the "Persistent Configuration" done, i.e.
/etc/sysctl.conf
Code: 
net.link.tap.up_on_open=1
/boot/loader.conf
Code: 
vmm_load="YES"
nmdm_load="YES"
if_bridge_load="YES"
if_tap_load="YES"
/etc/rc.conf
Code: 
pf_enabled="YES"
cloned_interfaces="bridge0 tap0"
ifconfig_bridge0="addm igb0 addm tap0"
I think these should be enough to enable NAT on wlan0?
 
After a bit of googling and reading, I came to know that following steps from the handbook will not work if your only NIC is wlan0.
There is an email I sent to freebsd-virtualization mailing list (in short, Trent suggests me to have a at this)

Here is the steps one need to follow to get bhyve(8) working with wlan0 interface (should work for FreeBSD guests).
  1. Create tap0 interface as described here. Do not create bridge0, skip steps related to it.
  2. Assign a static IP address to tap0.
  3. Use static network configuration, which is described here, for the vtnet0 interface on guest OS.
  4. Use pf(4) to redirect/NAT traffic from tap0 to wlan0.
Below is some key files on the guest and host operating systems;

Guest OS
Code: 
test@current:~ % cat /etc/resolv.conf
nameserver 8.8.8.8
test@current:~ % cat /etc/rc.conf
# interface
ifconfig_vtnet0="inet 10.0.0.2 netmask 255.255.255.0"
defaultrouter="10.0.0.1"

hostname="current"
sshd_enable="YES"
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="AUTO"
test@current:~ % ifconfig vtnet0
vtnet0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
   options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
   ether 00:a0:98:bb:af:1f
   inet 10.0.0.2 netmask 0xffffff00 broadcast 10.0.0.255
   nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
   media: Ethernet 10Gbase-T <full-duplex>
   status: active
test@current:~ %

Host OS
Code: 
~ % tail -n 4 /boot/loader.conf
# bhyve
vmm_load="YES"
nmdm_load="YES"
if_tap_load="YES"
~ % tail -n 11 /etc/rc.conf
# required for bhyve
# make sure tapX IP address matches the one in pf.conf
cloned_interfaces="tap0"
ifconfig_tap0="inet 10.0.0.1 netmask 255.255.255.0"

# Enable pf(4)
# required for bridge0 NAT
pf_enable="YES"
pf_rules="/etc/pf.conf"
pf_flags=""

~ % cat /etc/pf.conf
wlan_if="wlan0"

# create all VM interfaces in this subnet
vm_net="10.0.0.0/24"
vm_fbsd_current="10.0.0.2"

nat on $wlan_if from $vm_fbsd_current to !$vm_net -> $wlan_if

pass out all
pass in all
~ %

Things to note;
  • tap0 has the default route IP address for the guest OS
  • all tap0 traffic is redirected (with NAT) to wlan0.
 
Hi,
I applied the recipe but unfortunately my guest can not reach anything behind wlan0. I'll appreciate any help because I can not solve the issue by my self.
Ping request from the guest to the host default gateway fails:
Code: 
ping -c1 192.168.1.1
PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.

--- 192.168.1.1 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms

Code: 
tcpdump -i tap0 net 10.0.0.0/24
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tap0, link-type EN10MB (Ethernet), capture size 262144 bytes
21:38:23.157384 IP 10.0.0.2 > 192.168.1.1: ICMP echo request, id 1144, seq 1, length 64

Here some network configurations of the host:
Code: 
wlan0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
   ether b4:b6:76:ca:05:08
   inet 192.168.1.9 netmask 0xffffff00 broadcast 192.168.1.255
   nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
   media: IEEE 802.11 Wireless Ethernet MCS mode 11ng
   status: associated
   ssid na channel 4 (2427 MHz 11g ht/20) bssid 5c:35:3b:c3:7c:44
   regdomain ETSI country DE authmode WPA2/802.11i privacy ON
   deftxkey UNDEF TKIP 2:128-bit txpower 30 bmiss 10 scanvalid 60
   protmode CTS ampdulimit 64k ampdudensity 8 -amsdutx amsdurx shortgi
   -stbc wme roaming MANUAL
   groups: wlan
tap0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
   options=80000<LINKSTATE>
   ether 00:bd:88:f9:50:00
   inet 10.0.0.1 netmask 0xffffff00 broadcast 10.0.0.255
   nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
   media: Ethernet autoselect
   status: no carrier
   groups: tap

Code: 
netstat -nr4
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            192.168.1.1        UGS       wlan0
10.0.0.0/24        link#4             U          tap0
10.0.0.1           link#4             UHS         lo0
127.0.0.1          link#2             UH          lo0
192.168.1.0/24     link#3             U         wlan0
192.168.1.9        link#3             UHS         lo0
 
Using wlan0 is a bit tricky, and, to be honest, the method I outlined is not very feasible (e.g. if you have more than one guest, you may have issues)

I came to know that wlan is not the best interface to be used with bhyve(8). If you use FreeBSD on your laptop (like me), this becomes a pain in the arse :confused:....

You may want to flick an email to virtualization mailing list on the matter.
 
You must log in or register to reply here.
Back
Top