bhyve Help with bhyve networking

S

simon

Hi friends,
I decided to use FreeBSD as a daily and it has been great so far (figured out lots on my own with some help from this great community of course!).
But I have something that has me tearing out my hair - I have VM's on bhyve and they run beautifully but for one thing: networking.
I cannot seem to figure out how to get them online!!!
This despite having almost the same system but on a server in the house that uses a cabled connection and works beautifully!

This problem is on my laptop.
Now, what do I do?
Can someone please help me and tell me what goes where or point out what I need to do?
This is driving me nuts guys!!
The host is FreeBSD 13.1 p6

Here are some of the configs/settings currently in place:

Code: 
 vm switch list
NAME    TYPE      IFACE      ADDRESS           PRIVATE  MTU  VLAN  PORTS
public  standard  vm-public  192.168.0.254/24  no       -    -     ue0 wlan0 em0

Code: 
ifconfig
em0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=481249b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LRO,WOL_MAGIC,VLAN_HWFILTER,NOMAP>
        ether b4:b6:86:fb:0a:83
        inet 192.168.89.101 netmask 0xffffff00 broadcast 192.168.89.255
        media: Ethernet autoselect
        status: no carrier
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
wlan0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether d4:6d:6d:ed:9c:cf
        inet 10.5.82.152 netmask 0xfffffc00 broadcast 10.5.83.255
        groups: wlan
        ssid EXECUTIVE_WIFI channel 149 (5745 MHz 11a) bssid 0c:f4:d5:1d:2a:ec
        regdomain FCC country US authmode WPA2/802.11i privacy ON
        deftxkey UNDEF AES-CCM 3:128-bit txpower 23 bmiss 10 mcastrate 6
        mgmtrate 6 scanvalid 60 wme roaming MANUAL
        parent interface: iwm0
        media: IEEE 802.11 Wireless Ethernet OFDM/54Mbps mode 11a
        status: associated
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
vm-public: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 5e:a0:3e:91:0f:31
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        groups: bridge vm-switch viid-4c918@
        nd6 options=9<PERFORMNUD,IFDISABLED>
 
The problem here is probably caused by a lot of wireless network interfaces refusing to spoof the source MAC address. Which needs to be done when bridging the two networks. A bridge is a layer 2 connection, which means outgoing packets are going to have the source MAC address of the VM's interface, not the MAC address of the wlan0 interface.

If you just need to have your VMs access to the rest of your network (or the internet) it's probably best to set up NAT and translate the traffic of the vm-public 'switch' to the wlan0 interface.
 
SirDice said:
The problem here is probably caused by a lot of wireless network interfaces refusing to spoof the source MAC address. Which needs to be done when bridging the two networks. A bridge is a layer 2 connection, which means outgoing packets are going to have the source MAC address of the VM's interface, not the MAC address of the wlan0 interface.

If you just need to have your VMs access to the rest of your network (or the internet) it's probably best to set up NAT and translate the traffic of the vm-public 'switch' to the wlan0 interface.
Click to expand...
Hey!
Thank you so much for this.
Yes! That is what I need/want - to have the VM's get online and be accessible from the network!
Do you have a sample config I could go with?
Even better - please use my interfaces in the post to sort of "mock" one up so I see what I have to do - I could use "PF".
I'd really appreciate it!
 
SirDice said:
The problem here is probably caused by a lot of wireless network interfaces refusing to spoof the source MAC address. Which needs to be done when bridging the two networks. A bridge is a layer 2 connection, which means outgoing packets are going to have the source MAC address of the VM's interface, not the MAC address of the wlan0 interface.

If you just need to have your VMs access to the rest of your network (or the internet) it's probably best to set up NAT and translate the traffic of the vm-public 'switch' to the wlan0 interface.
Click to expand...
Great, now I have established that I can indeed use PF for this.
1 - Would the "vm-public" switch having an IP be a concern or should I create it without one?
2 - Can a DHCP server on the host assign addresses to the guests once the NATis set up?
 
1) It doesn't need one
2) if you want to run a DHCP server then you would need an IP address on the 'switch'. And attach the DHCP service to that IP address, so it only responds to VMs requesting IPs. You don't want to send this out on your LAN.
 
SirDice said:
1) It doesn't need one
2) if you want to run a DHCP server then you would need an IP address on the 'switch'. And attach the DHCP service to that IP address, so it only responds to VMs requesting IPs. You don't want to send this out on your LAN.
Click to expand...
Ah!
What a surprise man!
Would you believe that option 2 is exactly what I went with yesterday?
I just thought "lets try this", I tried it and it worked.
Then I log on here and find you've given the exact same advice!
My reason for choosing it was because we have a corporate DHCP server and I have seven guest machines in bhyve so I just wanted the switch to sort of "hand off" everything.
It was a bit of bother figuring out but I finally did and it worked!
You can imagine my surprise when I thought: "I haven't checked for advice, let me check as I thank him (you) while updating him" only to find you recommending what I did!
Thank you so much SirDice and everyone.
I'll keep you updated as I go along!
 
next question: How do I automatically add the 'tap' devices to the switch whenever they're "created"?
 
I'm searching how to in this forum and find your post. Docyou mind how you did that? Maybe a command or a configurations are more helpful. I can conceptually understand what you say but have a hard time conbert that to the execution. Thanks so much. I appreciate it.
 
I am trying to setup Whonix gateway and Whonix workstation in bhyve. After long hours of playing with vm-bhyve switches i just decided to use native ifconfig bridge. Currently i am having three VMs. Archlinux, WGateway and WWorkstation. My ifconfig has been build like:

Code: 
ifconfig bridge0 create
ifconfig bridge0 addm re0 up


Create tap interfaces:
Code: 
ifconfig tap0 create
ifconfig bridge0 addm tap0 up

To make it persistent:
My /etc/rc.conf is
Code: 
cloned_interfaces="bridge0 tap0"
ifconfig_bridge0="addm tap0 addm re0 up"
ipfw_enable="NO"
firewall_enable="NO"
gateway_enable="YES"

My /boot/loader.conf has
Code: 
if_bridge_load="YES"
bridgestp_load="YES"

My Gateway VM config has:
Code: 
             loader="uefi"
             cpu=2
             memory=2G
             network0_type="virtio-net"
             network0_device="tap0"
             network1_type="virtio-net"
             network1_switch="internal"  # for Workstation
             disk0_type="virtio-blk"
             disk0_name="disk0.img"
I was testing tap0 by temporarly adding to my archlinux .conf file and the arch VM is getting network, therefore tap0 is working fine. However gateway not.
I was also able to getting network with
Code: 
vm switch create -t manual -b bridge0 public
vm switch create -t manual -b bridge0 external
vm switch create internal
but the result had the same. Working network with arch VM but not working with WhonixGateway. So my question is what else need to configure in the host or in gateway to have network??
 
In your setup, this may be wrong:
network0_device="tap0"
You should connect to the bridge name in your vm switch create: public or external or the name you have.

Example, if you:
Code: 
vm switch create -b bridge0 external

You need:
Code: 
network0_device="external"

But.. why not make a vm switch to your re0 interface direct?
 
amilis said:
In your setup, this may be wrong:
network0_device="tap0"
You should connect to the bridge name in your vm switch create: public or external or the name you have.

Example, if you:
Code: 
vm switch create -b bridge0 external

You need:
Code: 
network0_device="external"

But.. why not make a vm switch to your re0 interface direct?
Click to expand...
Sorry, my mistake, i did not explain before... The native ifconfig method and the vm bhyve switch method are two different things. If you dont use vm-bhyve manager you can still do it with bridge0 and tap0. So i was destroying all vm switches and was creating bridge0 and tap0, not using vm-bhyve switches for networking at all. However i did try with "external" as u mentioned.... And also does work ONLY with archlinux VM but not with Whonix Gateway.
If u use vm switches then obviously u no need to create bridge and tap as it will created by vm-bhyve.
 
Ok.. I don’t use Whonix Gateway, so can’t say if there is some configurations to make. But do you get any connection at all in the interface in Whonix Gateway? If you run tcpdump on the interface that is connected to, do you see any traffic at all? Or just empty?
 
amilis said:
Ok.. I don’t use Whonix Gateway, so can’t say if there is some configurations to make. But do you get any connection at all in the interface in Whonix Gateway? If you run tcpdump on the interface that is connected to, do you see any traffic at all? Or just empty?
Click to expand...
I dont get any connection in whonix gateway, however if i add tap0 on my archlinux VM conf file, it does work. So the interface itself is okay. The issue might be with Whonix Gateway diskimage. As no official diskimge of Whonix exists for bhyve, i was using .qcow2 and convert it to .raw. Alternativatelly also was trying convert the .ova file into .raw. Both method does work, the VM will boot in bhyve, but no network.
 
Hmm.. interesting..
But, as you sad: “however if i add tap0 on my archlinux VM conf file, it does work”
So, the network works.

In your vm folder and .config/system.conf, you don’t have anything left from the old destroyed vm switch? I don’t think so as it works. But..

Sound like something with Whonix Gateway as you mention. And I have no clue about that! :/
 
kceiw said:
I'm searching how to in this forum and find your post. Docyou mind how you did that? Maybe a command or a configurations are more helpful. I can conceptually understand what you say but have a hard time conbert that to the execution. Thanks so much. I appreciate it.
Click to expand...
I am so sorry - I got sidetracked by life here in Kenya.
I just logged on and saw this.
Here's what I added to /etc/rc.conf:

vm switch create public
vm switch add public <network interface of host>

I then added an IP to that switch and natted any guest to it (with the external interface being the switch):
ext_if = "<switch's name here"
icmp_types = "echoreq"
allproto = "{ tcp, udp, ipv6, icmp, esp, ipencap }"
privnets = "{ 127.0.0.0/8, 192.168.0.0/16, 172.16.0.0/12, 10.0.0.0/8 }"
ext_addr = "158.247.196.143"
int_if = "{vm1, vm2}"
jail_net = "$int_if":network

nat on $ext_if from !nat on $ext_if from !($ext_if) to any -> ($ext_if)($ext_if) to any -> ($ext_if)

I'm pretty sure it can be done better but that's basically it.
 
You must log in or register to reply here.
Back
Top