Howto: Bhyve, using OpenBSD as main firewall in FreeBSD

My idea is to run the latest OpenBSD in FreeBSD as a guest using bhyve. Then pass a network interface to the OpenBSD guest using PCI pass through. So we can use the OpenBSD guest as the firewall of the host FreeBSD system. I got this idea from reading the following thread. I had learned a lot from the handbook, wiki and the article Intel Wifi Via Bhyve on FreeBSD. And I had also learned from many other sources. Let us now proceed to the howto.

First, we identify the network interface that we would like to pass through to OpenBSD. In the following example, I use an Intel 82576L network interface. Identifying the interface:
Code:
# pciconf -lv

snippet
em1@pci0:5:0:0: class=0x020000 card=0x00000000 chip=0x10d38086 rev=0x00 hdr=0x00
  vendor  = 'Intel Corporation'
  device  = '82574L Gigabit Network Connection'
  class  = network
  subclass  = ethernet
snippet
Here I will use em1@pci0:5:0:0.

Now add the following lines to /boot/loader.conf.
Code:
vmm_load="YES"
nmdm_load="YES"
if_tap_load="YES"
if_bridge_load="YES"
pptdevs="5/0/0"
We need to reboot for PCI pass through to work. During boot, instead of attaching the em driver to the interface, we see the following.
Code:
pcib5: <ACPI PCI-PCI bridge> irq 19 at device 28.3 on pci0
pci5: <ACPI PCI bus> on pcib5
ppt0 port 0xb000-0xb01f mem 0xf7a00000-0xf7a1ffff,0xf7a20000-0xf7a23fff irq 19 at device 0.0 on pci5
Next, we need a working image of OpenBSD. Because of current limitation of bhyve, we have to obtain it using other virtualization software. Moreover, as we are interested in using OpenBSD as a firewall, we need only to install the basic components of OpenBSD. See post #4 for how to install OpenBSD in bhyve.
Also, remember to check /etc/ttys in OpenBSD to allow serial console. Otherwise, booting the OpenBSD image will stuck at the TIME and DATE line.

After obtaining such an image, say, obsd56.img, I put it in /root/bhyve/openbsd. Also create the following two files:
Code:
 1. /root/bhyve/openbsd/obsd56.map:

(hd0) /root/bhyve/openbsd/obsd56.img

2. /root/bhyve/openbsd/obsd56.in:

kopenbsd -h com0 -r sd0a (hd0,openbsd1)/bsd
boot
Now, install sysutils/grub2-bhyve and prepare the network interfaces.
Code:
# ifconfig tap0 create
# sysctl net.link.tap.up_on_open=1
# ifconfig bridge0 create
# ifconfig bridge0 addm tap0 up
# ifconfig bridge0 inet 192.168.0.1 netmask 255.255.255.0
Run the following commands:
Code:
# grub-bhyve -m /root/bhyve/openbsd/obsd56.map -r hd0 \
> -M 256M obsd56 < /root/bhyve/openbsd/obsd56.in > /dev/null
# bhyve -m 256M -A -H -P -s 0:0,amd_hostbridge -s 1:0,lpc \
> -s 2:0,passthru,5/0/0 -s 3:0,virtio-net,tap0 \
> -s 4:0,virtio-blk,/root/bhyve/openbsd/obsd56.img \
>  -l com1,stdio -W obsd56
You should be able to see OpenBSD booting. Log into the OpenBSD system at the login prompt. In OpenBSD, you should see two network drivers, vio0 and em0. First, create the following file /etc/hostname.vio0
Code:
inet 192.168.1.2 255.255.255.0
Now run sh /etc/netstart in OpenBSD.

Test the connection between the host and the guest. In FreeBSD, run ping -c 2 192.168.1.2 and in OpenBSD run ping -c 2 192.168.1.1.

If there is no problem, set up em0 as your main gateway in OpenBSD. See the OpenBSD FAQ for further information. Then set up firewall in the guest OpenBSD. Finally, rerun sh /etc/netstart.

Return to the host FreeBSD system. We need to add the following lines to /etc/rc.conf.
Code:
cloned_interfaces="bridge0 tap0"
ifconfig_bridge0="inet 192.168.1.1 netmask 255.255.255.0 addm tap0 up"
defaultrouter="192.168.1.2"
and add net.link.tap.up_on_open=1 to /etc/sysctl.conf.
We also have to make changes to /etc/resolv.conf if you are using dhcp. We can test our setting here by restarting network in FreeBSD. A simple test is to look up www.freebsd.org. And then ping www.freebsd.org.

Finally, create the following file, /etc/rc.local.
Code:
#/bin/sh
/usr/local/sbin/grub-bhyve -m /root/bhyve/openbsd/obsd56.map -r hd0 \
> -M 256M obsd56 < /root/bhyve/openbsd/obsd56.in > /dev/null
/usr/sbin/bhyve -m 256M -A -H -P -s 0:0,amd_hostbridge -s 1:0,lpc \
> -s 2:0,passthru,5/0/0 -s 3:0,virtio-net,tap0 \
> -s 4:0,virtio-blk,/root/bhyve/openbsd/obsd56.img \
>  -l com1,nmdm0A -W obsd56
Note that we replace stdio with nmdm0A in the above script. Reboot the host system to test that it works. Enjoy using a OpenBSD firewall in FreeBSD.
 
Last edited:
Wow great, I hoped my thread would be of use and now it gets even cited, I am honored :). Bhyve is a masterpiece. I cannot imagine what will
be possible in the future using that :).
 
To clarify:

Because of current limitation of bhyve, we have to obtain it using other virtualization software.

bhyve has been able to boot the OpenBSD install kernel (bsd.rd) for quite a while now. For example, to install from the 5.6 ISO image, you would put the following line into the obsd56.map file

Code:
(cd0) /path/to/install56.iso

The grub-bhyve command line would be

Code:
grub-bhyve -m /root/bhyve/openbsd/obsd56.map -r cd0 -M 256M obsd56
kopenbsd -h com0 /5.6/amd64/bsd.rd
boot

bhyve is started with the same command line with one addition:

Code:
-s 5:0,ahci-cd,/path/to/install56.iso

Later,

Peter.
 
Having some issues with bhyve, pci passthru mode isn't working, the -S switch is not available except in CURRENT but could not compile the grub2-bhyve port, due to other compiler issues in freebsd ports, but the compiler itself can build small binaries.
 
The idea isn't totally new ... I'm doing exactly the same thing on my home server for years, but using XEN and Linux. But great to hear it works with bhyve, as I'm planning to use FreeBSD on my new server, and this was what I had in mind anyways. I didn't think about using OpenBSD for the firewall VM, but this might be a good idea as well, so thanks, I'll come back soon and give it a try :)
 
I'd like to ask/clarify:

If the network card is passed through directly to OpenBSD, how do I assign IP addresses (say, 192.168...) to jails? (Normally, I'd use the interface, like em0, to assign to jails.)

Please clarify what is the right way to assign IP address to jails if the network card is no longer visible to the FreeBSD operating system.

Thanks!
 
I'd like to ask/clarify:

If the network card is passed through directly to OpenBSD, how do I assign IP addresses (say, 192.168...) to jails? (Normally, I'd use the interface, like em0, to assign to jails.)

Please clarify what is the right way to assign IP address to jails if the network card is no longer visible to the FreeBSD operating system.

Thanks!
I do not use jail. But I would guess you use the interface tap0 for jails. Also, I had encountered a problem that hot swap does not work when vmm.ko is loaded.
 
Back
Top