Working Bhyve-Bastille-General config files/network for reference

[vukodlak]

I decided to post a simple working setup from my FreeBSD box to help others that were struggling to find a working system to learn from like I was.
This isn't meant to be an end-all solution but rather a reference and learning opportunity and I would gladly accept help and suggestions on how to do things better!

So, here it is:

System:
FreeBSD 13.2 with ZFS
1 nic card set static to 192.168.1.11
Jail running on 192.168.1.91
Debian Bhyve VM at 192.168.1.3

Bhyve, Bastille, and OpenVPN with FiBs are installed.
I have Debian running with Bhyve and vncserver running inside of it. I VNC to it and use LXDE wm fullscreen on a 2nd monitor of my windows gaming box.

Jail start command:
bastille create -B azkaban3 13.2-RELEASE 192.168.0.3/8 em0

Bastille instructions:
Bastille

Bhyve instructions:
Bhyve


Conf files:

RC.CONF

#=============RC.CONF==============================================

zfs_enable="YES"
sshd_enable="YES"

######################## NETWORKING
gateway_enable="YES"
pf_enable="YES"
bastille_enable="YES"
vm_enable="YES"

###################### MAIN FREEBSD BOX
hostname="xxx.com"
ifconfig_em0="inet 192.168.1.11 netmask 255.255.255.0"
defaultrouter="192.168.1.1"

##################### BASTILLE NETWORKING
cloned_interfaces="lo1"
ifconfig_lo1_name="bastille0"

##############################SetFiB VPN and Bhyve vm stuff####################
static_routes="vpn"
route_vpn="-fib 1 -net 0.0.0.0/0 192.168.1.254"

vm_dir="/home/vms"

/boot/loader.conf

net.fibs=2
net.add_addr_allfibs=1
vmm_load="YES"

pf.conf

ext_if="em0"

set block-policy return
scrub in on $ext_if all fragment reassemble
set skip on lo

# Forward incoming traffic on port 81 to the jail at 192.168.1.91:80 and port 5901 to the Debian VM
rdr pass on $ext_if proto tcp from any to any port 81 -> 192.168.1.91 port 80
rdr pass on $ext_if proto tcp from any to any port 5901 -> 192.168.1.3 port 5901


# Allow outgoing traffic from the jail to the internet
pass out on $ext_if from 192.168.1.91 to any

block in all
pass out quick keep state
antispoof for $ext_if inet
pass in inet proto tcp from any to any port ssh flags S/SA modulate state
pass in inet proto tcp from any to any port 8096 flags S/SA modulate state

OpenVPN:
start with setfib 1 openvpn -c /home/openvpn.ovpn
anything you start on the main box with setfib 1 in front of it like setfib 1 qbittorrent-nox for example will use the vpn on Tun0
you can check if the VPN is working by setfib 1 curl icanhazip.com


The one issue I have is the Jails and the VMs being on the same network as the outside card em0.
For the life of me i cannot get a vnet to work for the jails or the VM. Would be nice to have the jail and bhyves running on a completely separate network like 10.0.0.0/24 instead of having to reroute them in the firewall for incoming services like nginx etc. Technically, firewall isn't meant to be used for routing is it?