bhyve Advice on Firewall Deployment Architecture

I wanted to get your opinion on an infrastructure decision I’m considering. I have an HP DL60 Gen9 server with multiple physical network interfaces, and my plan is to deploy it as a firewall appliance.

Instead of installing the firewall OS directly on bare metal, I’m considering installing a lightweight and reliable hypervisor first, and then running my OPNsense as a virtual machine. The plan is to assign the physical WAN/LAN NICs directly to the firewall VM (using PCI passthrough or a similar mechanism) to achieve near bare-metal performance.

Specifically, I wanted to ask:
  • Would you recommend using bhyve as the hypervisor for this use case?
  • In your experience, is bhyve stable and secure enough for a production firewall?
  • Or would you advise a different approach altogether?
I’d really appreciate your thoughts and guidance before I move forward.
 
I think their are not stable as 13.5 (freebsd.org says Legacy)
ok after eol i'll upgrade to in time stable version.
but first i need a verification from some experts for this scenario
 
I think putting a firewall in virtualization adds too much complexity. Firewall should be about rules. Don't make it too complicated.

I just fired my latest build. A Jetway NF-9G ITX board with Chelsio 40G adapter to top of rack..
The board was old but I bought it new in 2018. Been sitting and was only QM77 I had (needed>=PCI3x16 for Chelsio).
I repurposed an old HTPC chassis for it. I wanted a shelf sized firewall with QM3630 with good low power performance.

Six months later...
I was noticing hiccups on my network and noticed gateway/firewall was barfing. Hung at BIOS screen. Why is had rebooted several times I dunno.

Crux of the story. Server gear for a firewall is not outrageous but it is overkill.

There is no reason to consider any 13 Branch for a security device. 14-RELEASE works quite fine.
Plus you get any Bhyve improvements over the 13 branch that were not backported if you do go that route..
 
I kind of like having the NAT/gw/firewall as a virtual appliance, it doesn't feel overly complex. It is just an interface to another network.
 
I view firewalls as being in the same application category as time servers and name servers.

They are "core" infrastructure. You need to plan for the services that they furnish to be available at all times. i.e. most other applications will fail if your time servers, name servers, and firewalls are not running.

Before you decide where to site these core services, figure out precisely how you are going to make them genuinely reliable.

That problem is sufficiently complex that, in data centres, I have always used redundant physical servers for core infrastructure and plugged them directly into the principal backbone routers. I then test by simulating a full power outage recovery. It's common to see inter-dependencies you never considered, especially if you need a hypervisor running before anything else.

I'm not saying don't put all your firewalls into a hypervisor. Just make very sure that you test to make sure it does what you want when the chips are down.
 
That problem is sufficiently complex that, in data centres, I have always used redundant physical servers for core infrastructure and plugged them directly into the principal backbone routers. I then test by simulating a full power outage recovery. It's common to see inter-dependencies you never considered, especially if you need a hypervisor running before anything else.
I always run my firewalls on bare metal. Virtualization adds a layer than can have problems.

For an enterprise setup, I'd look at two machines and carp(4) interfaces. Test by powering each off by unplugging the power cable. Make sure the network hiccup is minimal if noticeable, and that the machine will recover fully when power is restored. This makes updates simple, take one machine down and update it. Repeat with the other once you've validated the updated machine.

I'm looking at something like this with a couple of Protectli boxes for home:

You wouldn't need the ifstated trick for an enterprise setup since you'd likely not have to rely on DHCP for your external IP address.
 
Thanks you all for your reply
yes I know put core security infrastructure in a hypervisor has some challenges but in my situation it should be like that.
I am looking for a right solution on hypervisor (bhyve or esxi or proxmox) simple secure reliable
 
I'm not sure, if it is a good idea to virtualise the firewall, protecting your enterprise Network. Call me old school, but I would use a firewall on bare metal.

As Jose an others said, there could be problems with the Hosts, even in FreeBSD with bhyve (as in 15.0-RELEASE with pci passthru) which could cause security issues.

This is the front line to the Internet, and should work as secure as possible. So a good piece of hardware with a good Firewall (e.g. vanilla FreeBSD or opnsense) should be a good choice.
 
As I researched I can create a Vswitch for each WAN and LAN totaly seperated and then port group and then assign to my Firewall VM.
all of LLMs verified that this config could be secure for this scenario
 
I'm not sure that LLMs should have more experience than real life workers running business for years.

Trust me, it is no good idea running the front line firewall on a virtualisation hosts.

But it is your project. I wouldn't do it.

Hackers are asking the same LLMs to hack your firewalll. ;)
 
Yes ur right, but I'm sure there is a solution for hardening this situation.
everything is changing It should be a solution Im not sure focusing on bare-metal is the right one.
 
Why add a full virtualization layer if you run FreeBSD (kind of...) on top of the VM, especially if you care about bare-metal-performance?
Just use jails and run PF and other services for your network without any overhead. You can even run multiple firewall-jails with pfsync and carp for redundancy.
 
Why are you putting the firewall into a VM on a single node? If the node breaks, you're gone anyway. You have no fault isolation. The way you're formulating this now, the hypervisor does nothing but increase your operational complexity leading to more points of potential failure. Don't do that.

If you're looking for a resilient solution for an enterprise environment, get OPNSense from Deciso with a proper support contract from them and put that product on two separate physical nodes (ensure redundant network paths, power supply etc.). OPNSense itself has features to allow it to run in high-available mode so you don't need to muddy those waters by leaning on HA-features from a virtualization layer underneath it.

If you already have a high-available virtualization platform in place and operationally taken care of, you could get away with running OPNSense on there in a single VM that gets protected by the platform. You don't, though, or you wouldn't be asking about this single HP box.

TLDR; You want a firewall to run on the metal and have full use of the hardware directly, run them in pairs if availability is a concern and get a support contract if this is business critical.

PS. Don't question the advice of actual living engineers from the trenches just because some biased LLM with the wisdom of a slot machine happens to disagree.
 
Two in fact. There is a new patch by the way, I will try it tonight.
That said, even if it's ok now, we must wait at least for the p1 update before to install 15.0-RELEASE, hoping it will correct these passthru problems.
Can you link the PRs in question? I'm on STABLE 15 so hopefully not needing to wait for too long.
 
Bash:
#!/bin/sh
# ========================================
# FreeBSD 13.5 Hypervisor Hardening Script - FINAL
# Secure, POSIX-compliant, with interface checks and snapshot
# ========================================

echo "=== Advanced Hypervisor Hardening Started ==="

# ----------------------------
# Configuration - Change as needed
# ----------------------------
MGMT_NIC="em0"
MGMT_IP="192.168.1.100/24"
MGMT_GATEWAY="192.168.1.1"

BRIDGES="bridge0 bridge1 bridge2"
TAPS="tap0 tap1 tap2"
WAN_NICS="em1 em2 em3"

# ----------------------------
# Helper function for error checking
# ----------------------------
check() {
    if [ $? -ne 0 ]; then
        echo "Error: $1" >&2
    fi
}

# ----------------------------
# 1️⃣ User and Access Management
# ----------------------------
echo "-> Configuring User and Access Management"
if ! id -u admin >/dev/null 2>&1; then
    pw useradd admin -d /home/admin -s /bin/sh -m
    echo "admin ALL=(ALL) NOPASSWD: ALL" > /usr/local/etc/sudoers.d/admin
    chmod 440 /usr/local/etc/sudoers.d/admin
fi
passwd -l root

# ----------------------------
# 2️⃣ SSH Hardening
# ----------------------------
echo "-> Hardening SSH"
cat << EOF > /etc/ssh/sshd_config
Port 2222
Protocol 2
PermitRootLogin no
PasswordAuthentication no
KbdInteractiveAuthentication no
ChallengeResponseAuthentication no
PubkeyAuthentication yes
UsePAM yes
AllowUsers admin
EOF
service sshd restart
check "SSHD restart"

# ----------------------------
# 3️⃣ Network Hygiene
# ----------------------------
echo "-> Configuring Management LAN only"
ifconfig $MGMT_NIC inet $MGMT_IP up
for nic in $WAN_NICS $BRIDGES $TAPS; do
    ifconfig $nic inet delete 2>/dev/null
done

# Disable routing
sysctl net.inet.ip.forwarding=0
sysctl net.inet6.ip6.forwarding=0
sysrc gateway_enable="NO"

# ----------------------------
# 4️⃣ PF Firewall
# ----------------------------
echo "-> Configuring PF Firewall"
cat << EOF > /etc/pf.conf
set block-policy drop
set skip on lo

block all

# Management SSH
pass in on $MGMT_NIC proto tcp to port 2222 keep state
# DNS and NTP outbound
pass out proto { tcp, udp } to port 53 keep state
pass out proto udp to port 123 keep state

# Optional: log blocked packets
block in log all
EOF
sysrc pf_enable="YES"
service pf restart
check "PF restart"

# ----------------------------
# 5️⃣ Kernel and Sysctl Hardening
# ----------------------------
echo "-> Applying sysctl hardening"
cat << EOF >> /etc/sysctl.conf
net.inet.ip.redirect=0
net.inet.ip.accept_sourceroute=0
net.inet.tcp.blackhole=2
net.inet.udp.blackhole=1
security.bsd.see_other_uids=0
security.bsd.see_other_gids=0
security.bsd.unprivileged_read_msgbuf=0
security.bsd.unprivileged_proc_debug=0
net.inet.icmp.drop_redirect=1
EOF
sysctl -f /etc/sysctl.conf

# ----------------------------
# 6️⃣ Disable unnecessary services
# ----------------------------
echo "-> Disabling unnecessary services"
sysrc sendmail_enable="NONE"
sysrc rpcbind_enable="NO"
sysrc ntpd_enable="NO"

service ntpd stop 2>/dev/null
service sendmail stop 2>/dev/null
service nfsd stop 2>/dev/null

# ----------------------------
# 7️⃣ Filesystem & ZFS
# ----------------------------
echo "-> Filesystem and permissions"
chmod 700 /root
chmod 750 /usr/local/etc
if zfs list >/dev/null 2>&1; then
    zfs set atime=off zroot
    echo "-> Creating ZFS snapshot"
    zfs snapshot zroot/ROOT/default@hypervisor-clean
fi

# ----------------------------
# 8️⃣ Interface and Bridge Checks
# ----------------------------
echo "-> Checking bridge and tap interfaces"
for iface in $BRIDGES $TAPS $WAN_NICS $MGMT_NIC; do
    if ifconfig $iface >/dev/null 2>&1; then
        status=$(ifconfig $iface | grep "status: active")
        if [ -n "$status" ]; then
            echo "Interface $iface: ACTIVE"
        else
            echo "Warning: Interface $iface: INACTIVE"
        fi
    else
        echo "Warning: Interface $iface: NOT FOUND"
    fi
done

# ----------------------------
# 9️⃣ Load essential kernel modules
# ----------------------------
echo "-> Loading necessary kernel modules"
sysrc kld_list="vmm if_bridge if_tap nmdm"

# ----------------------------
# 10️⃣ Final Report
# ----------------------------
echo "=== Hypervisor Hardening Completed ==="
echo "- Check: No IP on WAN interfaces or bridges"
echo "- SSH only accessible from $MGMT_NIC"
echo "- PF active and blocking all other traffic"
echo "- Bridge/tap interfaces checked above"
echo "- Backup snapshot exists (if ZFS)"

# Reboot prompt
echo -n "Do you want to reboot now? (y/n): "
read REPLY
case "$REPLY" in [Yy]*) reboot;; *) echo "Reboot skipped. Please reboot manually later.";; esac
 
Why are you using a hypervisor at all? This is completely pointless complexity for no tangible benefit if the plan is to run OPNSense.

PS. I'm not going to be your LLM's unpaid code review monkey. If you're going to vibe-code this stuff and sell it to an actual enterprise, you really should reconsider your line of work.
 
Back
Top