I'm planning an enterprise edge setup and I'm deciding between FreeBSD/bhyve and Linux KVM as the hypervisor. I'd like to hear from people who run bhyve in production, ideally in a similar role.
Goal
One physical edge host (a second node for HA possibly later) running three VMs:
My priorities are security (small attack surface), stability, low overhead, and predictable behavior over a long uptime.
Hardware
Where I'd like your experience
Benchmarks, configs (bhyve args, loader.conf, sysctls) and war stories would help a lot. Thanks!
Two quick notes:
Goal
One physical edge host (a second node for HA possibly later) running three VMs:
- Firewall: [pfSense / OPNsense / FreeBSD+pf]
- WAF: [Coraza / Reverse proxy: [nginx ]
My priorities are security (small attack surface), stability, low overhead, and predictable behavior over a long uptime.
Hardware
- CPU: 2620 v3
- RAM: 48 GB
- Storage: 2x SSD (planning a ZFS mirror)
- WAN: 1 Gbps
- LAN: 3 Gbps [3x 1GbE ports, termination on the proxy or WAF?]
Where I'd like your experience
- Networking: At 1G WAN and about 3G LAN, would you pass the physical NICs through to the firewall VM (VT-d), or use virtio-net with if_bridge/vale/netgraph? What throughput, pps and CPU overhead did you see?
- Passthrough stability: Which NIC models work well or badly? Any issues with device reset, interrupts or MSI-X?
- Security: How do you harden the host (Capsicum, minimal services, management-only interface)? Have bhyve security advisories affected you, and how quickly were fixes available?
- HA and failover: With no native clustering, how do you handle host failure? ZFS replication with scripted restart, CARP between two hosts? How long did recovery take?
- Management tooling: vm-bhyve, CBSD or your own scripts? Which has been the most reliable?
- Upgrades: How painful are FreeBSD upgrades with running VMs? Any bhyve regressions between releases?
- Guest compatibility: Any trouble running Linux or FreeBSD-based firewall guests (clock drift, virtio quirks, console)?
- Compared with KVM: If you've used both, why did you stay with or leave bhyve?
- Monitoring and backups: What do you use for VM metrics, logs and backups (sanoid/syncoid, zrepl)?
- Biggest regrets: What do you wish you had known before going live?
Benchmarks, configs (bhyve args, loader.conf, sysctls) and war stories would help a lot. Thanks!
Two quick notes:
- 48 GB is plenty for three VMs at this traffic level (for example 4 GB firewall, 8-16 GB WAF, 8 GB proxy, with the rest for ZFS ARC and headroom).
- If you have only 2 SSDs, a ZFS mirror is the right call. Mention whether they're enterprise drives with power-loss protection.