bhyve Bhyve finally got nested virtualization.

Here we go again.

After a full month of work,night and day,I've implemented the nested virtualization for (qemu) + bhyve and it works with every permutation of FreeBSD,Linux and Windows,as you can see from the pictures below :

Permutation 1 :

L0=FreeBSD 16.0-CURRENT
L1=Windows 11
L2=FreeBSD 15.2 (on top of Hyper-V and WSL2)


L0=FreeBSD-16.0-CURRENT_L1=Windows-11_L2=FreeBSD-15.2.jpg


Permutation 2 :

L0=FreeBSD 16.0-CURRENT
L1=KUbuntu-Noble-Wayland-KDE
L2=FreeBSD-15.1-RELEASE


L0=FreeBSD-16.0-CURRENT_L1=KUbuntu-Noble-Wayland-KDE_L2=FreeBSD-15.1-RELEASE.jpg


Permutation 3 :

L0=FreeBSD 16.0-CURRENT
L1=FreeBSD-15.1-RELEASE
L2=Linux omarchy


L0=FreeBSD-16.0-CURRENT_L1=FreeBSD-15.1-RELEASE_L2=omarchy.jpg



This has been the most complicated, delicate, treacherous project that I've worked on so far. But what matters is that I had fun and I've learnt a lot.

So. See you on the next project.
 
I don’t know to what extent they rely on AI.
Code:
AI assistance: this change was developed with an AI assistant
(Anthropic's Claude) under my direction, as keelOS itself is.
The assistant wrote the code and this description, working from the keelOS
implementation and the SDM. The tests in the test plan were run on real hardware,
and the emulation was compared with the SDM as described there.
 
Frankly speaking, this generated report is pretty hard to swallow. It'd be silly to ask yet another AI to summarize something that Claude generated.

Instead, it'd be best to summarize 2-3 items that are worth the attention and feed more information in smaller pieces as needed or as new information is uncovered.
 
What I published so far was a working diary, not something another person can rebuild from. So here is a small kit:

https://github.com/Marietto2008/bhyve-nested-strumenti/tree/main/kit

- **host-manifest.sh** records CPU, microcode revision, board, BIOS, VMX capability MSRs, kernel, and the hashes of the vmm module, bhyve and QEMU, so results from different machines can be compared.

- **smoke-test.sh** is one command. It boots the official FreeBSD 15.1 cloud image as an L1 with nesting, checks that the L1 can use VMX, boots an L2 kernel inside it, runs the benchmark, and prints PASS/FAIL. On my machine it ends with ALL PASSED.

- **README.md** lists the exact branches and commits and how to build and load the module.

- **KNOWN-ISSUES.md** says up front what still fails.

If you try it, please attach the manifest and the smoke-test output.

For the record, my host is consumer hardware: an i9-9900K on a Gigabyte Z390 AORUS PRO, BIOS F12g (2020), running FreeBSD 16-CURRENT. The CPU has VMCS shadowing but no APICv and no posted interrupts, and that shapes the results below.

**On AI use, since that is what the policy discussion is about.**

Code, the test scripts and the performance report were produced with Claude 5.5 with the help of Astra 6. Design reviews during development were also done by AI models, not by people. KVM (GPL) was studied for its behaviour, never copied. Commits carry a "Co-Authored-By: Claude" line. Nothing here has had independent human review yet, and I'd welcome it.

**The three performance findings worth attention** (same static benchmark binary at every level):

1. One level of bhyve is essentially free. A FreeBSD or Ubuntu guest runs CPU-bound code, fork/exec and memcpy at host speed, and exposing VMX to the guest costs nothing measurable.

2. In every nested (L2) setup the dominant cost is waking another CPU. A pipe round trip between two CPUs takes 3.8 µs on the host and 1–1.9 ms in the L2, whether the L1 hypervisor is bhyve, KVM or Hyper-V/WSL2. Without APICv every IPI goes through L0. This is the first thing to fix.

3. The L1 hypervisor matters more than bhyve underneath. With the same FreeBSD 15.1 L2 image, first-touch page faults are 4× slower than on the host under KVM, but 71× slower under bhyve (EPT violations resolved across two levels). That is the second target.

From here on I'll post one item at a time, when there is something new, instead of long reports. Raw data and scripts are in kit/full-report/.
 
bhyve redone in rust

Impressive work getting Windows 11 24H2 up on a Rust bhyve — is the code public ? One note: "A hypervisor has been detected" only shows that Windows runs as a guest, and VBS reads "Not enabled", so Hyper-V isn't running inside it. Nested VT-x would show up as Hyper-V/VBS running in the guest (VBS "Running", bcdedit hypervisorlaunchtype Auto), or WSL2 / a Hyper-V VM starting. If you get there, I'd be glad to compare numbers with the same smoke test.
 
ok I should've mentioned this is it's L0 = Freebsd 15.1 p7 , L1= Ubuntu 24.04 , L2=Win 11 , that's why 'VBS reads "Not enabled"'

Thanks for clarifying — that makes sense: the Windows you showed is the L2 under KVM, so VBS being off is expected. Nice to see KVM running inside a Rust bhyve, and on AMD too. My work is Intel-only so far; the case I focused most of the efforts is Hyper-V itself as the L1 (Windows with VBS/WSL2 inside bhyve). When your code is public I'd be glad to run the same smoke test on both: KVM-in-bhyve is one of the permutations I measured, so we'd have a direct comparison.
 
Thanks, that's a useful overview. A few things could help both of us: which TSC_AUX host panic did you hit in stock vmm.ko? My nested work started from stock vmm and TSC handling has bitten me before. Also, for bhyve-in-bhyve I measure first-touch page faults in the L2 at ~71× slower than the host (KVM as L1: ~4×); I'd be curious what your nested EPT path gives on the same test (claude-banco in the kit). On my side the unusual case is Hyper-V as the L1 (Windows with VBS/WSL2). If you ever publish the vmm.ko patches, comparing the two nested implementations would be worth it.
 
Back
Top