bhyve bhyve ported to Apple Silicon (Hypervisor.framework): boots Windows 11 ARM and FreeBSD/arm64, plus fixes we'd like to upstream

Hi all,

We have ported bhyve to macOS on Apple Silicon and would like to show it here, mostly because the work produced a set of patches against usr.sbin/bhyve that we want to send back, and we would like advice on how to do that well.

The project is hyperMAC: https://github.com/keelos-dev/hyperMAC
The bhyve part is in vm/bhyve (BSD-2-Clause, FreeBSD's files keep their own headers).

What it is

xhyve brought bhyve to Intel Macs in 2015. This is the arm64 counterpart, and as far as we know the first bhyve on Apple Silicon:

  • FreeBSD's usr.sbin/bhyve sources from head, unmodified, with a numbered patch series on top and a small compat layer for the headers and functions macOS lacks. Following head should stay cheap.
  • libvmmapi reimplemented over Hypervisor.framework (about 2,000 lines): vCPUs, the framework's in-kernel GICv3, MSIs through a GICv2m frame, and instruction decoding for the MMIO accesses the CPU does not describe in the syndrome.
  • No kernel extension and no root. A VM is one ordinary user process.

What arm64 bhyve needed to boot Windows

  • UEFI: stock EDK2 ArmVirtQemuKernel as the bootrom (u-boot-bhyve-arm64 from pkg also still works)
  • fw_cfg with an MMIO interface, DMA and writable files, and a node in the device tree
  • ACPI tables for arm64: FADT, MADT, GTDT, MCFG, SPCR, DBG2, DSDT, a CPU SSDT, power button through GED
  • A CFI NOR flash for UEFI variables (-o bootvars=FILE)
  • QEMU's ramfb on fbuf, so UEFI has a GOP
  • A virtio 1.0 (modern) transport next to the legacy one. The ARM64 Windows virtio drivers cannot use I/O BARs.
  • A TPM 2.0 (libtpms in-process) and Secure Boot

What runs

  • Windows 11 ARM from Microsoft's unmodified ISO. The installer's TPM and Secure Boot checks pass as they are. Unattended install to the desktop takes about 8 minutes on an M4.
  • FreeBSD/arm64, 15.1 tested, from the release ISO.
  • Debian arm64 is part of the test scripts.

Command lines look like bhyve on FreeBSD:

Code:
bhyve -c 4 -m 4G \
    -o bootrom=QEMU_EFI.fd -o bootvars=vars.fd \
    -o console=stdio \
    -s 0,hostbridge \
    -s 1,nvme,disk.img \
    -s 2,virtio-net,nat \
    -s 3,xhci,tablet,kbd -s 5,virtio-rnd \
    -s 7,fbuf,rfb=127.0.0.1:5900 \
    vm0

Differences from FreeBSD: networking is vmnet (nat or bridged) instead of tap, there is no passthrough, no virtio-9p / virtio-scsi / virtio-input, and -p / -S do nothing.

Bugs found in bhyve on the way

Running the same sources on a different libc exposed a few things that FreeBSD hides, because its default mutexes are error-checking and its pthread objects are pointers:

  • block_if: error paths compare pthread_mutex_t / pthread_cond_t against NULL
  • virtio: vi_interrupt() locks vs_mtx on a path that already holds it. On FreeBSD the second lock returns EDEADLK, which is ignored, and the inner unlock then releases the outer lock early.
  • xhci: the same pattern in pci_xhci_handle_transfer() / pci_xhci_try_usb_xfer()
  • nvme: aen_cond is used without being initialised
  • rfb: a mutex that was never initialised gets destroyed
  • bhyvegc: bhyvegc_set_fbaddr() frees a framebuffer the caller owns
  • PCI BAR: if the guest temporarily points a BAR at an address another device occupies, registration fails and is never retried

Separately, we believe the kernel's vmm_emulate_instruction() on arm64 has the same gap we had to fill on our side for sign-extending MMIO loads (LDRSB and friends). We have not verified that on FreeBSD.

Upstreaming: where we need advice

Eighteen patches are written with commit messages for review, against head 1de93128e75a. The grouping and dependencies are in vm/bhyve/patches/REVIEW.md (the notes are in Chinese, the patch descriptions in English). Nothing has been submitted yet, and we want to be clear about the state: everything has only run on Hypervisor.framework. None of it has been tested on FreeBSD's own arm64 vmm or regression-tested on amd64, because we do not have that environment yet.

Questions:

  1. Device addresses. To boot stock EDK2 ArmVirt we moved the UART, RTC and fw_cfg into the windows that firmware assumes (from 0x09000000, like QEMU's virt machine). Would FreeBSD prefer to move as well, or to have a bhyve platform in EDK2?
  2. The arm64 DSDT. Ours is compiled offline with iasl and checked in; amd64 bhyve runs iasl at run time. Which is wanted for arm64?
  3. virtio 1.0. Is there existing work or an open review for a modern virtio transport in bhyve? We would rather build on it than duplicate it. Ours has no packed virtqueues, and only blk, net, rnd and console are tested.
  4. Is anyone with an arm64 machine with GICv3 willing to try the generic patches on FreeBSD's vmm?

Our plan is to send the seven small fixes first as separate reviews, then raise the UEFI questions on freebsd-virtualization@. Tell us if that is the wrong order or the wrong place.

Rough edges

The VM part is young. No graphics acceleration in Windows (basic display driver, up to 1920x1200), no guest sound, and bridged networking, sleep/wake and long uptimes are not tested enough. Secure Boot keeps its keys where the guest kernel could reach them, since the EDK2 platform has no protected variable store: enough for Windows' requirements, not the strength of a real PC. Needs Apple Silicon and macOS 26 or later.

For completeness: the app around it also runs macOS and Linux guests on Apple's Virtualization.framework and includes an RDP server for the Mac and its VMs, but that is off topic here.

Thanks to everyone who works on bhyve. The fact that the userland could be carried to another hypervisor API with its sources untouched says a lot about how it is put together.
 
Yes, there's a tap now:

brew install --cask keelos-dev/tap/hypermac
hyperMAC updates itself after that ("Check for Updates…"); the cask also gets bumped with every release. brew uninstall --cask hypermac stops the service and removes the login item; add --zap to also delete ~/Library/Application Support/hyperMAC (your VMs live there).
 
Great project, but we only support FreeBSD on these forums. So any support questions regarding this project should be asked elsewhere, not here.

With regards to upstreaming fixes, you can discus this on the freebsd-virtualization mailing list. The devs there will be able to guide you how and where to do that.
 
Back
Top