FreeBSD 11-STABLE VIRT Kernel under Linux Qemu with armkvm


New Member

Reaction score: 1
Messages: 2


I'd like to document my successful attempt to run FreeBSD 11.2 virtualized on an Orange Pi Plus 2E (ARMv7 H3+, 2Gb RAM, 1Gb Ethernet, GICv2).
The reason for me using the VIRT target is incomplete hardware and driver support in 11.2 (the situation with 12-CURRENT might be better but I need to run on the upcoming 11.2).

My kernel configuration /sys/src/sys/arm/conf/VIRT-YVL:
include VIRT
ident VIRT-YVL

options ROOTDEVNAME=\"ufs:/dev/da0p2\"

options DDB

device  pci
device  pci_host_generic
options NEW_PCIB
options PCI_HP

device  scbus
device  da
device  pass

device  virtio_pci
device  virtio_scsi
device  virtio_balloon
device  virtio_console
device  virtio_random
The most important additions are device pci, device pci_host_generic and device virtio_pci.
These are required to work around a qemu(?) bug with virtio-mmio networking described here,
by using -device virtio-net-pci instead of the default -device virtio-net-device.

Also attached to this post is a patch to enable working SMP under armkvm, without it -smp 2 will fail to boot with the ULE scheduler panic
caused by a bad topology (because the second AP appears in the FDT but failes to activate) .
The patch enables virtual timer for arm >= 7 (similarly to aarch64), handles the PSCI_RETVAL_ALREADY_ON error in virt_start_ap
which is returned under armkvm (but not software-only emulation).
Additionally I had to disable GIC_DEBUG_SPURIOUS and increase the wait time in check_ap
(on my system it doesn't really take 30s for the AP to come up, but the 2000ms value used was too low).

The Linux host is a modified armbian with a 4.17.0 kernel compiled with KVM support CONFIG_NO_HZ_FULL=y and qemu-system-arm 2.12+dfsg-1+b1.
The UEFI payload being used is QEMU_EFI_rel_16.02.fd from Linaro.
Kernel cmdline to isolate cores 2,3 for virtualization only: irqaffinity=0-1 isolcpus=2-3 nohz_full=2-3 rcu_nocbs=2-3
qemu command line to run on cores 2,3 with RT priority:
chrt -r 1 taskset -c 2-3 qemu-system-arm \
        -enable-kvm -machine virt,accel=kvm,gic-version=2,kernel-irqchip=off \
        -cpu host -smp cores=2,threads=1,sockets=1 -m 1536 -nographic \
        -chardev tty,path=/dev/pts/3,id=pts3 -serial chardev:pts3 \
        -device virtio-scsi-pci,id=scsi \
        -drive if=none,file=./arm_freebsd_gpt.img,format=raw,cache=none,aio=native,id=hd0 \
        -device scsi-hd,drive=hd0,bus=scsi.0 \
        -netdev user,hostfwd=tcp::8022-:22,id=net0 -device virtio-net-pci,netdev=net0 \
        -object rng-random,filename=/dev/random,id=rng0 \
        -device virtio-rng-pci,max-bytes=1024,period=1000,rng=rng0 \
        -nodefaults -monitor stdio \
        -mem-prealloc -mem-path /dev/hugepages/arm-virt \
        -bios ./QEMU_EFI_rel_16.02.fd
kernel-irqchip=off seems mandatory, otherwise booting in SMP with 2 cores just spins at 100% cpu usage, single core does work.