general/other FreeBSD 13.1 under OpenBSD 7.2 vmd?

C

cmoerz

Has anyone gotten FreeBSD to run under OpenBSD's vmm/vmd?
What I've tried to far:
  1. Managed to boot a 12.1 installation ISO via serial until it just hangs
  2. Managed to install 13.1 in a QEMU emulation, then put the qcow2 image to boot under vmd; got a kernel panic that is identical to what someone posted in 2018 (https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=231468)
  3. Did not manage to get to the boot loader menu with 13.1 installation iso, same for 14
  4. attempted to boot a qcow2 image for 14 - again, no luck
According to a 2018 slide deck, some developer had FreeBSD running back then (https://www.openbsd.org/papers/asiabsdcon2018-vmm-slides.pdf); I'm seriously wondering what magic was applied to make this work. Maybe I need to backtrack and attempt booting 11 (I know, I know...)?

Obviously, one needs usual serial console settings in /boot/loader.conf but it seems, that's not the only tweaking necessary. Maybe there's some fiddling required with kernel settings?

If you have any ideas or recommendations, I'm all ears.
Thanks
 
Here's the output with 12.3 ISO:
Code: 
...
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 12.3-RELEASE r371126 GENERIC amd64
FreeBSD clang version 10.0.1 (git@github.com:llvm/llvm-project.git llvmorg-10.0.1-0-gef32c611aa2)
Firmware Error (ACPI): A valid RSDP was not found (20200430/tbxfroot-369)
VT(vga): text 80x25
CPU: 12th Gen Intel(R) Core(TM) i5-1240P (2111.83-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x906a3  Family=0x6  Model=0x9a  Stepping=3
  Features=0x793a97f<FPU,VME,DE,PSE,TSC,MSR,PAE,CX8,SEP,PGE,CMOV,PAT,PSE36,<b20>,MMX,FXSR,SSE,SSE2>
  Features2=0xfed83203<SSE3,PCLMULQDQ,SSSE3,FMA,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV>
  AMD Features=0x24100800<SYSCALL,NX,Page1GB,LM>
  AMD Features2=0x121<LAHF,ABM,Prefetch>
  Structured Extended Features=0x219c23e9<FSGSBASE,BMI1,AVX2,FDPEXC,SMEP,BMI2,ERMS,NFPUSG,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,SHA>
  Structured Extended Features2=0x984007ac<UMIP,PKU,WAITPKG,GFNI,VAES,VPCLMULQDQ,RDPID,MOVDIRI,MOVDIRI64B>
  Structured Extended Features3=0x400<MD_CLEAR>
  TSC: P-state invariant
real memory  = 2147483648 (2048 MB)
avail memory = 2044350464 (1949 MB)
arc4random: no preloaded entropy cache
Timecounter "TSC" frequency 2111834856 Hz quality 1000
random: entropy device external interface
WARNING: Device "g_ctl" is Giant locked and may be deleted before FreeBSD 14.0.
000.000018 [4344] netmap_init               netmap: loaded module
WARNING: Device "pci" is Giant locked and may be deleted before FreeBSD 14.0.
[ath_hal] loaded
module_register_init: MOD_LOAD (vesa, 0xffffffff811848f0, 0) error 19
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
WARNING: Device "kbd" is Giant locked and may be deleted before FreeBSD 14.0.
kbd0 at kbdmux0
mlx5en: Mellanox Ethernet driver 3.6.0 (December 2020)
Firmware Error (ACPI): A valid RSDP was not found (20200430/tbxfroot-369)
ACPI: Table initialisation failed: AE_NOT_FOUND
ACPI: Try disabling either ACPI or apic support.
vtvga0: <VT VGA driver> on motherboard
cryptosoft0: <software crypto> on motherboard
pcib0 pcibus 0 on motherboard
pci0: <PCI bus> on pcib0
virtio_pci0: <VirtIO PCI Entropy adapter> port 0x1000-0x1fff irq 3 at device 1.0 on pci0
virtio_pci1: <VirtIO PCI Network adapter> port 0x2000-0x2fff irq 5 at device 2.0 on pci0
vtnet0: <VirtIO Networking Adapter> on virtio_pci1
vtnet0: Ethernet address: fe:e1:bb:d1:31:f9
virtio_pci1: exhausted all interrupt allocation attempts
vtnet0: cannot setup virtqueue interrupts

Here's the 13.1 if I add a vtnet device; console output:
Code: 
Firmware Error (ACPI): A valid RSDP was not found (20201113/tbxfroot-369)
---<<BOOT>>---
Copyright (c) 1992-2021 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 13.1-RELEASE-p3 GENERIC amd64
FreeBSD clang version 13.0.0 (git@github.com:llvm/llvm-project.git llvmorg-13.0.0-0-gd7b669b3a303)
Firmware Error (ACPI): A valid RSDP was not found (20201113/tbxfroot-369)
VT(vga): text 80x25
CPU: 12th Gen Intel(R) Core(TM) i5-1240P (2112.00-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x906a3  Family=0x6  Model=0x9a  Stepping=3
  Features=0x793a97f<FPU,VME,DE,PSE,TSC,MSR,PAE,CX8,SEP,PGE,CMOV,PAT,PSE36,<b20>,MMX,FXSR,SSE,SSE2>
  Features2=0xfed83203<SSE3,PCLMULQDQ,SSSE3,FMA,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV>
  AMD Features=0x24100800<SYSCALL,NX,Page1GB,LM>
  AMD Features2=0x121<LAHF,ABM,Prefetch>
  Structured Extended Features=0x219c23e9<FSGSBASE,BMI1,AVX2,FDPEXC,SMEP,BMI2,ERMS,NFPUSG,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,SHA>
  Structured Extended Features2=0x984007ac<UMIP,PKU,WAITPKG,GFNI,VAES,VPCLMULQDQ,RDPID,MOVDIRI,MOVDIR64B>
  Structured Extended Features3=0x400<MD_CLEAR>
  TSC: P-state invariant
real memory  = 2147483648 (2048 MB)
avail memory = 2047279104 (1952 MB)
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
random: unblocking device.
random: entropy device external interface
kbd0 at kbdmux0
Firmware Error (ACPI): A valid RSDP was not found (20201113/tbxfroot-369)
ACPI: Table initialisation failed: AE_NOT_FOUND
ACPI: Try disabling either ACPI or apic support.
vtvga0: <VT VGA driver>
smbios0: <System Management BIOS> at iomem 0xf36c0-0xf36de
smbios0: Version: 2.4, BCD Revision: 2.4
aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS,SHA1,SHA256>
pcib0 pcibus 0
pci0: <PCI bus> on pcib0
virtio_pci0: <VirtIO PCI (legacy) Entropy adapter> port 0x1000-0x1fff irq 3 at device 1.0 on pci0
virtio_pci1: <VirtIO PCI (legacy) Network adapter> port 0x2000-0x2fff irq 5 at device 2.0 on pci0
vtnet0: <VirtIO Networking Adapter> on virtio_pci1
vtnet0: Ethernet address: fe:e1:bb:d1:73:77
virtio_pci1: exhausted all interrupt allocation attempts
vtnet0: cannot setup interrupts

Without the vtnet device, I get following panic:
Code: 
pci0: <PCI bus> on pcib0
virtio_pci0: <VirtIO PCI (legacy) Entropy adapter> port 0xc000-0xcfff irq 3 at device 1.0 on pci0
virtio_pci1: <VirtIO PCI (legacy) Block adapter> port 0xd000-0xdfff irq 5 at device 2.0 on pci0
vtblk0: <VirtIO Block Adapter> on virtio_pci1
virtio_pci1: exhausted all interrupt allocation attempts
vtblk0: cannot setup virtqueue interrupt


Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address   = 0x20
fault code              = supervisor write data, page not present
instruction pointer     = 0x20:0xffffffff80b5c2f7
stack pointer           = 0x28:0xffffffff821e29c0
frame pointer           = 0x28:0xffffffff821e29f0
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 0 (swapper)
trap number             = 12
panic: page fault
cpuid = 0
time = 1
KDB: stack backtrace:
#0 0xffffffff80c694a5 at kdb_backtrace+0x65
#1 0xffffffff80c1bb5f at vpanic+0x17f
#2 0xffffffff80c1b9d3 at panic+0x43
#3 0xffffffff810afdf5 at trap_fatal+0x385
#4 0xffffffff810afe4f at trap_pfault+0x4f
#5 0xffffffff810875b8 at calltrap+0x8
#6 0xffffffff80b5c2d3 at g_post_event_ep+0x43
#7 0xffffffff80a5626b at vtblk_detach+0x45b
#8 0xffffffff80a55b64 at vtblk_attach+0x154
#9 0xffffffff80c58633 at device_attach+0x3c3
#10 0xffffffff80a4c9c6 at vtpci_legacy_probe_and_attach_child+0x76
#11 0xffffffff80a4bee6 at vtpci_legacy_attach+0x206
#12 0xffffffff80c58633 at device_attach+0x3c3
#13 0xffffffff80c598cb at bus_generic_attach+0x4b
#14 0xffffffff808668a3 at pci_attach+0xd3
#15 0xffffffff80c58633 at device_attach+0x3c3
#16 0xffffffff80c598cb at bus_generic_attach+0x4b
#17 0xffffffff8106f407 at legacy_pcib_attach+0x77
Uptime: 1s
Automatic reboot in 15 seconds - press a key on the console to abort

EDIT - I managed to get the panic also with 12.3 iso after booting without network devices:
Code: 
isa0: <ISA bus> on motherboard
orm0: <ISA Option ROMs> at iomem 0xbf000-0xbffff,0xee800-0xeffff pnpid ORM0000 on isa0
vga0: <Generic ISA VGA> at port 0x3b0-0x3bb iomem 0xb0000-0xb7fff pnpid PNP0900 on isa0
Firmware Error (ACPI): A valid RSDP was not found (20200430/tbxfroot-369)
Firmware Error (ACPI): A valid RSDP was not found (20200430/tbxfroot-369)
atrtc0: <AT realtime clock> at port 0x70 irq 8 on isa0
atrtc0: registered as a time-of-day clock, resolution 1.000000s
atrtc0: Can't map interrupt.
atrtc0: non-PNP ISA device will be removed from GENERIC in FreeBSD 14.
attimer0: <AT timer> at port 0x40 on isa0
Timecounter "i8254" frequency 1193182 Hz quality 0
attimer0: Can't map interrupt.
attimer0: non-PNP ISA device will be removed from GENERIC in FreeBSD 14.
fdc0: cannot reserve interrupt line
uart0: <8250 or 16450 or compatible> at port 0x3f8 irq 4 flags 0x10 on isa0
uart0: console (115200,n,8,1)
uart0: non-PNP ISA device will be removed from GENERIC in FreeBSD 14.
panic: No usable event timer found!
cpuid = 0
time = 1
KDB: stack backtrace:
#0 0xffffffff80c2d465 at kdb_backtrace+0x65
#1 0xffffffff80be15fb at vpanic+0x17b
#2 0xffffffff80be1473 at panic+0x43
#3 0xffffffff8126c5ca at cpu_initclocks_bsp+0x3ba
#4 0xffffffff8129b255 at cpu_initclocks+0x15
#5 0xffffffff80b7be40 at initclocks+0x20
#6 0xffffffff80b78aff at mi_startup+0xdf
#7 0xffffffff8038102c at btext+0x2c
Uptime: 1s
Automatic reboot in 15 seconds - press a key on the console to abort

I assume, I'm not seeing the panic when adding a vtnet device, because it instead hangs during initalization and never reaches the culprit of the panic.
 
You should probably ask your question in an OpenBSD forums, pal, since it is clearly OpenBSD related. This forum is only suited to answer the other way around, namely running OpenBSD under bhyve, but almost nobody here will use OpenBSD, and even less its hypervisor.
 
To be honest, I don't expect any OpenBSD related answers. I'm wondering how to troubleshoot the FreeBSD kernel panic - that is not OpenBSD related at all.

Yes, there may be some component in OpenBSD that is causing the issue, however for the moment I'm looking at FreeBSD to better understand what's going on and because I've got more experience with FreeBSD than OpenBSD.

I do get your point though and while I agree, that for a full resolution I may likely need some help from the OpenBSD side, there's another aspect I'd like to point out: unless I better understand what's going on with FreeBSD, I'm afraid this will become a "pointer fest" where each party points to the other side as the culprit. I'd like to avoid that by doing a bit more research first.
 
cmoerz Thanks for pasting the dumps. This is a problem of two worlds - both OpenBSD and FreeBSD. OpenBSD's because of its hypervisor and FreeBSD's because how it deals with a possible issue.
the 12.x crash happens very early in boot when timers were being initialized. I'd say it is related to the ACPI issues early in the boot. As I'm not familiar with the OpenBSD's vmm I wonder how other VMs deal with this.

The 13.x crash is interesting in the way that the vtpci code just stumped into a bad addres (0x20) and crashed. The errros "vtnet0: cannot setup interrupts" along with other ACPI errors suggest again errors related to ACPI.

Out of my own curiosity I'd try to reproduce this on 13.x. Does OpenBSD's vmm have any options related to a type of computer it tries to emulate (similar to qemu's -M) or does it provide an additional options to add (similar to qemu's bios option and/or additional parameters specifying device).

It would be interesting to see what Linux does during the boot to compare.
 
Thanks for the feedback, _martin. You're confirming a hunch I had previously.
I was already looking at a way to disable acpi for the FreeBSD guest to kind of "boot around" the problem.
I'll keep digging a bit deeper. If I find anything more of value or somehow manage to finally boot, I'll post with an update.
 
I had to go buy a Xmas tree, kids were going crazy. :)

I tried to install OpenBSD in proxmox and there I pushed FreeBSD i386 VM I had around. It didn't boot, it got frozen when root was being mounted. Sadly I don't feel at home on OpenBSD, not sure what all tools I have to trace this. I really love qemu+gdb combination when it comes to running and debugging VM (mini rant: I'll go as far to say qemu figured it out - that's how one should run VMs. If only Free|OpenBSD provided kvm side of things world would be perfect. Sadly it rarely is).

I was a bit surprised that I was not able to find much about vmm on OpenBSD. Some say that vmm was intended only for OpenBSD guests. Some reported issues with running Linux with newer kernel versions.

I am curious to see what does FreeBSD do that crash occurs. I want to install OpenBSD on HW and try to get coredump to have better idea. I wonder if vmm has gdb stub such as qemu does. It would be very convenient to attach debugger and set breakpoint on that g_post_event_ep() function.
As mentioned though I'm curious so I'll dig deeper too.
 
Hey, once you have kids your priorities shift. Or they make them shift. Kind of a feedback loop - I still need to get a tree as well. So, good on you for having completed that task! Now to decorating and lighting... with rising energy prices, I keep thinking back to National Lampoon's Christmas Vacation (
View: https://www.youtube.com/watch?v=iXaw70X7wb4
if you haven't seen it).

Yup, same here. I'm feeling way more home on FreeBSD and doing my bidding on OpenBSD purely out of necessity (Alder Lake's integrated graphics not working yet). I've considered stepping down to Intel 11th gen and shelve the 12th gen, but I figure if I can get a decent virtualized FreeBSD going on OpenBSD it'll be fine until graphics/drm-kmod catches up.

My curiosity is keeping me digging as well and with the holidays coming up I've got a few hours to spare. I'm compiling a custom kernel at the moment. Removed acpi and adding a bunch of debugging printf's to virtio block device driver.
I'd love to just run it through a debugger but I haven't seen anything in OpenBSD that would support that.

I agree that vmm is well under-documented. There's a few man pages but that's about it. Seems like this is the OpenBSD mantra: if you want to know more, read the source... (haha).

I've stumbled across some ".m" files in the FreeBSD kernel that threw me for a loop for a moment. Those kobj files are where we might have some issues but I'll wait for the debug printfs to do their job before I'll point fingers...

While this might all just be an exercise in futility, it tickles me just in the right places to keep going.
 
I'm not gonna use such fancy lights for sure. ;)

I was able to reproduce both panics as you showed. The panic on 12 makes sense and is a proper "shutdown" in the situation - no way out, call panic. On 13 panic is caused by regular crash, wild panic as device was detached ( vtblk_attach()-> irq alloc issues -> vtblk_detach() -> panic on 0x20 /probably accessing NULL ptr in g_post_event_ep_va()/ ). Not that it helps in this case, both issues are stemming from a common problem - hypervisor.

As all these issues are about/around interrupt settings I was thinking about legacy fallback to PIC/PIT. But on FreeBSD 12 atpic is no longer included in generic. It might be interesting to include it (device atpic) and check.

But with very little information I was able to find, and from source that's hard to quote (reddit article where "OpenBSD developer" answered) FreeBSD is not supported.
 
You must log in or register to reply here.
Back
Top