bhyve FreeBSD as Virtualbox Guest: Bhyve: vm_create: Device not configured

S

specter456

Versions
FreeBSD: 13.1-RELEASE
Virtualbox: 7.0.2 r154219
Host: ArchLinux 6.0.5-arch1-1


Description
At the moment I try to use bhyve of FreeBSD within Virtualbox.
If I try to run following cmd I get the output: vm_create: Device not configured
Code: 
sudo bhyve -AHP -s 0:0,hostbridge -s 1:0,lpc -s 2:0,virtio-net,tap0 -s3:0,virtio-blk,/dev/zvol/zroot/vms/alpinevm -s 4:0,ahci-cd,./alpine-virt-3.16.2-x86_64.iso -c 4 -m 1024M \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
-l com1,/dev/nmdm0A \
alpinevm

vm_create: Device not configured

Do you know what I have to do, that bhyve is working within a VM?

What I would expect
Working bhyve

Further informations
Host:
BIOS/UEFI: SVM is enabled
Virtualbox: Nested VT-x/AMD-V enabled

Guest:
dmesg: POPCNT processor feature flag on the Features2 line for AMD visible
Code: 
sudo kldstat
Password:
Id Refs Address                Size Name
 1   32 0xffffffff80200000  1f30590 kernel
 2    1 0xffffffff82131000     a158 cryptodev.ko
 3    1 0xffffffff8213c000   57a950 vmm.ko
 4    1 0xffffffff826b7000     3978 nmdm.ko
 5    1 0xffffffff826bb000   5b93a0 zfs.ko
 6    1 0xffffffff82c75000     e518 if_bridge.ko
 7    2 0xffffffff82c84000     78c0 bridgestp.ko
 8    1 0xffffffff82f18000     3218 intpm.ko
9 1 0xffffffff82f1c000 2180 smbus.ko

Code: 
cat /boot/loader.conf
kern.geom.label.disk_ident.enable="0"
kern.geom.label.gptid.enable="0"
cryptodev_load="YES"
zfs_load="YES"
virtio_blk_load="YES"
# Modules for our vms
vmm_load="YES"
nmdm_load="YES"
if_bridge_load="YES"
if_tap_load="YES"
 
specter456 said:
FreeBSD: 12.2-RELEASE-p12
Click to expand...
Upgrade your system first. FreeBSD 12.2 is now end-of-life and not supported anymore.

forums.freebsd.org

Topics about unsupported FreeBSD versions

The FreeBSD Forums cater primarily to end-users and systems administrators. As such, the Forums focus almost exclusively on FreeBSD versions that are officially supported according to the official FreeBSD website. Since resources are scarce, the FreeBSD Forums strongly suggest that anyone asking...
forums.freebsd.org forums.freebsd.org
www.freebsd.org

Unsupported FreeBSD Releases

FreeBSD is an operating system used to power modern servers, desktops, and embedded platforms.
www.freebsd.org www.freebsd.org
 
As vmm.ko is loaded in FreeBSD guest I'd assume nested virtualization configuration is ok. I don't use bhyve so a long shot: missing virtio drivers in FreeBSD guest? Are you able to launch the VM without block devices? I.e. just boot it to UEFI without any devices attached.
 
SirDice said:
Upgrade your system first. FreeBSD 12.2 is now end-of-life and not supported anymore.

forums.freebsd.org

Topics about unsupported FreeBSD versions

The FreeBSD Forums cater primarily to end-users and systems administrators. As such, the Forums focus almost exclusively on FreeBSD versions that are officially supported according to the official FreeBSD website. Since resources are scarce, the FreeBSD Forums strongly suggest that anyone asking...
forums.freebsd.org forums.freebsd.org
www.freebsd.org

Unsupported FreeBSD Releases

FreeBSD is an operating system used to power modern servers, desktops, and embedded platforms.
www.freebsd.org www.freebsd.org
Click to expand...
Sorry I wrote the false version... I actually use 13.1... (I edited my Post)
I mixed it up with my other FreeBSD installation on a hardware....
 
_martin said:
As vmm.ko is loaded in FreeBSD guest I'd assume nested virtualization configuration is ok. I don't use bhyve so a long shot: missing virtio drivers in FreeBSD guest? Are you able to launch the VM without block devices? I.e. just boot it to UEFI without any devices attached.
Click to expand...
If I start bhyve with this cmd I have the same error:
sudo bhyve -AHP \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
alpinevm

vm_create: Device not configured
Click to expand...
 
Can you confirm/verify that loading vmm under FreeBSD doesn't produce error ? dmesg would show you this. vmm module, while loaded, still can report error such as:
Code: 
module_register_init: MOD_LOAD (vmm, 0xffffffff82a19e60, 0) error 22
 
I did some checks. If you load the vmm with verbose flag kldload -v vmm you get better information on what's going on:
Code: 
vmx_init: processor does not support desired primary processor-based controls
module_register_init: MOD_LOAD (vmm, 0xffffffff82a4de60, 0) error 22
In my case nesting feature is enabled but fails to init properly. For my Intel CPU problem is here. Problem is, though, if I read it correctly: oracle: nested virtualization you can nest only the same hypervisors. On host (Ubuntu in my case) there is Logs/ directory, maybe they have something valid why it failed in your case. In my case I see, e.g.
Code: 
00:00:01.194008 HM: HMR3Init: VT-x w/ nested paging and unrestricted guest execution hw support

In your case error 6 (ENXIO) means that the CPU doesn't have a feature enabled which is also shown by the SVM feature. When you do the VBoxManage showvminfo $vmname what does these say?
Code: 
Nested VT-x/AMD-V:           enabled
Hardware Virtualization:     enabled
Nested Paging:               enabled
Large Pages:                 disabled
VT-x VPID:                   enabled
VT-x Unrestricted Exec.:     enabled
Paravirt. Provider:          KVM
Effective Paravirt. Prov.:   KVM
 
Code: 
kldload -v vmm
Loaded vmm, id=6
SVM: Nested Paging feature not available.
module_register_init: MOD_LOAD (vmm, 0xffffffff82c04200, 0) error 6

You speake about the Vbox.log file?
I attached it as file for you

Code: 
VBoxManage showvminfo $vmname
Nested VT-x/AMD-V:           enabled
CPUID Portability Level:     0
CPUID overrides:             None
Boot menu mode:              message and menu
Boot Device 1:               Floppy
Boot Device 2:               DVD
Boot Device 3:               HardDisk
Boot Device 4:               Not Assigned
ACPI:                        enabled
IOAPIC:                      enabled
BIOS APIC mode:              APIC
Time offset:                 0ms
BIOS NVRAM File:             /home/alexander/VirtualBox VMs/FreeBSD/FreeBSD.nvram
RTC:                         local time
Hardware Virtualization:     enabled
Nested Paging:               enabled
Large Pages:                 disabled
VT-x VPID:                   enabled
VT-x Unrestricted Exec.:     enabled
AMD-V Virt. Vmsave/Vmload:   enabled
IOMMU:                       None
Paravirt. Provider:          Default
Effective Paravirt. Prov.: None
 

Attachments

Yes, that log. And it seems that nested paging is enabled. I don't know why is VirtualBox having problem with it; neither did I debug which feature is exactly is missing. I was using 13.1-stable in this test.

However I did install VMWare Workstation. Host: Ubuntu 22.04. FreeBSD VM with VT-x/AMD-V/RVI and virt IOMMU enabled. open-vm-tools installed, vmm loads without problem. bhyve machine launched as: bhyve -AHP -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -s 31,lpc vm02, connected via vnc and I see EFI prompt. So at least this test works ok.
If you can't get your hands on VMware workstation maybe Player would be enough.
 
Thank you for testing out!

In the meantime I tested following.

Linux-Host:
- Virtualbox: not working (Error 6)
- VmWare Player: working
- Qemu: working

Windows-Host:
- Hyper-V: not working (Error 22)
- VMwarePlayer: working
- Qemu: working (but problems to setup a working networkbridge)

So I need to use my setup for commercial usage... so I sadly can not use VmWare without buying licenses...
My workstation has to be a windows-host... so I have to do some research to get working networking within my freebsd guest...
Too bad that virtualbox is not working on this point :/.
 
specter456 said:
- Qemu: working
Click to expand...
Did you use any specific CPU features ? I think I used neither and then -cpu max but neither worked. I don't remember what was the error from top of my head, I did move to VMware to test.

KVM is a good choice (option of many cloud providers). It does require few tweaks but once you have it running it's ok.
 
I have to update the list, because I made a mistake...

Linux-Host:
- Virtualbox: not working (Error 6)
- VmWare Player: working
- Qemu: working (but problems to setup a working networkbridge)

Windows-Host:
- Hyper-V: not working (Error 22)
- Virtualbox: not working (Error 6)
- VMwarePlayer: working
- Qemu: not working (Error 6)

Sadly I do not get qemu working on windows, because there kvm is not available and
Haxm is not working too...
 
_martin said:
Did you use any specific CPU features ? I think I used neither and then -cpu max but neither worked. I don't remember what was the error from top of my head, I did move to VMware to test.

KVM is a good choice (option of many cloud providers). It does require few tweaks but once you have it running it's ok.
Click to expand...
Qemu on linux I start with the following cmd and then all is working:
qemu-system-x86_64 -m 10G -smp 7 -drive file=./freebsd-13.qcow2 -netdev user,id
=net0,hostfwd=tcp::5555-:22 -device e1000,netdev=net0 -enable-kvm -cpu host
Click to expand...

Do you have any idea how I can write a bug report for virtualbox without to login with my company id?
I have to get it working on windows, but VmWare is no option...
 
Thanks.
On my test box (Intel N6005) under qemu with -cpu host (or -cpu max) I crash the qemu instance (v6.2.0 and qemu7) when I launch the bhyve. Quick check with gdb shows that vm crashes on (guest's) vmlaunch instruction.

specter456 said:
Do you have any idea how I can write a bug report for virtualbox without to login with my company id?
Click to expand...
I'd say create a new account that is not linked to a company. It's been too long since I opened anything directly towards VirtualBox though.
 
You must log in or register to reply here.
Back
Top