bhyve vm-bhyve: bhyve exited with status 4

L

leonardorame

Hi, I'm trying to create an Ubuntu 24.04 vm with vm-bhyve but I get "bhyve exited with status 4", any hint?

Code: 
sudo vm install -f ubuntuvm ubuntu-24.04.2-desktop-amd64.iso
Starting ubuntuvm
  * found guest in /zroot/bhyve/ubuntuvm
  * booting...

Config:
Code: 
loader="uefi"
cpu=4
memory=4096M
network0_type="virtio-net"
network0_switch="public"
graphics="yes"
xhci_mouse="yes"
zfs_zvol_opts="volblocksize=128k"
disk0_name="disk0"
disk0_dev="sparse-zvol"
disk0_type="virtio-blk"
uuid="6ca89125-fdea-11ef-9054-74563c7e2bd0"
network0_mac="58:9c:fc:0b:21:be"
uefi_vars="yes"
bhyve_options="-A"

Log:
Code: 
Mar 10 18:12:32: initialising
Mar 10 18:12:32:  [loader: uefi]
Mar 10 18:12:32:  [cpu: 4]
Mar 10 18:12:32:  [memory: 4096M]
Mar 10 18:12:32:  [hostbridge: standard]
Mar 10 18:12:32:  [com ports: com1]
Mar 10 18:12:32:  [uuid: 6ca89125-fdea-11ef-9054-74563c7e2bd0]
Mar 10 18:12:32:  [debug mode: no]
Mar 10 18:12:32:  [primary disk: disk0]
Mar 10 18:12:32:  [primary disk dev: sparse-zvol]
Mar 10 18:12:32: initialising network device tap0
Mar 10 18:12:32: failed to find virtual switch 'public'
Mar 10 18:12:32: dynamically allocated port 5900 for vnc connections
Mar 10 18:12:32: booting
Mar 10 18:12:32:  [bhyve options: -c 4 -m 4096M -Hwl bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd,/zroot/bhyve/ubuntuvm/uefi-vars.fd -A -U 6ca89125-fdea-11ef-9054-74563c7e2bd0 -u]
Mar 10 18:12:32:  [bhyve devices: -s 0,hostbridge -s 31,lpc -s 4:0,virtio-blk,/dev/zvol/zroot/bhyve/ubuntuvm/disk0 -s 5:0,virtio-net,tap0,mac=58:9c:fc:0b:21:be -s 6:0,fbuf,tcp=0.0.0.0:5900,wai
t -s 7:0,xhci,tablet]
Mar 10 18:12:32:  [bhyve console: -l com1,stdio]
Mar 10 18:12:32:  [bhyve iso device: -s 3:0,ahci-cd,/zroot/bhyve/.iso/ubuntu-24.04.2-desktop-amd64.iso,ro]
Mar 10 18:12:32: starting bhyve (run 1)
Mar 10 18:12:32: bhyve exited with status 4
Mar 10 18:12:32: destroying network device tap0
Mar 10 18:12:32: stopped
 
Hi CeXP1917, why can this be?

Code: 
sudo vm switch list
Password:
NAME    TYPE      IFACE  ADDRESS  PRIVATE  MTU  VLAN  PORTS
public  standard  -      -        no       -    -     re0
 
Ok, I created a new switch:

Code: 
sudo vm switch list
NAME    TYPE      IFACE      ADDRESS        PRIVATE  MTU  VLAN  PORTS
public standard vm-public 10.0.0.100/24 no - - re0

But when I try to install the vm ( sudo vm install -f ubuntuvm ubuntu-24.04.2-desktop-amd64.iso) I get:

Code: 
Mar 10 19:15:35: initialising
Mar 10 19:15:35:  [loader: uefi]
Mar 10 19:15:35:  [cpu: 4]
Mar 10 19:15:35:  [memory: 4096M]
Mar 10 19:15:35:  [hostbridge: standard]
Mar 10 19:15:35:  [com ports: com1]
Mar 10 19:15:35:  [uuid: 6ca89125-fdea-11ef-9054-74563c7e2bd0]
Mar 10 19:15:35:  [debug mode: no]
Mar 10 19:15:35:  [primary disk: disk0]
Mar 10 19:15:35:  [primary disk dev: sparse-zvol]
Mar 10 19:15:35: initialising network device tap0
Mar 10 19:15:35: adding tap0 -> vm-public (public addm)
Mar 10 19:15:35: bring up tap0 -> vm-public (public addm)
Mar 10 19:15:35: dynamically allocated port 5900 for vnc connections
Mar 10 19:15:35: booting
Mar 10 19:15:35:  [bhyve options: -c 4 -m 4096M -Hwl bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd,/zroot/bhyve/ubuntuvm/uefi-vars.fd -A -U 6ca89125-fdea-11ef-9054-74563c7e2bd0 -u]
Mar 10 19:15:35:  [bhyve devices: -s 0,hostbridge -s 31,lpc -s 4:0,virtio-blk,/dev/zvol/zroot/bhyve/ubuntuvm/disk0 -s 5:0,virtio-net,tap0,mac=58:9c:fc:0b:21:be -s 6:0,fbuf,tcp=0.0.0.0:5900,wai
t -s 7:0,xhci,tablet]
Mar 10 19:15:35:  [bhyve console: -l com1,stdio]
Mar 10 19:15:35:  [bhyve iso device: -s 3:0,ahci-cd,/zroot/bhyve/.iso/ubuntu-24.04.2-desktop-amd64.iso,ro]
Mar 10 19:15:35: starting bhyve (run 1)
Mar 10 19:15:35: bhyve exited with status 4
Mar 10 19:15:35: destroying network device tap0
Mar 10 19:15:35: stopped
 
Right now vm-bhyve is having issues with several versions of Linux. https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=279901

Anecdotally, I've found it to have trouble with Ubuntu, though I don't remember which version, and several others, though Arch and Debian, though not Devuan, worked for me. This has only happened on an AMD machine, so if you're using Intel it probably isn't your issue. I don't remember which Ubuntu I tried. One thing that has worked for some things, including (again my anecdotal experience) with Alma, was installing an older version then upgrading. In other words, install Alma-9.4 and upgrade to 9.5. I don't know if that will work with Ubuntu, but if the issue is with glibc, it's possible an older version will install and be able to be upgraded.
 
Hi scottro, yes, the problem happens on an AMD Ryzen 5 machine. But I'm running bhyve with Linux and Windows VMs on the machine I'm using to write this right now, which also is a Ryzen 5 without issues.

Maybe some BIOS feature not enabled?, how can I check?
 
Status 4, afaik, means something wrong with devices, but your bhyve parameters looks correct. What if for testing set uefi_vars="no"? I'm using Ubuntu-based Mint and it does not need uefi_vars.
 
leonardorame said:
Hi nxjoseph, vm init gave this error:

Code: 
/usr/local/sbin/vm: ERROR: unable to load if_tuntap.ko!
Click to expand...
Hi, are you running custom kernel ? Maybe try loading the required module, see tun(4).

Code: 
SYNOPSIS
       To  compile  this  driver  into the kernel, place the following line in
       your kernel configuration file:

         device tuntap

       Alternatively, to load the driver as a module at    boot time,  place  the
       following lines in loader.conf(5):

         if_tuntap_load="YES"
Code: 
% grep -i tuntap /usr/src/sys/amd64/conf/GENERIC
device          tuntap                  # Packet tunnel.
 
In point 24.7 Freebsd as a Host with bhyve (from https://docs.freebsd.org/en/books/handbook/virtualization/#virtualization-host-bhyve) it tells that dmesg must show POPCNT in Features2:

Code: 
Copyright (c) 1992-2023 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 14.1-RELEASE releng/14.1-n267679-10e31f0946d8 GENERIC amd64
FreeBSD clang version 18.1.5 (https://github.com/llvm/llvm-project.git llvmorg-18.1.5-0-g617a15a9eac9)
VT(vga): resolution 640x480             
CPU: AMD Ryzen 5 5600G with Radeon Graphics          (3892.92-MHz K8-class CPU)
  Origin="AuthenticAMD"  Id=0xa50f00  Family=0x19  Model=0x50  Stepping=0
  Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT>
  Features2=0x7ef8320b<SSE3,PCLMULQDQ,MON,SSSE3,FMA,CX16,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
  AMD Features=0x2e500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM>
  AMD Features2=0x75c237ff<LAHF,CMP,SVM,ExtAPIC,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,IBS,SKINIT,WDT,TCE,Topology,PCXC,PNXC,DBE,PL2I,MWAITX,ADMSKX>
  Structured Extended Features=0x219c97a9<FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,PQM,PQE,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,SHA>
  Structured Extended Features2=0x40069c<UMIP,PKU,OSPKE,VAES,VPCLMULQDQ,RDPID>
  Structured Extended Features3=0x10<FSRM>
  XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
  AMD Extended Feature Extensions ID EBX=0x191ef657<CLZERO,IRPerf,XSaveErPtr,RDPRU,WBNOINVD,IBPB,IBRS,STIBP,STIBP_ALWAYSON,PREFER_IBRS,SSBD>
  SVM: (disabled in BIOS) NP,NRIP,VClean,AFlush,DAssist,NAsids=32768
  TSC: P-state invariant, performance statistics

But, as you can see SVM is disabled in BIOS..
 
I'm running into this with a similar hardware configuration. AMD SVM, and all other virtualization features supported by my AMD 7950X, are enabled in my motherboard's BIOS. All virtualization works as expected, including PCI passthrough, when using Proxmox as the host operating system, so I don't thing it's my hardware or BIOS settings.

Here's my FreeBSD system:
Code: 
# uname -a
FreeBSD myhost 14.2-RELEASE FreeBSD 14.2-RELEASE releng/14.2-n269506-c8918d6c7412 GENERIC amd64
Here's what happens when I try to start the VM:
Code: 
# vm start myvm
Starting myvm
  * found guest in /bhyve/myvm
  * booting...
And the resulting vm-bhyve.log:
Code: 
# cat vm-bhyve.log
Mar 26 15:07:25: initialising
Mar 26 15:07:25:  [loader: uefi]
Mar 26 15:07:25:  [cpu: 1,sockets=1,cores=24]
Mar 26 15:07:25:  [memory: 64g]
Mar 26 15:07:25:  [hostbridge: standard]
Mar 26 15:07:25:  [com ports: com1]
Mar 26 15:07:25:  [uuid: xxxx]
Mar 26 15:07:25:  [debug mode: no]
Mar 26 15:07:25:  [primary disk: disk0]
Mar 26 15:07:25:  [primary disk dev: sparse-zvol]
Mar 26 15:07:25: initialising network device tap0
Mar 26 15:07:25: adding tap0 -> vm-public (public addm)
Mar 26 15:07:25: bring up tap0 -> vm-public (public addm)
Mar 26 15:07:25: dynamically allocated port 5900 for vnc connections
Mar 26 15:07:25: booting
Mar 26 15:07:25:  [bhyve options: -c 1,sockets=1,cores=24 -m 64g -Hwl bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -U xxxx -u]
Mar 26 15:07:25:  [bhyve devices: -s 0,hostbridge -s 31,lpc -s 4:0,virtio-blk,/dev/zvol/zroot/bhyve/myvm/disk0 -s 5:0,virtio-net,tap0,mac=xx:xx:xx:xx:xx:xx -s 6:0,fbuf,tcp=0.0.0.0:5900,w=1920,h=1080 -s 7:0,xhci,tablet]
Mar 26 15:07:25:  [bhyve console: -l com1,/dev/nmdm-myvm.1A]
Mar 26 15:07:25: starting bhyve (run 1)
Mar 26 15:07:25: bhyve exited with status 4
Mar 26 15:07:25: destroying network device tap0
Mar 26 15:07:25: stopped
This isn't too useful. So I converted my configuration from using the `vm-bhyve` wrappers to just a plain, raw script (let's say `run.sh`):
Code: 
#!/bin/sh

VM_NAME="myvm"
UUID="xxxx"
BOOTROM="/usr/local/share/uefi-firmware/BHYVE_UEFI.fd"
DISK="/dev/zvol/zroot/bhyve/$VM_NAME/disk0"
MAC_ADDR="xx:xx:xx:xx:xx:xx"
TAP_IF="tap0"
VNC_PORT="5900"
VNC_WIDTH="1920"
VNC_HEIGHT="1080"
VNC_PASS="abcd"

bhyve \
    -c 1,sockets=1,cores=24 \
    -m 64g \
    -H \
    -w \
    -l bootrom,$BOOTROM \
    -U $UUID \
    -u \
    -s 0,hostbridge \
    -s 31,lpc \
    -s 4:0,virtio-blk,$DISK \
    -s 5:0,virtio-net,$TAP_IF,mac=$MAC_ADDR \
    -s 6:0,fbuf,tcp=0.0.0.0:$VNC_PORT,w=$VNC_WIDTH,h=$VNC_HEIGHT,password=$VNC_PASS \
    -s 7:0,xhci,tablet \
    $VM_NAME
After running the script (and therefore, getting output directly from Bhyve), I got something much more useful:
Code: 
bhyve: Topology (1 sockets, 24 cores, 1 threads) does not match 1 vCPUs
Yea, that definitely seems wrong. My AMD 7950X is a single socket, 16 core, 32 thread processor. It seems like I had a fundamental misunderstanding about the `-c` option. The first number seems to be the number of logical CPU cores, not the number of physical packages/dies. Cores should also be the number of physical cores, and threads should be 2 (two per physical core).

After changing the option to -c 24,sockets=1,cores=12,threads=2, the VM starts and I'm able to VNC into it. I haven't done anything with the VM yet (haven't booted into a guest OS), but I wanted to share my notes now, in case this might help someone else. I'll follow up later when I have more time to tinker with it.
 
If you're using sysutils/vm-bhyve then you can also add debug="yes" to your VM's config which will save the output of bhyve to a bhyve.log file in your VM's dir.

No need to create a script and is handy for finding issues like:
  • CPU / core / thread topology mismatch
  • PCI devices not configured for pass-through
  • unable to find files backing virtual drives (nvme, virtio-blk, ahci-cd, ...)
  • unsupported instruction errors to report back to bhyve devs
I run with debug="yes" all the time as it's pretty much zero cost and I've not seen a bhyve.log bigger than a few KB.
 
I realize this thread was last updated in March, but at any rate, RHEL10, (not the beta) does not have this problem. Right now, it's only available if you have a RH subscription, but there should be Rocky and Alma versions shortly.
 
You must log in or register to reply here.
Back
Top