bhyve bhyve (via vm) guest no longer starts - bhyve exited with status 4

dvl@

dvl@

Developer
I've managed to get this vm running via vm start -f hass on FreeBSD 15 with vm-bhyve-1.7.3

Recently, I was moving /usr/local/vm to a new ZFS dataset. This problem occurred there, so I went back to the original dataset and tried it there. Both datasets are not active at the same time (imagine data02/vm (original and what I'm using for this post) and data04/vm (new))

It starts in the foreground, with interesting messages:

Code: 
GNU GRUB  version 2.12
   *Slot B (OK=1 TRY=1)
    Slot A (rescue shell)
    Slot B (rescue shell)
  
     Booting `Slot B (OK=1 TRY=1)'

None of the above is interactive (i.e. I'm not selecting anything).

The VM (Home Assistant) comes to a login prompt, I can login, etc.

From the logs, I see:

Code: 
*** /usr/local/vm/hass/vm-bhyve.log ***
2026-04-23T14:48:47+00:00: initialising
2026-04-23T14:48:47+00:00:  [loader: uefi]
2026-04-23T14:48:47+00:00:  [cpu: 4]
2026-04-23T14:48:47+00:00:  [memory: 8GB]
2026-04-23T14:48:47+00:00:  [hostbridge: standard]
2026-04-23T14:48:47+00:00:  [com ports: com1]
2026-04-23T14:48:47+00:00:  [uuid: 9aae377a-6c06-11ed-a655-002590fa0f10]
2026-04-23T14:48:47+00:00:  [debug mode: no]
2026-04-23T14:48:47+00:00:  [primary disk: disk0.img]
2026-04-23T14:48:47+00:00:  [primary disk dev: file]
2026-04-23T14:48:47+00:00: initialising network device tap1
2026-04-23T14:48:47+00:00: adding tap1 -> bridge0 (public addm)
2026-04-23T14:48:47+00:00: bring up tap1 -> bridge0 (public addm)
2026-04-23T14:48:47+00:00: booting
2026-04-23T14:48:47+00:00:  [bhyve options: -c 4 -m 8GB -AHPw -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -A -U 9aae377a-6c06-11ed-a655-002590fa0f10 -u]
2026-04-23T14:48:47+00:00:  [bhyve devices: -s 0,hostbridge -s 31,lpc -s 0:4:0,virtio-blk,/usr/local/vm/hass/disk0.img -s 0:5:0,virtio-net,tap1,mac=58:9c:fc:08:5d:13]
2026-04-23T14:48:47+00:00:  [bhyve console: -l com1,stdio]
2026-04-23T14:48:47+00:00: starting bhyve (run 1)

*** /var/log/messages ***
Apr 23 14:48:47 r730-01 kernel: tap1: Ethernet address: 58:9c:fc:10:e2:5d
Apr 23 14:48:47 r730-01 kernel: tap1: promiscuous mode enabled
Apr 23 14:48:48 r730-01 kernel: tap1: link state changed to UP

Next, I run (from another ssh connection)
Code: 
% sudo vm stop hass
Sending ACPI shutdown to hass

And it stops.

However, I am unable to get the VM to run via:

Code: 
% sudo vm start hass 
Starting hass
  * found guest in /usr/local/vm/hass
  * booting...

When I see in the log is:

Code: 
*** /usr/local/vm/hass/vm-bhyve.log ***
2026-04-23T14:47:12+00:00: initialising
2026-04-23T14:47:12+00:00:  [loader: uefi]
2026-04-23T14:47:12+00:00:  [cpu: 4]
2026-04-23T14:47:12+00:00:  [memory: 8GB]
2026-04-23T14:47:12+00:00:  [hostbridge: standard]
2026-04-23T14:47:12+00:00:  [com ports: com1]
2026-04-23T14:47:12+00:00:  [uuid: 9aae377a-6c06-11ed-a655-002590fa0f10]
2026-04-23T14:47:12+00:00:  [debug mode: no]
2026-04-23T14:47:12+00:00:  [primary disk: disk0.img]
2026-04-23T14:47:12+00:00:  [primary disk dev: file]
2026-04-23T14:47:12+00:00: initialising network device tap1
2026-04-23T14:47:12+00:00: adding tap1 -> bridge0 (public addm)
2026-04-23T14:47:12+00:00: bring up tap1 -> bridge0 (public addm)
2026-04-23T14:47:12+00:00: booting
2026-04-23T14:47:12+00:00:  [bhyve options: -c 4 -m 8GB -AHPw -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -A -U 9aae377a-6c06-11ed-a655-002590fa0f10 -u]
2026-04-23T14:47:12+00:00:  [bhyve devices: -s 0,hostbridge -s 31,lpc -s 0:4:0,virtio-blk,/usr/local/vm/hass/disk0.img -s 0:5:0,virtio-net,tap1,mac=58:9c:fc:08:5d:13]
2026-04-23T14:47:12+00:00:  [bhyve console: -l com1,/dev/nmdm-hass.1A]
2026-04-23T14:47:12+00:00: starting bhyve (run 1)
2026-04-23T14:47:12+00:00: bhyve exited with status 4
2026-04-23T14:47:12+00:00: destroying network device tap1
2026-04-23T14:47:12+00:00: stopped
 
I see recent updates:

Apr 21 17:29:03 r730-01 pkg[74249]: grub2-bhyve upgraded: 0.40_11 -> 0.40_12
Apr 21 17:29:14 r730-01 pkg[74249]: vm-bhyve upgraded: 1.7.0_1 -> 1.7.3


Let's try reverting some of those. Nope... running with this has the same "bhyve exited with status 4" message.

Code: 
grub2-bhyve-0.40_11
vm-bhyve-1.7.0_1
 
In case it's relevant, here are the other updates from yesterday. The host was rebooted at Tue Apr 21 17:39 - shortly after those pkg updates.

Interesting: the first Nagios alert was: [04-21-2026 17:51:52] HOST ALERT: ha;DOWN;HARD;10;CRITICAL - Socket timeout

Coincidence?

Code: 
Apr 21 17:29:02 r730-01 pkg[74249]: bacula15-client upgraded: 15.0.3 -> 15.0.3_1
Apr 21 17:29:02 r730-01 pkg[74249]: broot upgraded: 1.52.0_3 -> 1.52.0_4
Apr 21 17:29:03 r730-01 pkg[74249]: cpu-microcode-intel upgraded: 20260210 -> 20260227
Apr 21 17:29:03 r730-01 pkg[74249]: expat upgraded: 2.7.4 -> 2.7.5
Apr 21 17:29:03 r730-01 pkg[74249]: gnutls upgraded: 3.8.11 -> 3.8.12
Apr 21 17:29:03 r730-01 pkg[74249]: grub2-bhyve upgraded: 0.40_11 -> 0.40_12
Apr 21 17:29:03 r730-01 pkg[74249]: hwdata upgraded: 0.404,1 -> 0.406,1
Apr 21 17:29:03 r730-01 pkg[74249]: iperf3 upgraded: 3.20_1 -> 3.21
Apr 21 17:29:03 r730-01 pkg[74249]: jo upgraded: 1.7 -> 1.9
Apr 21 17:29:04 r730-01 pkg[74249]: libX11 upgraded: 1.8.12,1 -> 1.8.13,1
Apr 21 17:29:04 r730-01 pkg[74249]: at-spi2-core upgraded: 2.56.7 -> 2.56.8
Apr 21 17:29:04 r730-01 pkg[74249]: libgcrypt upgraded: 1.12.0_1 -> 1.12.2
Apr 21 17:29:04 r730-01 pkg[74249]: freeipmi upgraded: 1.6.16 -> 1.6.17
Apr 21 17:29:05 r730-01 pkg[74249]: libksba upgraded: 1.6.7 -> 1.6.8
Apr 21 17:29:05 r730-01 pkg[74249]: libtlsrpt-0.5.0_1 installed
Apr 21 17:29:05 r730-01 pkg[74249]: libuv upgraded: 1.52.0 -> 1.52.1
Apr 21 17:29:05 r730-01 pkg[74249]: libxml2 upgraded: 2.15.1_1 -> 2.15.2_1
Apr 21 17:29:05 r730-01 pkg[74249]: llhttp upgraded: 9.3.0 -> 9.3.1
Apr 21 17:29:05 r730-01 pkg[74249]: lmdb-0.9.35,1 installed
Apr 21 17:29:05 r730-01 pkg[74249]: lsblk upgraded: 4.0 -> 4.1_1
Apr 21 17:29:06 r730-01 pkg[74249]: mbuffer upgraded: 20250809 -> 20260301
Apr 21 17:29:06 r730-01 pkg[74249]: mpc upgraded: 1.3.1_1 -> 1.4.1
Apr 21 17:29:06 r730-01 pkg[74249]: nmap upgraded: 7.98 -> 7.99_1
Apr 21 17:29:07 r730-01 pkg[74249]: nut upgraded: 2.8.4 -> 2.8.5_1
Apr 21 17:29:07 r730-01 pkg[74249]: png upgraded: 1.6.55 -> 1.6.58
Apr 21 17:29:07 r730-01 pkg[74249]: freetype2 upgraded: 2.14.1 -> 2.14.3
Apr 21 17:29:07 r730-01 pkg[74249]: gstreamer1-plugins upgraded: 1.28.0 -> 1.28.1
Apr 21 17:29:07 r730-01 pkg[74249]: gstreamer1-plugins-gl upgraded: 1.28.0 -> 1.28.1
Apr 21 17:29:08 r730-01 pkg[74249]: harfbuzz upgraded: 12.3.2 -> 13.2.1
Apr 21 17:29:08 r730-01 pkg[74249]: gstreamer1-plugins-bad upgraded: 1.28.0 -> 1.28.1
Apr 21 17:29:08 r730-01 pkg[74249]: librsvg2-rust upgraded: 2.61.4 -> 2.62.0_1
Apr 21 17:29:08 r730-01 pkg[74249]: protobuf upgraded: 29.5,1 -> 29.6,1
Apr 21 17:29:08 r730-01 pkg[74249]: protobuf-c upgraded: 1.5.1_3 -> 1.5.1_4
Apr 21 17:29:11 r730-01 pkg[74249]: python312 upgraded: 3.12.12_4 -> 3.12.13_2
Apr 21 17:29:12 r730-01 pkg[74249]: gtk3 upgraded: 3.24.51 -> 3.24.52
Apr 21 17:29:12 r730-01 pkg[74249]: gtk4 upgraded: 4.18.6 -> 4.20.4
Apr 21 17:29:13 r730-01 pkg[74249]: sdl2 upgraded: 2.32.10 -> 2.32.10_1
Apr 21 17:29:13 r730-01 pkg[74249]: tinycdb-0.81 installed
Apr 21 17:29:14 r730-01 pkg[74249]: postfix upgraded: 3.10.6,1 -> 3.11.1_2,1
Apr 21 17:29:14 r730-01 pkg[74249]: vm-bhyve upgraded: 1.7.0_1 -> 1.7.3
Apr 21 17:29:14 r730-01 pkg[74249]: vte3 upgraded: 0.80.4 -> 0.80.5
Apr 21 17:29:14 r730-01 pkg[74249]: nagios-plugins-2.4.4_1,1 deinstalled
Apr 21 17:29:14 r730-01 pkg[74249]: bind-tools upgraded: 9.20.19 -> 9.20.22
Apr 21 17:29:14 r730-01 pkg[74249]: mysql80-client-8.0.45 deinstalled
Apr 21 17:29:15 r730-01 pkg[74249]: curl upgraded: 8.19.0_1 -> 8.19.0_2
Apr 21 17:29:15 r730-01 pkg[74249]: elfutils upgraded: 0.187_2 -> 0.187_3
Apr 21 17:29:15 r730-01 pkg[74249]: groff upgraded: 1.23.0_5 -> 1.24.1_1
Apr 21 17:29:16 r730-01 pkg[74249]: openldap26-client upgraded: 2.6.12 -> 2.6.13
Apr 21 17:29:16 r730-01 pkg[74249]: mysql84-client-8.4.8 installed
Apr 21 17:29:16 r730-01 pkg[74249]: postgresql18-client upgraded: 18.2_1 -> 18.3
Apr 21 17:29:16 r730-01 pkg[74249]: nagios-plugins-2.4.4_1,1 installed

Going back further:

Code: 
Apr 18 18:06:49 r730-01 pkg[73299]: pkg upgraded: 2.6.2_1 -> 2.7.4
Apr 18 18:06:50 r730-01 pkg[73368]: libnghttp2 upgraded: 1.68.0 -> 1.68.1
Apr 18 18:07:15 r730-01 pkg[73850]: jpeg-turbo upgraded: 3.1.3 -> 3.1.4.1
Apr 18 18:07:15 r730-01 pkg[73850]: lerc upgraded: 4.0.0 -> 4.1.0
Apr 18 18:07:15 r730-01 pkg[73850]: tiff upgraded: 4.7.1 -> 4.7.1_1
Apr 18 18:07:58 r730-01 pkg[75214]: gettext-runtime upgraded: 0.26 -> 1.0
Apr 18 18:07:58 r730-01 pkg[75214]: gstreamer1 upgraded: 1.28.0 -> 1.28.1_1
 
From what I understand, the vm-bhyve is a convenience layer.
So does it start without vm-bhyve?

Anther approach could be to look into the source what might trigger the rc=4 - that could be enlightening, or it could be devastating. ;)

I for my part have never felt the need to use vm-bhyve; instead I created my own rc.d runner script, as of here https://gitr.daemon.contact/tools/tree/rc.d/guest.[1] This one suits server operation and fits into my netgraph fabric. Obviouesly people are welcome to grab this and adapt as desired.

[1] This does currently take about a minute to open, due to unacceptable behaviour of Huawei etc.
 
> So does it start without vm-bhyve?

I am happy to try that if someone shows me how. I see the options listed above. Can it be that easy.

From what I know, byhve does get started, as shown in the logs.

Looking at your script, that's probably the reason I'm still using vm-bhyve - barriers to entry, etc.
 
dvl@ said:
> So does it start without vm-bhyve?

I am happy to try that if someone shows me how.
Click to expand...
I see. :rolleyes:
It is an ugly lenghty commandline and bhyve is quite picky and not very verbose.

What you could do, if that "vm" is simply a shell script, run it after inserting "set -v -x", then it should somewhere print that commandline. But then it still depends of the netif being set up correctly and maybe other things.
But then also you could compare these two invocations, and maybe that gives a clue.

dvl@ said:
From what I know, byhve does get started, as shown in the logs.
Click to expand...
Yes, and then it gets disappointed by whatever and terminates. I've seen this here too: as soon as for instance the CPUs are not correctly mapped, it just terminates.

dvl@ said:
Looking at your script, that's probably the reason I'm still using vm-bhyve - barriers to entry, etc.
Click to expand...
Sure. I wrote that once, I don't remember the exact details - but at least this doesn't change without me knowing. ;)
 
Maxnix said:
I'm probably wrong, but are you sure that /usr/local/sbin/grub-bhyve has the wxneeded feature enabled?
Click to expand...
I don't know and searching for that term doesn't help me answer your question, so:

Code: 
[22:07 r730-01 dvl ~] % ls -l /usr/local/sbin/grub-bhyve
-r-xr-xr-x  1 root wheel 1081496 2026.02.09 03:11 /usr/local/sbin/grub-bhyve
[22:07 r730-01 dvl ~] % file /usr/local/sbin/grub-bhyve
/usr/local/sbin/grub-bhyve: ELF 64-bit LSB executable, x86-64, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 15.0 (1500068), FreeBSD-style, stripped
[22:08 r730-01 dvl ~] % ldd /usr/local/sbin/grub-bhyve
/usr/local/sbin/grub-bhyve:
    libncursesw.so.9 => /lib/libncursesw.so.9 (0x29edd399f000)
    libtinfow.so.9 => /lib/libtinfow.so.9 (0x29edd5182000)
    libzfs.so.4 => /lib/libzfs.so.4 (0x29edd4178000)
    libnvpair.so.2 => /lib/libnvpair.so.2 (0x29edd5e6b000)
    libgeom.so.5 => /lib/libgeom.so.5 (0x29edd6e6f000)
    libvmmapi.so.7 => /usr/lib/libvmmapi.so.7 (0x29edd77a6000)
    libutil.so.10 => /lib/libutil.so.10 (0x29edd83dd000)
    libc.so.7 => /lib/libc.so.7 (0x29edd9c28000)
    libavl.so.2 => /lib/libavl.so.2 (0x29edd91d1000)
    libbsdxml.so.4 => /lib/libbsdxml.so.4 (0x29edda474000)
    libcrypto.so.35 => /lib/libcrypto.so.35 (0x29eddc000000)
    libm.so.5 => /lib/libm.so.5 (0x29eddccdd000)
    libmd.so.7 => /lib/libmd.so.7 (0x29eddb68b000)
    librt.so.1 => /lib/librt.so.1 (0x29eddd642000)
    libumem.so.2 => /lib/libumem.so.2 (0x29edde454000)
    libuutil.so.2 => /lib/libuutil.so.2 (0x29eddec55000)
    libz.so.6 => /lib/libz.so.6 (0x29eddf18d000)
    libzfs_core.so.2 => /lib/libzfs_core.so.2 (0x29eddfe68000)
    libzutil.so.2 => /lib/libzutil.so.2 (0x29ede02a5000)
    libthr.so.3 => /lib/libthr.so.3 (0x29ede08c0000)
    libspl.so.2 => /lib/libspl.so.2 (0x29ede144f000)
    libsbuf.so.6 => /lib/libsbuf.so.6 (0x29ede1d4d000)
    libsys.so.7 => /lib/libsys.so.7 (0x29ede2a11000)
    libtpool.so.2 => /lib/libtpool.so.2 (0x29edda7b3000)
    [vdso] (0x29edd31a4000)
[22:08 r730-01 dvl ~] %

pkg which /usr/local/sbin/grub-bhyve
/usr/local/sbin/grub-bhyve was installed by package grub2-bhyve-0.40_11
[22:09 r730-01 dvl ~] % pkg info grub2-bhyve
grub2-bhyve-0.40_11
Name           : grub2-bhyve
Version        : 0.40_11
Installed on   : Thu Apr 23 15:04:39 2026 UTC
Origin         : sysutils/grub2-bhyve
Architecture   : FreeBSD:15:amd64
Prefix         : /usr/local
Categories     : sysutils
Licenses       : GPLv3
Maintainer     : ports@FreeBSD.org
WWW            : https://github.com/grehan-freebsd/grub2-bhyve
Comment        : Grub-emu loader for bhyve
Shared Libs required:
    libc.so.7
    libgeom.so.5
    libncursesw.so.9
    libnvpair.so.2
    libtinfow.so.9
    libutil.so.10
    libvmmapi.so.7
    libzfs.so.4
Annotations    :
    FreeBSD_version: 1500068
    build_timestamp: 2026-02-09T03:08:14+0000
    built_by       : poudriere-git-3.4.4
    port_checkout_unclean: no
    port_git_hash  : 66ab465f4d15
    ports_top_checkout_unclean: yes
    ports_top_git_hash: cbfc3f6b15d6
Flat size      : 1.03MiB
Description    :
GNU GRUB is a multiboot boot loader.  It was derived from GRUB, the GRand
Unified Bootloader, which was originally designed and implemented by Erich
Stefan Boleyn.

This port builds the grub-bhyve binary, allowing booting of non-FreeBSD
operating systems in bhyve.
 
Maxnix said:
I'm probably wrong, but are you sure that /usr/local/sbin/grub-bhyve has the wxneeded feature enabled?
Click to expand...
Found it:

Code: 
[22:12 r730-01 dvl ~] % elfctl -l /usr/local/sbin/grub-bhyve
Known features are:
noaslr          Disable ASLR
noprotmax       Disable implicit PROT_MAX
nostackgap      Disable stack gap
wxneeded        Requires W+X mappings
la48            amd64: Limit user virtual addresses to 48 bits
la57            amd64: Allow the use of 57-bit virtual addresses when available
File '/usr/local/sbin/grub-bhyve' features:
noaslr          'Disable ASLR' is unset.
noprotmax       'Disable implicit PROT_MAX' is unset.
nostackgap      'Disable stack gap' is unset.
wxneeded        'Requires W+X mappings' is unset.
la48            'amd64: Limit user virtual addresses to 48 bits' is unset.
la57            'amd64: Allow the use of 57-bit virtual addresses when available' is unset.
[22:13 r730-01 dvl ~] %
 
dvl@ said:
Found it:

Code: 
[22:12 r730-01 dvl ~] % elfctl -l /usr/local/sbin/grub-bhyve
Known features are:
noaslr          Disable ASLR
noprotmax       Disable implicit PROT_MAX
nostackgap      Disable stack gap
wxneeded        Requires W+X mappings
la48            amd64: Limit user virtual addresses to 48 bits
la57            amd64: Allow the use of 57-bit virtual addresses when available
File '/usr/local/sbin/grub-bhyve' features:
noaslr          'Disable ASLR' is unset.
noprotmax       'Disable implicit PROT_MAX' is unset.
nostackgap      'Disable stack gap' is unset.
wxneeded        'Requires W+X mappings' is unset.
la48            'amd64: Limit user virtual addresses to 48 bits' is unset.
la57            'amd64: Allow the use of 57-bit virtual addresses when available' is unset.
[22:13 r730-01 dvl ~] %
Click to expand...
Comparing with a snapshot:

Code: 
[22:16 r730-01 dvl /.zfs/snapshot/autosnap_2026-04-14_00:00:14_daily/usr/local/sbin] % elfctl -l grub-bhyve
Known features are:
noaslr          Disable ASLR
noprotmax       Disable implicit PROT_MAX
nostackgap      Disable stack gap
wxneeded        Requires W+X mappings
la48            amd64: Limit user virtual addresses to 48 bits
la57            amd64: Allow the use of 57-bit virtual addresses when available
File 'grub-bhyve' features:
noaslr          'Disable ASLR' is unset.
noprotmax       'Disable implicit PROT_MAX' is unset.
nostackgap      'Disable stack gap' is unset.
wxneeded        'Requires W+X mappings' is unset.
la48            'amd64: Limit user virtual addresses to 48 bits' is unset.
la57            'amd64: Allow the use of 57-bit virtual addresses when available' is unset.
[22:16 r730-01 dvl /.zfs/snapshot/autosnap_2026-04-14_00:00:14_daily/usr/local/sbin] %
 
You must log in or register to reply here.
Back
Top