Solved bhyve (via vm) guest no longer starts - bhyve exited with status 4

[dvl@]

EDIT: my solution was to load the nmdm kernel module.

I've managed to get this vm running via vm start -f hass on FreeBSD 15 with vm-bhyve-1.7.3

Recently, I was moving /usr/local/vm to a new ZFS dataset. This problem occurred there, so I went back to the original dataset and tried it there. Both datasets are not active at the same time (imagine data02/vm (original and what I'm using for this post) and data04/vm (new))

It starts in the foreground, with interesting messages:

GNU GRUB  version 2.12
   *Slot B (OK=1 TRY=1)
    Slot A (rescue shell)
    Slot B (rescue shell)
 
     Booting `Slot B (OK=1 TRY=1)'

None of the above is interactive (i.e. I'm not selecting anything).

The VM (Home Assistant) comes to a login prompt, I can login, etc.

From the logs, I see:

*** /usr/local/vm/hass/vm-bhyve.log ***
2026-04-23T14:48:47+00:00: initialising
2026-04-23T14:48:47+00:00:  [loader: uefi]
2026-04-23T14:48:47+00:00:  [cpu: 4]
2026-04-23T14:48:47+00:00:  [memory: 8GB]
2026-04-23T14:48:47+00:00:  [hostbridge: standard]
2026-04-23T14:48:47+00:00:  [com ports: com1]
2026-04-23T14:48:47+00:00:  [uuid: 9aae377a-6c06-11ed-a655-002590fa0f10]
2026-04-23T14:48:47+00:00:  [debug mode: no]
2026-04-23T14:48:47+00:00:  [primary disk: disk0.img]
2026-04-23T14:48:47+00:00:  [primary disk dev: file]
2026-04-23T14:48:47+00:00: initialising network device tap1
2026-04-23T14:48:47+00:00: adding tap1 -> bridge0 (public addm)
2026-04-23T14:48:47+00:00: bring up tap1 -> bridge0 (public addm)
2026-04-23T14:48:47+00:00: booting
2026-04-23T14:48:47+00:00:  [bhyve options: -c 4 -m 8GB -AHPw -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -A -U 9aae377a-6c06-11ed-a655-002590fa0f10 -u]
2026-04-23T14:48:47+00:00:  [bhyve devices: -s 0,hostbridge -s 31,lpc -s 0:4:0,virtio-blk,/usr/local/vm/hass/disk0.img -s 0:5:0,virtio-net,tap1,mac=58:9c:fc:08:5d:13]
2026-04-23T14:48:47+00:00:  [bhyve console: -l com1,stdio]
2026-04-23T14:48:47+00:00: starting bhyve (run 1)

*** /var/log/messages ***
Apr 23 14:48:47 r730-01 kernel: tap1: Ethernet address: 58:9c:fc:10:e2:5d
Apr 23 14:48:47 r730-01 kernel: tap1: promiscuous mode enabled
Apr 23 14:48:48 r730-01 kernel: tap1: link state changed to UP

Next, I run (from another ssh connection)

% sudo vm stop hass
Sending ACPI shutdown to hass

And it stops.

However, I am unable to get the VM to run via:

% sudo vm start hass
Starting hass
  * found guest in /usr/local/vm/hass
  * booting...

When I see in the log is:

*** /usr/local/vm/hass/vm-bhyve.log ***
2026-04-23T14:47:12+00:00: initialising
2026-04-23T14:47:12+00:00:  [loader: uefi]
2026-04-23T14:47:12+00:00:  [cpu: 4]
2026-04-23T14:47:12+00:00:  [memory: 8GB]
2026-04-23T14:47:12+00:00:  [hostbridge: standard]
2026-04-23T14:47:12+00:00:  [com ports: com1]
2026-04-23T14:47:12+00:00:  [uuid: 9aae377a-6c06-11ed-a655-002590fa0f10]
2026-04-23T14:47:12+00:00:  [debug mode: no]
2026-04-23T14:47:12+00:00:  [primary disk: disk0.img]
2026-04-23T14:47:12+00:00:  [primary disk dev: file]
2026-04-23T14:47:12+00:00: initialising network device tap1
2026-04-23T14:47:12+00:00: adding tap1 -> bridge0 (public addm)
2026-04-23T14:47:12+00:00: bring up tap1 -> bridge0 (public addm)
2026-04-23T14:47:12+00:00: booting
2026-04-23T14:47:12+00:00:  [bhyve options: -c 4 -m 8GB -AHPw -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -A -U 9aae377a-6c06-11ed-a655-002590fa0f10 -u]
2026-04-23T14:47:12+00:00:  [bhyve devices: -s 0,hostbridge -s 31,lpc -s 0:4:0,virtio-blk,/usr/local/vm/hass/disk0.img -s 0:5:0,virtio-net,tap1,mac=58:9c:fc:08:5d:13]
2026-04-23T14:47:12+00:00:  [bhyve console: -l com1,/dev/nmdm-hass.1A]
2026-04-23T14:47:12+00:00: starting bhyve (run 1)
2026-04-23T14:47:12+00:00: bhyve exited with status 4
2026-04-23T14:47:12+00:00: destroying network device tap1
2026-04-23T14:47:12+00:00: stopped

[/FILE][/FILE]

 

[dvl@]

I see recent updates:

Apr 21 17:29:03 r730-01 pkg[74249]: grub2-bhyve upgraded: 0.40_11 -> 0.40_12
Apr 21 17:29:14 r730-01 pkg[74249]: vm-bhyve upgraded: 1.7.0_1 -> 1.7.3


Let's try reverting some of those. Nope... running with this has the same "bhyve exited with status 4" message.

grub2-bhyve-0.40_11
vm-bhyve-1.7.0_1

 

[dvl@]

In case it's relevant, here are the other updates from yesterday. The host was rebooted at Tue Apr 21 17:39 - shortly after those pkg updates.

Interesting: the first Nagios alert was: [04-21-2026 17:51:52] HOST ALERT: ha;DOWN;HARD;10;CRITICAL - Socket timeout

Coincidence?

Apr 21 17:29:02 r730-01 pkg[74249]: bacula15-client upgraded: 15.0.3 -> 15.0.3_1
Apr 21 17:29:02 r730-01 pkg[74249]: broot upgraded: 1.52.0_3 -> 1.52.0_4
Apr 21 17:29:03 r730-01 pkg[74249]: cpu-microcode-intel upgraded: 20260210 -> 20260227
Apr 21 17:29:03 r730-01 pkg[74249]: expat upgraded: 2.7.4 -> 2.7.5
Apr 21 17:29:03 r730-01 pkg[74249]: gnutls upgraded: 3.8.11 -> 3.8.12
Apr 21 17:29:03 r730-01 pkg[74249]: grub2-bhyve upgraded: 0.40_11 -> 0.40_12
Apr 21 17:29:03 r730-01 pkg[74249]: hwdata upgraded: 0.404,1 -> 0.406,1
Apr 21 17:29:03 r730-01 pkg[74249]: iperf3 upgraded: 3.20_1 -> 3.21
Apr 21 17:29:03 r730-01 pkg[74249]: jo upgraded: 1.7 -> 1.9
Apr 21 17:29:04 r730-01 pkg[74249]: libX11 upgraded: 1.8.12,1 -> 1.8.13,1
Apr 21 17:29:04 r730-01 pkg[74249]: at-spi2-core upgraded: 2.56.7 -> 2.56.8
Apr 21 17:29:04 r730-01 pkg[74249]: libgcrypt upgraded: 1.12.0_1 -> 1.12.2
Apr 21 17:29:04 r730-01 pkg[74249]: freeipmi upgraded: 1.6.16 -> 1.6.17
Apr 21 17:29:05 r730-01 pkg[74249]: libksba upgraded: 1.6.7 -> 1.6.8
Apr 21 17:29:05 r730-01 pkg[74249]: libtlsrpt-0.5.0_1 installed
Apr 21 17:29:05 r730-01 pkg[74249]: libuv upgraded: 1.52.0 -> 1.52.1
Apr 21 17:29:05 r730-01 pkg[74249]: libxml2 upgraded: 2.15.1_1 -> 2.15.2_1
Apr 21 17:29:05 r730-01 pkg[74249]: llhttp upgraded: 9.3.0 -> 9.3.1
Apr 21 17:29:05 r730-01 pkg[74249]: lmdb-0.9.35,1 installed
Apr 21 17:29:05 r730-01 pkg[74249]: lsblk upgraded: 4.0 -> 4.1_1
Apr 21 17:29:06 r730-01 pkg[74249]: mbuffer upgraded: 20250809 -> 20260301
Apr 21 17:29:06 r730-01 pkg[74249]: mpc upgraded: 1.3.1_1 -> 1.4.1
Apr 21 17:29:06 r730-01 pkg[74249]: nmap upgraded: 7.98 -> 7.99_1
Apr 21 17:29:07 r730-01 pkg[74249]: nut upgraded: 2.8.4 -> 2.8.5_1
Apr 21 17:29:07 r730-01 pkg[74249]: png upgraded: 1.6.55 -> 1.6.58
Apr 21 17:29:07 r730-01 pkg[74249]: freetype2 upgraded: 2.14.1 -> 2.14.3
Apr 21 17:29:07 r730-01 pkg[74249]: gstreamer1-plugins upgraded: 1.28.0 -> 1.28.1
Apr 21 17:29:07 r730-01 pkg[74249]: gstreamer1-plugins-gl upgraded: 1.28.0 -> 1.28.1
Apr 21 17:29:08 r730-01 pkg[74249]: harfbuzz upgraded: 12.3.2 -> 13.2.1
Apr 21 17:29:08 r730-01 pkg[74249]: gstreamer1-plugins-bad upgraded: 1.28.0 -> 1.28.1
Apr 21 17:29:08 r730-01 pkg[74249]: librsvg2-rust upgraded: 2.61.4 -> 2.62.0_1
Apr 21 17:29:08 r730-01 pkg[74249]: protobuf upgraded: 29.5,1 -> 29.6,1
Apr 21 17:29:08 r730-01 pkg[74249]: protobuf-c upgraded: 1.5.1_3 -> 1.5.1_4
Apr 21 17:29:11 r730-01 pkg[74249]: python312 upgraded: 3.12.12_4 -> 3.12.13_2
Apr 21 17:29:12 r730-01 pkg[74249]: gtk3 upgraded: 3.24.51 -> 3.24.52
Apr 21 17:29:12 r730-01 pkg[74249]: gtk4 upgraded: 4.18.6 -> 4.20.4
Apr 21 17:29:13 r730-01 pkg[74249]: sdl2 upgraded: 2.32.10 -> 2.32.10_1
Apr 21 17:29:13 r730-01 pkg[74249]: tinycdb-0.81 installed
Apr 21 17:29:14 r730-01 pkg[74249]: postfix upgraded: 3.10.6,1 -> 3.11.1_2,1
Apr 21 17:29:14 r730-01 pkg[74249]: vm-bhyve upgraded: 1.7.0_1 -> 1.7.3
Apr 21 17:29:14 r730-01 pkg[74249]: vte3 upgraded: 0.80.4 -> 0.80.5
Apr 21 17:29:14 r730-01 pkg[74249]: nagios-plugins-2.4.4_1,1 deinstalled
Apr 21 17:29:14 r730-01 pkg[74249]: bind-tools upgraded: 9.20.19 -> 9.20.22
Apr 21 17:29:14 r730-01 pkg[74249]: mysql80-client-8.0.45 deinstalled
Apr 21 17:29:15 r730-01 pkg[74249]: curl upgraded: 8.19.0_1 -> 8.19.0_2
Apr 21 17:29:15 r730-01 pkg[74249]: elfutils upgraded: 0.187_2 -> 0.187_3
Apr 21 17:29:15 r730-01 pkg[74249]: groff upgraded: 1.23.0_5 -> 1.24.1_1
Apr 21 17:29:16 r730-01 pkg[74249]: openldap26-client upgraded: 2.6.12 -> 2.6.13
Apr 21 17:29:16 r730-01 pkg[74249]: mysql84-client-8.4.8 installed
Apr 21 17:29:16 r730-01 pkg[74249]: postgresql18-client upgraded: 18.2_1 -> 18.3
Apr 21 17:29:16 r730-01 pkg[74249]: nagios-plugins-2.4.4_1,1 installed

Going back further:

Apr 18 18:06:49 r730-01 pkg[73299]: pkg upgraded: 2.6.2_1 -> 2.7.4
Apr 18 18:06:50 r730-01 pkg[73368]: libnghttp2 upgraded: 1.68.0 -> 1.68.1
Apr 18 18:07:15 r730-01 pkg[73850]: jpeg-turbo upgraded: 3.1.3 -> 3.1.4.1
Apr 18 18:07:15 r730-01 pkg[73850]: lerc upgraded: 4.0.0 -> 4.1.0
Apr 18 18:07:15 r730-01 pkg[73850]: tiff upgraded: 4.7.1 -> 4.7.1_1
Apr 18 18:07:58 r730-01 pkg[75214]: gettext-runtime upgraded: 0.26 -> 1.0
Apr 18 18:07:58 r730-01 pkg[75214]: gstreamer1 upgraded: 1.28.0 -> 1.28.1_1

 

[PMc]

From what I understand, the vm-bhyve is a convenience layer.
So does it start without vm-bhyve?

Anther approach could be to look into the source what might trigger the rc=4 - that could be enlightening, or it could be devastating.

I for my part have never felt the need to use vm-bhyve; instead I created my own rc.d runner script, as of here https://gitr.daemon.contact/tools/tree/rc.d/guest.[1] This one suits server operation and fits into my netgraph fabric. Obviouesly people are welcome to grab this and adapt as desired.

[1] This does currently take about a minute to open, due to unacceptable behaviour of Huawei etc.

 

[dvl@]

> So does it start without vm-bhyve?

I am happy to try that if someone shows me how. I see the options listed above. Can it be that easy.

From what I know, byhve does get started, as shown in the logs.

Looking at your script, that's probably the reason I'm still using vm-bhyve - barriers to entry, etc.

 

[PMc]

> So does it start without vm-bhyve?

I am happy to try that if someone shows me how.

I see.
It is an ugly lenghty commandline and bhyve is quite picky and not very verbose.

What you could do, if that "vm" is simply a shell script, run it after inserting "set -v -x", then it should somewhere print that commandline. But then it still depends of the netif being set up correctly and maybe other things.
But then also you could compare these two invocations, and maybe that gives a clue.

From what I know, byhve does get started, as shown in the logs.

Yes, and then it gets disappointed by whatever and terminates. I've seen this here too: as soon as for instance the CPUs are not correctly mapped, it just terminates.

Looking at your script, that's probably the reason I'm still using vm-bhyve - barriers to entry, etc.

Sure. I wrote that once, I don't remember the exact details - but at least this doesn't change without me knowing.

 

[Maxnix]

I'm probably wrong, but are you sure that /usr/local/sbin/grub-bhyve has the wxneeded feature enabled?

 

[dvl@]

I'm probably wrong, but are you sure that /usr/local/sbin/grub-bhyve has the wxneeded feature enabled?

I don't know and searching for that term doesn't help me answer your question, so:

[22:07 r730-01 dvl ~] % ls -l /usr/local/sbin/grub-bhyve
-r-xr-xr-x  1 root wheel 1081496 2026.02.09 03:11 /usr/local/sbin/grub-bhyve
[22:07 r730-01 dvl ~] % file /usr/local/sbin/grub-bhyve
/usr/local/sbin/grub-bhyve: ELF 64-bit LSB executable, x86-64, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 15.0 (1500068), FreeBSD-style, stripped
[22:08 r730-01 dvl ~] % ldd /usr/local/sbin/grub-bhyve
/usr/local/sbin/grub-bhyve:
    libncursesw.so.9 => /lib/libncursesw.so.9 (0x29edd399f000)
    libtinfow.so.9 => /lib/libtinfow.so.9 (0x29edd5182000)
    libzfs.so.4 => /lib/libzfs.so.4 (0x29edd4178000)
    libnvpair.so.2 => /lib/libnvpair.so.2 (0x29edd5e6b000)
    libgeom.so.5 => /lib/libgeom.so.5 (0x29edd6e6f000)
    libvmmapi.so.7 => /usr/lib/libvmmapi.so.7 (0x29edd77a6000)
    libutil.so.10 => /lib/libutil.so.10 (0x29edd83dd000)
    libc.so.7 => /lib/libc.so.7 (0x29edd9c28000)
    libavl.so.2 => /lib/libavl.so.2 (0x29edd91d1000)
    libbsdxml.so.4 => /lib/libbsdxml.so.4 (0x29edda474000)
    libcrypto.so.35 => /lib/libcrypto.so.35 (0x29eddc000000)
    libm.so.5 => /lib/libm.so.5 (0x29eddccdd000)
    libmd.so.7 => /lib/libmd.so.7 (0x29eddb68b000)
    librt.so.1 => /lib/librt.so.1 (0x29eddd642000)
    libumem.so.2 => /lib/libumem.so.2 (0x29edde454000)
    libuutil.so.2 => /lib/libuutil.so.2 (0x29eddec55000)
    libz.so.6 => /lib/libz.so.6 (0x29eddf18d000)
    libzfs_core.so.2 => /lib/libzfs_core.so.2 (0x29eddfe68000)
    libzutil.so.2 => /lib/libzutil.so.2 (0x29ede02a5000)
    libthr.so.3 => /lib/libthr.so.3 (0x29ede08c0000)
    libspl.so.2 => /lib/libspl.so.2 (0x29ede144f000)
    libsbuf.so.6 => /lib/libsbuf.so.6 (0x29ede1d4d000)
    libsys.so.7 => /lib/libsys.so.7 (0x29ede2a11000)
    libtpool.so.2 => /lib/libtpool.so.2 (0x29edda7b3000)
    [vdso] (0x29edd31a4000)
[22:08 r730-01 dvl ~] %

pkg which /usr/local/sbin/grub-bhyve
/usr/local/sbin/grub-bhyve was installed by package grub2-bhyve-0.40_11
[22:09 r730-01 dvl ~] % pkg info grub2-bhyve
grub2-bhyve-0.40_11
Name           : grub2-bhyve
Version        : 0.40_11
Installed on   : Thu Apr 23 15:04:39 2026 UTC
Origin         : sysutils/grub2-bhyve
Architecture   : FreeBSD:15:amd64
Prefix         : /usr/local
Categories     : sysutils
Licenses       : GPLv3
Maintainer     : ports@FreeBSD.org
WWW            : https://github.com/grehan-freebsd/grub2-bhyve
Comment        : Grub-emu loader for bhyve
Shared Libs required:
    libc.so.7
    libgeom.so.5
    libncursesw.so.9
    libnvpair.so.2
    libtinfow.so.9
    libutil.so.10
    libvmmapi.so.7
    libzfs.so.4
Annotations    :
    FreeBSD_version: 1500068
    build_timestamp: 2026-02-09T03:08:14+0000
    built_by       : poudriere-git-3.4.4
    port_checkout_unclean: no
    port_git_hash  : 66ab465f4d15
    ports_top_checkout_unclean: yes
    ports_top_git_hash: cbfc3f6b15d6
Flat size      : 1.03MiB
Description    :
GNU GRUB is a multiboot boot loader.  It was derived from GRUB, the GRand
Unified Bootloader, which was originally designed and implemented by Erich
Stefan Boleyn.

This port builds the grub-bhyve binary, allowing booting of non-FreeBSD
operating systems in bhyve.

 

[dvl@]

I'm probably wrong, but are you sure that /usr/local/sbin/grub-bhyve has the wxneeded feature enabled?

Found it:

[22:12 r730-01 dvl ~] % elfctl -l /usr/local/sbin/grub-bhyve
Known features are:
noaslr          Disable ASLR
noprotmax       Disable implicit PROT_MAX
nostackgap      Disable stack gap
wxneeded        Requires W+X mappings
la48            amd64: Limit user virtual addresses to 48 bits
la57            amd64: Allow the use of 57-bit virtual addresses when available
File '/usr/local/sbin/grub-bhyve' features:
noaslr          'Disable ASLR' is unset.
noprotmax       'Disable implicit PROT_MAX' is unset.
nostackgap      'Disable stack gap' is unset.
wxneeded        'Requires W+X mappings' is unset.
la48            'amd64: Limit user virtual addresses to 48 bits' is unset.
la57            'amd64: Allow the use of 57-bit virtual addresses when available' is unset.
[22:13 r730-01 dvl ~] %

 

[dvl@]

Found it:

[22:12 r730-01 dvl ~] % elfctl -l /usr/local/sbin/grub-bhyve
Known features are:
noaslr          Disable ASLR
noprotmax       Disable implicit PROT_MAX
nostackgap      Disable stack gap
wxneeded        Requires W+X mappings
la48            amd64: Limit user virtual addresses to 48 bits
la57            amd64: Allow the use of 57-bit virtual addresses when available
File '/usr/local/sbin/grub-bhyve' features:
noaslr          'Disable ASLR' is unset.
noprotmax       'Disable implicit PROT_MAX' is unset.
nostackgap      'Disable stack gap' is unset.
wxneeded        'Requires W+X mappings' is unset.
la48            'amd64: Limit user virtual addresses to 48 bits' is unset.
la57            'amd64: Allow the use of 57-bit virtual addresses when available' is unset.
[22:13 r730-01 dvl ~] %

Comparing with a snapshot:

[22:16 r730-01 dvl /.zfs/snapshot/autosnap_2026-04-14_00:00:14_daily/usr/local/sbin] % elfctl -l grub-bhyve
Known features are:
noaslr          Disable ASLR
noprotmax       Disable implicit PROT_MAX
nostackgap      Disable stack gap
wxneeded        Requires W+X mappings
la48            amd64: Limit user virtual addresses to 48 bits
la57            amd64: Allow the use of 57-bit virtual addresses when available
File 'grub-bhyve' features:
noaslr          'Disable ASLR' is unset.
noprotmax       'Disable implicit PROT_MAX' is unset.
nostackgap      'Disable stack gap' is unset.
wxneeded        'Requires W+X mappings' is unset.
la48            'amd64: Limit user virtual addresses to 48 bits' is unset.
la57            'amd64: Allow the use of 57-bit virtual addresses when available' is unset.
[22:16 r730-01 dvl /.zfs/snapshot/autosnap_2026-04-14_00:00:14_daily/usr/local/sbin] %

 

[dvl@]

After adding -x to the top of /usr/local/sbin/vm, sudo vm start -f hass revealed:

+ bhyve -c 4 -m 8GB -AHPw -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
-A -U 9aae377a-6c06-11ed-a655-002590fa0f10 -u -s 0,hostbridge -s 31,lpc \
-s 0:4:0,virtio-blk,/usr/local/vm/hass/disk0.img -s '0:5:0,virtio-net,tap1,mac=58:9c:fc:08:5d:13' \
-l com1,stdio hass

The \ have been inserted by myself to make this easier to read here.

Running that command as root, Home Assistant starts, but I cannot access via the browser. The IP address cannot be pinged.

My next task: figure out how to stop this VM and I can it running again (via the old method).

Solution: enter shutdown after logged in as root. Wait for to shutdown, then issues this command: bhyvectl --vm=hass --destroy - if I didn't enter that command, sudo vm start -f hass would give

guest appears to be running already

 

[PMc]

Running that command as root, Home Assistant starts, but I cannot access via the browser. The IP address cannot be pinged.

Good. A failing network was to be expected. The tap1 interface needs to exist before bhyve can use it.

bhyvectl --vm=hass --destroy

Exactly. "shutdown" terminates the OS, "bhyvectl --destroy" removes the bhyve and restores a pristine condition.

 

[dvl@]

This seems to be the main difference between -f and not using -f:

This is the failed start:

2026-04-25T17:31:05+00:00:  [bhyve console: -l com1,/dev/nmdm-hass.1A]

This is the start with -f which succeeds:

2026-04-25T17:32:01+00:00:  [bhyve console: -l com1,stdio]

Could it be a problem related to /dev/nmdm-hass.1A ?

 

[dvl@]

I tried creating a second HASS instance. It has the same start problem.

I was able to create a new VM and get it running, but only when using -f

Followed https://community.home-assistant.io/t/homeassistant-on-freebsd-using-bhyve/859127/1

  [16:40 r730-01 dvl ~] % pkg info -x vm-bhyve bhyve-firmware qemu-tools
vm-bhyve-1.7.0_1
pkg: No package(s) matching bhyve-firmware
[16:41 r730-01 dvl ~] % pkg search firmware
FreeBSD-firmware-iwm-15.0      Firmware for iwm(4) Intel 802.11ac network interfaces
bhyve-firmware-1.0_2           Collection of Firmware for bhyve

It's not qemu-tools here, I have qemu installed.

[16:41 r730-01 dvl ~] % sudo pkg install bhyve-firmware
Updating FreeBSD-base repository catalogue...
FreeBSD-base repository is up to date.
Updating local repository catalogue...
Fetching meta.conf: 100%     179 B   0.2 kB/s    00:01  
Fetching data: 100%   364 KiB 372.6 kB/s    00:01  
Processing entries: 100%
local repository update completed. 996 packages processed.
All repositories are up to date.
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
    bhyve-firmware: 1.0_2 [local]

Number of packages to be installed: 1

577 B to be downloaded.

Proceed with this action? [y/N]: y
[1/1] Fetching bhyve-firmware-1.0_2: 100%     577 B   0.6 kB/s    00:01  
Checking integrity... done (0 conflicting)
[1/1] Installing bhyve-firmware-1.0_2...
[16:41 r730-01 dvl ~] % grep bhyve-firmware /var/log/messages*
/var/log/messages:Apr 25 16:41:45 r730-01 pkg[51784]: bhyve-firmware-1.0_2 installed
[16:41 r730-01 dvl ~] % ls -l /usr/local/share/examples/vm-bhyve/*
-rw-r--r--  1 root wheel 17137 2025.11.15 14:07 /usr/local/share/examples/vm-bhyve/config.sample
-rw-r--r--  1 root wheel   136 2025.11.15 14:07 /usr/local/share/examples/vm-bhyve/default.conf
-rw-r--r--  1 root wheel   460 2025.11.15 14:07 /usr/local/share/examples/vm-bhyve/dragonfly.conf
-rw-r--r--  1 root wheel   156 2025.11.15 14:07 /usr/local/share/examples/vm-bhyve/freebsd-zvol.conf
-rw-r--r--  1 root wheel   755 2025.11.15 14:07 /usr/local/share/examples/vm-bhyve/freepbx.conf
-rw-r--r--  1 root wheel  1005 2025.11.15 14:07 /usr/local/share/examples/vm-bhyve/legacy-alpine.conf
-rw-r--r--  1 root wheel   848 2025.11.15 14:07 /usr/local/share/examples/vm-bhyve/legacy-arch.conf
-rw-r--r--  1 root wheel   745 2025.11.15 14:07 /usr/local/share/examples/vm-bhyve/legacy-debian.conf
-rw-r--r--  1 root wheel  1036 2025.11.15 14:07 /usr/local/share/examples/vm-bhyve/legacy-gentoo.conf
-rw-r--r--  1 root wheel   700 2025.11.15 14:07 /usr/local/share/examples/vm-bhyve/legacy-ubuntu.conf
-rw-r--r--  1 root wheel   400 2025.11.15 14:07 /usr/local/share/examples/vm-bhyve/linux-grub-zvol.conf
-rw-r--r--  1 root wheel   375 2025.11.15 14:07 /usr/local/share/examples/vm-bhyve/linux-grub.conf
-rw-r--r--  1 root wheel   261 2025.11.15 14:07 /usr/local/share/examples/vm-bhyve/linux-zvol.conf
-rw-r--r--  1 root wheel   241 2025.11.15 14:07 /usr/local/share/examples/vm-bhyve/linux.conf
-rw-r--r--  1 root wheel   212 2025.11.15 14:07 /usr/local/share/examples/vm-bhyve/netbsd.conf
-rw-r--r--  1 root wheel   452 2025.11.15 14:07 /usr/local/share/examples/vm-bhyve/openbsd.conf
-rw-r--r--  1 root wheel   222 2025.11.15 14:07 /usr/local/share/examples/vm-bhyve/resflash.conf
-rw-r--r--  1 root wheel   595 2025.11.15 14:07 /usr/local/share/examples/vm-bhyve/windows.conf
[16:42 r730-01 dvl ~] % sudo vm img https://github.com/home-assistant/operating-system/releases/download/14.2/haos_ova-14.2.qcow2.xz

/usr/local/vm/.img/haos_ova-14.2.qcow2.xz              369 MB   36 MBps    10s
[16:43 r730-01 dvl ~] % sudo vm img https://github.com/home-assistant/operating-system/releases/download/17.2/haos_ova-17.2.qcow2.xz

/usr/local/vm/.img/haos_ova-17.2.qcow2.xz              529 MB   36 MBps    15s
[16:44 r730-01 dvl ~] % vm create -t debian -c 4 -s 64G -m 8GB -i haos_ova-17.2.qcow2 hass2
/usr/local/sbin/vm: ERROR: virtual machines can only be managed by root
[16:45 r730-01 dvl ~] % sudo vm create -t debian -c 4 -s 64G -m 8GB -i haos_ova-17.2.qcow2 hass2
WARNING: Image format was not specified for '/usr/local/vm/hass2/disk0.img' and probing guessed raw.
         Automatically detecting the format is dangerous for raw images, write operations on block 0 will be restricted.
         Specify the 'raw' format explicitly to remove the restrictions.
Image resized.
[16:45 r730-01 dvl ~] % sudoedit /usr/local/vm/hass2/hass2.conf



^ resulting file:

[16:49 r730-01 dvl ~] % cat /usr/local/vm/hass2/hass2.conf
loader="uefi"
cpu="4"
memory="8GB"
network0_type="virtio-net"
network0_switch="public"
disk0_type="ahci-hd"
disk0_name="disk0.img"
grub_run_partition="1"
grub_run_dir="/boot/grub"
uuid="246bf908-40c6-11f1-95e6-ecf4bbefc954"
network0_mac="58:9c:fc:0a:ed:7c"


[16:47 r730-01 dvl ~] % sudo vm start hass2
Starting hass2
  * found guest in /usr/local/vm/hass2
  * booting...
[16:47 r730-01 dvl ~] % sudo vm list
NAME   DATASTORE  LOADER  CPU  MEMORY  VNC  AUTO     STATE
hass   default    uefi    4    8GB     -    Yes [1]  Running (14865)
hass2  default    uefi    4    8GB     -    No       Stopped


Nope. Still does not run.

Tried updating:

    at-spi2-core: 2.56.8 -> 2.60.1 [local]
    qemu: 10.2.2 -> 11.0.0 [local]


Still won't start.

Then I reverted the downgraded I had done after discovering the problem.

vm-bhyve: 1.7.0_1 -> 1.7.3

Did not help.

Then tried:     grub2-bhyve: 0.40_11 -> 0.40_12 [local]


Still getting bhyve exited with status 4

 

[PMc]

Could it be a problem related to /dev/nmdm-hass.1A ?

This looks strange. I have never seen such devices, all of mine are numbered /dev/nmdm0A, /dev/nmdm1A, /dev/nmdm2A, ...

 

[dvl@]

This looks strange. I have never seen such devices, all of mine are numbered /dev/nmdm0A, /dev/nmdm1A, /dev/nmdm2A, ...

At run time, when that one running in a tmux session, I have no matches:

[21:04 r730-01 dvl ~] % ls -l /dev/nm*
zsh: no matches found: /dev/nm*
[21:04 r730-01 dvl ~] %

 

[DtxdF]

Sometimes, setting
debug=YES
(and checking the bhyve.log file) can be helpful when debugging.

 

[CeXP1917]

loader="uefi"
...
grub_run_partition="1"
grub_run_dir="/boot/grub"

Afaik, if you use eufi loader, you dont need grub (but need vnc).

 

[Emrion]

This looks strange. I have never seen such devices, all of mine are numbered /dev/nmdm0A, /dev/nmdm1A, /dev/nmdm2A, ...

You can name this at the VM start. IIRC, the name must begins by "nmdm". In my own script, it's set as:
/dev/nmdm${VMname}A

 

[Emrion]

I tried a Home Assistant VM, starting with uefi and adding a nmdm line, it works (but the nmdm connexion doesn't, it's not echoing my typings).
I can connect to HA with the url.

The bhyve command line is:

bhyve -AHP -c 2 -m 4G -s 0,hostbridge -s 1,nvme,/data/kvm/vm/ha/haos.img,sectorzise=512  \
-s 31,lpc -l com1,/dev/nmdmhaA -s 29,xhci,tablet -s 30,fbuf,tcp=0.0.0.0:5900 -s 4,virtio-net,tapHA \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd,/data/kvm/vm/ha/UEFI_VARS.fd ha

I don't use a serial console for the VMs since one year at least. All is using UEFI and get connected with vnc (as long as it's a test VM or during the testings, after what I disable this unsecure connexion).

 

[dvl@]

Sometimes, setting
debug=YES
(and checking the bhyve.log file) can be helpful when debugging.

Setting where? I tried /usr/local/vm/hass2/hass2.conf and then

[12:37 r730-01 dvl /usr/local/vm/hass2] % sudo vm start hass2
Starting hass2
  * found guest in /usr/local/vm/hass2
  * booting...

The output looks similar (see above) to that without the debug directive. From /usr/local/vm/hass2/vm-bhyve.log:

2026-04-26T12:38:06+00:00: initialising
2026-04-26T12:38:06+00:00:  [loader: uefi]
2026-04-26T12:38:07+00:00:  [cpu: 4]
2026-04-26T12:38:07+00:00:  [memory: 8GB]
2026-04-26T12:38:07+00:00:  [hostbridge: standard]
2026-04-26T12:38:07+00:00:  [com ports: com1]
2026-04-26T12:38:07+00:00:  [uuid: 246bf908-40c6-11f1-95e6-ecf4bbefc954]
2026-04-26T12:38:07+00:00:  [debug mode: YES]
2026-04-26T12:38:07+00:00:  [primary disk: disk0.img]
2026-04-26T12:38:07+00:00:  [primary disk dev: file]
2026-04-26T12:38:07+00:00: initialising network device tap3
2026-04-26T12:38:07+00:00: adding tap3 -> bridge0 (public addm)
2026-04-26T12:38:07+00:00: bring up tap3 -> bridge0 (public addm)
2026-04-26T12:38:08+00:00: booting
2026-04-26T12:38:08+00:00:  [bhyve options: -c 4 -m 8GB -AHPw -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -U 246bf908-40c6-11f1-95e6-ecf4bbefc954 -u]
2026-04-26T12:38:08+00:00:  [bhyve devices: -s 0,hostbridge -s 31,lpc -s 0:4:0,ahci-hd,/usr/local/vm/hass2/disk0.img -s 0:5:0,virtio-net,tap3,mac=58:9c:fc:0a:ed:7c]
2026-04-26T12:38:08+00:00:  [bhyve console: -l com1,/dev/nmdm-hass2.1A]
2026-04-26T12:38:08+00:00: starting bhyve (run 1)
2026-04-26T12:38:08+00:00: bhyve exited with status 4
2026-04-26T12:38:08+00:00: destroying network device tap3
2026-04-26T12:38:10+00:00: stopped

However, after looking deeper, I found:

[12:47 r730-01 dvl /usr/local/vm] % less ./hass2/bhyve.log     
Unable to initialize backend '/dev/nmdm-hass2.1A' for LPC device com1
Device emulation initialization error: No such file or directory

 

[bakul]

Add nmdm_load="YES" to /boot/loader.conf or add nmdm to kld_list in /etc/rc.conf.

 

[dvl@]

Add nmdm_load="YES" to /boot/loader.conf or add nmdm to kld_list in /etc/rc.conf.

DING! DING! DING! DING!

[14:27 r730-01 dvl /usr/local/vm/hass2] % sudo kldload nmdm

[14:27 r730-01 dvl /usr/local/vm/hass2] % sudo vm start hass2
Starting hass2
  * found guest in /usr/local/vm/hass2
  * booting...

[14:28 r730-01 dvl /usr/local/vm/hass2] % sudo vm list
NAME   DATASTORE  LOADER  CPU  MEMORY  VNC  AUTO     STATE
hass   default    uefi    4    8GB     -    Yes [1]  Running (14435)
hass2  default    uefi    4    8GB     -    No       Running (12827)

And from the logs:

[14:31 r730-01 dvl /usr/local/vm/hass2] % cat bhyve.log
Unhandled ps2 mouse command 0xe1

and from vm-bhyve.log

2026-04-26T14:27:57+00:00: initialising
2026-04-26T14:27:57+00:00:  [loader: uefi]
2026-04-26T14:27:57+00:00:  [cpu: 4]
2026-04-26T14:27:57+00:00:  [memory: 8GB]
2026-04-26T14:27:57+00:00:  [hostbridge: standard]
2026-04-26T14:27:57+00:00:  [com ports: com1]
2026-04-26T14:27:57+00:00:  [uuid: 246bf908-40c6-11f1-95e6-ecf4bbefc954]
2026-04-26T14:27:57+00:00:  [debug mode: YES]
2026-04-26T14:27:57+00:00:  [primary disk: disk0.img]
2026-04-26T14:27:57+00:00:  [primary disk dev: file]
2026-04-26T14:27:57+00:00: initialising network device tap3
2026-04-26T14:27:57+00:00: adding tap3 -> bridge0 (public addm)
2026-04-26T14:27:57+00:00: bring up tap3 -> bridge0 (public addm)
2026-04-26T14:27:57+00:00: booting
2026-04-26T14:27:57+00:00:  [bhyve options: -c 4 -m 8GB -AHPw -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -U 246bf908-40c6-11f1-95e6-ecf4bbefc954 -u]
2026-04-26T14:27:57+00:00:  [bhyve devices: -s 0,hostbridge -s 31,lpc -s 0:4:0,ahci-hd,/usr/local/vm/hass2/disk0.img -s 0:5:0,virtio-net,tap3,mac=58:9c:fc:0a:ed:7c]
2026-04-26T14:27:57+00:00:  [bhyve console: -l com1,/dev/nmdm-hass2.1A]
2026-04-26T14:27:57+00:00: starting bhyve (run 1)

next, I'll try the original VM

 

[dvl@]

Add nmdm_load="YES" to /boot/loader.conf or add nmdm to kld_list in /etc/rc.conf.

This also fixes the original problem. Thank you.

I would like to know why this now required.

I just checked an old snapshot. I'm still using the same config (apart from a newline difference).

[14:48 r730-01 dvl /usr/local/vm/hass/.zfs/snapshot/autosnap_2026-01-01_00:00:18_monthly] % diff -ruN hass.conf /usr/local/vm/hass/hass.conf
--- hass.conf    2023-09-07 18:14:02.068177000 +0000
+++ /usr/local/vm/hass/hass.conf    2026-04-25 11:33:12.873181000 +0000
@@ -11,4 +11,4 @@
 grub_run_dir="/boot/grub"
 uuid="9aae377a-6c06-11ed-a655-002590fa0f10"
 network0_mac="58:9c:fc:08:5d:13"
-bhyve_options="-A"
\ No newline at end of file
+bhyve_options="-A"

 

[bakul]

I would like to know why this now required.

I don't think nmdm was ever a part of the GENERIC kernel but I haven't checked. But if you're asking why whatever you used before nmdm doesn't work anymore, I've no idea!