Solved VMs don't start or crash under FreeBSD 15.0

[Emrion]

Hi,

I tried to upgrade my FreeBSD based router from 14.3-RELEASE to 15.0-RELEASE.

This machine has two bhyve VM: one is a pfsense with an ix interface "passthrued". It brings internet to the machine (and to my local network). The other VM is a debian 12 with a QCA6174A "passthrued" to allow wifi devices to connect to internet & local network. You have some details here: https://forums.freebsd.org/threads/a-freebsd-box-for-home.70451/#post-695571

The state of the upgrade is: new kernel installed and active, userland installed, old libraries not removed. The problem is that no VM is functioning. bhyve reports nothing. The VM are launched, but they are probably stuck or crashed during the startup (a VNC connection gives a black screen). So, I can't access anymore to internet.

I didn't find any log to help me.

The worst thing is that I have a test machine with a pfSense bhyve VM and a similar configuration (not concerning the hardware) that works well under 15.0-RELEASE. I'm unable to reproduce the problem on this test machine.

For the moment, my router lives on a 14.3-RELEASE BE.

What can I check to pinpoint the source of this problem?

PS1: I saw that often ifconfig doesn't report the bridge members. But I get the same on the functioning test machine.

PS2: sometimes, when I try to relaunch a VM (after destroyed it), bhyve report an error (4: exited due to an error).

 

[Emrion]

I eventually found the culprit. It was an old version of the UEFI firmware BHYVE_UEFI.fd that I used to start the VMs. It works well on 14.3 but hangs or crashes with 15.0.

The funny thing is: I kept this old version to avoid possible crashes with new firmwares as I'm on latest repos (normally, you should use /usr/local/share/uefi-firmware/BHYVE_UEFI.fd to use the current installed firmware). This time, I lose.

The rest of the upgrade went peacefully. Well, you have to deal with the change of the repos names (if you use the latest ones), non-upgrade of the kmods: delete them and install them anew; but, I already knew that since longtime, so it was ok.

I'm currently upgrading my thin jails. I'm thinking, as I read here and there, it'd better to pull a new base. Something to ponder for the future of my kjail scripts, but there is also pkgbase that entered in the arena. Many things will change soon.

Next thing to do: move the ip addresses on the bridge and not on the interfaces, as advised*.

Edit: *done 17/01. Put the ix0 and five jails ip addresses on the bridge. So far, so good.

The final words are: don't forget to upgrade your boot loaders.