Solved After upgrade from 12.4 p9 to 13.2 RELEASE - VM cannot start

D

dbtrade

Hi, running Home Assistant in a VM for quite awhile now, but after the upgrade, the VM will not start. The logs show an exit code I cannot find

Code: 
Feb 03 17:42:14: initialising
Feb 03 17:42:14:  [loader: uefi]
Feb 03 17:42:14:  [cpu: 4]
Feb 03 17:42:14:  [memory: 4GB]
Feb 03 17:42:14:  [hostbridge: standard]
Feb 03 17:42:14:  [com ports: com1]
Feb 03 17:42:14:  [uuid: 96476-REDACT]
Feb 03 17:42:14:  [debug mode: no]
Feb 03 17:42:14:  [primary disk: disk0.img]
Feb 03 17:42:14:  [primary disk dev: file]
Feb 03 17:42:14: initialising network device tap0
Feb 03 17:42:14: adding tap0 -> vm-public (public addm)
Feb 03 17:42:14: bring up tap0 -> vm-public (public addm)
Feb 03 17:42:14: booting
Feb 03 17:42:14:  [bhyve options: -c 4 -m 4GB -Hwl bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -U 96476cda-REDACT -u]
Feb 03 17:42:14:  [bhyve devices: -s 0,hostbridge -s 31,lpc -s 4:0,ahci-hd,/rtank/vm/hass/disk0.img -s 5:0,virtio-net,tap0,mac=58:REDACT -s 6:0,fbuf,tcp=XXX.XXX.XXX,XX:XXXX,w=800,h=600 -s 7:0,xhci,tablet]
Feb 03 17:42:14:  [bhyve console: -l com1,/dev/nmdm-hass.1A]
Feb 03 17:42:14: starting bhyve (run 1)
Feb 03 17:42:14: bhyve exited with status 71
Feb 03 17:42:14: destroying network device tap0
Feb 03 17:42:14: stopped
My config which is unchanged:
Code: 
loader="uefi"
graphics="yes"
xhci_mouse="yes"
#graphics_listen="XXX.XXX.XXX.XXX"
graphics_port="5900"
graphics_wait="no"
graphics_res="800x600"
cpu="4"
memory="4GB"
network0_type="virtio-net"
network0_switch="public"
disk0_type="ahci-hd"
disk0_name="disk0.img"
grub_run_partition="1"
grub_run_dir="/boot/grub"
uuid="96476cda-redacted"
network0_mac="58:redacted"
Any troubleshooting tips? The only other issue I have seen is the system was showing the incorrect time by about 30 minutes. However, networking is working fine.

I tried installing from scratch the same system, but the same problem with starting.
 
Last edited by a moderator:
shurik said:
May be this PR 273560 is related to you
Click to expand...
I do believe this may be true, but I have never done a "firmware" downgrade or version downgrade and have not found instructions on how to do so or which ones would be most applicable to my setup. More research I guess. I am still searching. My first question is around whether I follow a pkg downgrade or a ports?

Or any other debugging tips would be great.

I did try adding:

bhyve_options="-A"
Click to expand...
But it made no difference and in any case, I do not get far enough to see that it is responding with "ACPI not found".
 
Emrion said:
On what OS and version do you run this software?
Click to expand...
I am a little confused about what you are asking.
It's a bhyve instance running in FreeBSD 13.2 RELEASE which previously worked in FreeBSD 12.4
It is/was running Home assistant and am using the debian template.
I have tried creating a new VM for it from scratch, but I get the same result. The VM immediately returns with an exit 71 code.
 
Am now concerned that the processor I am using is no longer supported in the 13.2 version.
AMD Athlon II X2 240 - Athlon II X2 Regor Dual-Core 2.8 GHz Socket AM3 65W Processor
 
With regards to pursuing the idea that the edk2-bhyve doesn't work with my current hardware, I found this:
Code: 
pkg search edk2-bhyve
edk2-bhyve-g202308_4           EDK2 Firmware for bhyve
uefi-edk2-bhyve-csm-0.2_4,1    UEFI EDK2 firmware for bhyve with CSM (16-bit BIOS)
and in my /var/cache/pkg I have these:
Code: 
edk2-bhyve-g202202_1.pkg
edk2-bhyve-g202202_1~2c38f95ab7.pkg
edk2-bhyve-g202308_4.pkg
How much trouble can I get in trying to replace the
Code: 
edk2-bhyve-g202308_4 with edk2-bhyve-g202202_1.pkg
And how can I accomplish that?

Do I first remove the current package and then do an offline pkg add from the /var/cache/pkg directory
 
I used pkg remove edk2-bhyve-g202308_4 and then pkg install edk2-bhyve-g202202_1.pkg

I ran pkg check -d and pkg check -B and all was good.

I restarted and tried to start the VM's, got the same error,
So I removed the downgraded package and reinstalled the original.

Now when I run pkg check -d I get:
Code: 
Checking all packages: 100%
llvm12 is missing a required shared library: libthr.so.3
llvm12 is missing a required shared library: libncursesw.so.9
llvm12 is missing a required shared library: libm.so.5
llvm12 is missing a required shared library: libcxxrt.so.1
llvm12 is missing a required shared library: libc.so.7
llvm13 is missing a required shared library: libthr.so.3
llvm13 is missing a required shared library: libncursesw.so.9
llvm13 is missing a required shared library: libm.so.5
llvm13 is missing a required shared library: libcxxrt.so.1
llvm13 is missing a required shared library: libc.so.7
llvm15 is missing a required shared library: libz.so.6
llvm15 is missing a required shared library: libthr.so.3
llvm15 is missing a required shared library: libncursesw.so.9
llvm15 is missing a required shared library: libm.so.5
llvm15 is missing a required shared library: libkvm.so.7
llvm15 is missing a required shared library: libcxxrt.so.1
llvm15 is missing a required shared library: libc.so.7
What the heck? Fixed the missing libraries
 
Ok. So the OS is Debian. You should leave the "-A" option in all cases. I know that debian is a very sensitive OS in regard to bhyve. I experienced several types of hangs, but there, it's seems it's bhyve that does not start. Can't find what error code 71 means.
 
Same for me, works well on 13.2. But I'm not using generic image - it lacks ACPI support and vm cannot be stopped by
vm stop command. I'm using ova vmdk image converted with qemu-img.
 
shurik said:
Same for me, works well on 13.2. But I'm not using generic image - it lacks ACPI support and vm cannot be stopped by
vm stop command. I'm using ova vmdk image converted with qemu-img.
Click to expand...
Lol. My own scripts just force exit the VMs without asking nothing. It's not easy to gracefully stop that HAOS by the way. No button in the web interface for that. You have to call a "service". Maybe it doesn't work in my case, didn't try up to now.

Edit: I can shut it down via the web interface. Developper tool / Services / Home Assistant Supervisor: Poweroff the host system.
 
It's some other problem as I have tried to create another VM container for Home assistant, and it reports the same exit code 71, which the various AI can't agree on what that code means. I did find a suggestion to:
'you can manually download a copy of the UEFI to $vm_dir/.config/BHYVE_UEFI.fd and configure a guest to use it by setting loader="uefi-custom"'
 
dbtrade said:
It's some other problem as I have tried to create another VM container for Home assistant, and it reports the same exit code 71, which the various AI can't agree on what that code means. I did find a suggestion to:
'you can manually download a copy of the UEFI to $vm_dir/.config/BHYVE_UEFI.fd and configure a guest to use it by setting loader="uefi-custom"'
Click to expand...
It's a good idea. It happened to me one time, a VM that crashes before to start (not the same error code). I took an untouched BHYVE_UEFI_VARS.fd file and replaced the used one and that worked.

Normally, you can find one in: /usr/local/share/edk2-bhyve/. Copy it, don't use it directly.
 
Emrion said:
It's a good idea. It happened to me one time, a VM that crashes before to start (not the same error code). I took an untouched BHYVE_UEFI_VARS.fd file and replaced the used one and that worked.

Normally, you can find one in: /usr/local/share/edk2-bhyve/. Copy it, don't use it directly.
Click to expand...
It wasn't happy with the with the "BHYVE_UEFI_VARS.fd ", complained it was not a valid loader... I used the AI suggestion of BHYVE_UEFI.fd which was also in that folder you mentioned (thank you).
This time, the loader was happy, but no profit.
Feb 18 15:45:42: [bhyve options: -c 4 -m 4GB -Hwl bootrom,/rtank/vm/.config/BHYVE_UEFI.fd -A -U 9647-redact -u]
Feb 18 15:45:42: [bhyve devices: -s 0,hostbridge -s 31,lpc -s 4:0,ahci-hd,/rtank/vm/hass/disk0.img -s 5:0,virtio-net,tap0,mac=58:9c:fc:0b:e2:54 -s 6:0,fbuf,tcp=0.0.0.0:5900,w=800,h=600 -s 7:0,xhci,tablet]
Feb 18 15:45:42: [bhyve console: -l com1,/dev/nmdm-hass.1A]
Feb 18 15:45:42: starting bhyve (run 1)
Feb 18 15:45:42: bhyve exited with status 71
Feb 18 15:45:42: destroying network device tap0
Click to expand...
Going to try a new installation. I think I can use haos_ova-11.5.qcow2.xz
My original installation back in the day used: haos_ova-6.0.qcow2
 
Ok, I confused the files. This file contains the efi vars that an efi BIOS stores not the loader code. But, it's well the file I changed to make my VM works.

Have you tried to deactivate the com console (just use vnc)?
 
For UEFI vars to work you just have to add uefi_vars="yes" to the VM's configuration. If there's no file yet vm(8) automatically copies a clean one when the VM is started. You shouldn't need to mess with the EUFI firmware, it'll pick one up from /usr/local/share/uefi-firmware/.
 
Well, this was shelved and I put the Hass on a Raspberry Pi for a while, while dealing with other things. But that is not a happy thing.

In the meantime, I upgraded on up to 14.0. I still cannot get this rolling with vm-bhyve even after removing everything including the packages and starting from scratch.
I can create a Freebsd jail instance and will attempt to make a jail from some other random Debian Linux instance.
I have not been able to determine what an error 71 is, partly because I am not sure who to ask.

Other than that, I may forego the convenience of vm-bhyve and put together a bhyve jail directly to see if that offers more clues.
Emrion said:
Out of curiosity, I downloaded an image of HA here: https://github.com/home-assistant/operating-system/releases/tag/11.5 (haos_generic-x86-64-11.5.img.xz).
Click to expand...


shurik said:
Same for me, works well on 13.2. But I'm not using generic image - it lacks ACPI support and vm cannot be stopped by
vm stop command. I'm using ova vmdk image converted with qemu-img.
Click to expand...

Have given me hope. I am casting about for clues since I don't know what an error 71 is. Who can I ask without reading through the vm-bhyve code base?

Dan Langville over on Twitter had the same issue, but what fixed it for him, did me no good.
 
Please suggest anything! I am the OP.

Running home assistant on my server is a big use case for me.

I have completely removed all the vm-bhyve libraries, destroyed, and rebuilt the zfs dataset.
I have tried every permutation I can think of.

This system has been in use since OpenSolaris days.

I can run a jail that is a FreeBSD guest and got a Linux guest to load but I didn't bother to run the Grub install on it.

I even tried to build it exactly, including the hassio version 6.0 from when I originally installed it with no luck
You can see the settings used in the original post.

If I use cpu=4 (the original setting), I get a vm_specify_topography in the bhyve.log with an exit code of 71
If I set cpu=2, I get an exit code 134. Other variations get an exit code of 4.

Would someone please point me to a "How-to" using straight-up Bhyve in case that somehow gives me more insight?
Or perhaps suggest who I can turn to? I have considered filing a bug report with vm-bhyve since it all worked fine before FreeBSD 13.

Is it possible that the CPU is no longer supported between 12.4 and 13.0 for virtualization?
I did try the bhyve_options="-A".
 
Have you got another machine that you can try on? With a different processor? If you can get it going on another machine then at least you have a plan B and know it will work, but perhaps not on the original machine.

Error code 71 seems to usually mean file not found, but can't be sure that bhyve has a different meaning for it - but everything points to file not found. Obviously the next question is "what file?" (EDIT: having fun spelunking the bhyve code but cannot say I've found anything useful!)
 
dbtrade said:
I get a vm_specify_topography in the bhyve.log with an exit code of 71
Click to expand...
Hmmm, maybe you meant this (usr.sbin/bhyve/bhyverun.c):

577 error = vm_set_topology(ctx, cpu_sockets, cpu_cores, cpu_threads, 0);
578 if (error)
579 errx(EX_OSERR, "vm_set_topology");

What's EX_OSERR:

sysexits

man.freebsd.org man.freebsd.org

EX_OSERR (71) An operating system error has been detected. This
is intended to be used for such things as ``cannot
fork'', ``cannot create pipe'', or the like. It
includes things like getuid returning a user that
does not exist in the passwd file.

lib/libvmmapi/vmmapi.c

1112 int
1113 vm_set_topology(struct vmctx *ctx,
1114 uint16_t sockets, uint16_t cores, uint16_t threads, uint16_t maxcpus)
1115 {
1116 struct vm_cpu_topology topology;
1117
1118 bzero(&topology, sizeof (struct vm_cpu_topology));
1119 topology.sockets = sockets;
1120 topology.cores = cores;
1121 topology.threads = threads;
1122 topology.maxcpus = maxcpus;
1123 return (ioctl(ctx->fd, VM_SET_TOPOLOGY, &topology));
1124 }

... So that ioctl returns an error 71 (not sure I'm helping much!)

⚙ D9930 Bhyve cpu topology control

reviews.freebsd.org reviews.freebsd.org
 
FreeBSD 13.2 had some changes to bhyve - maybe (?) something from that.

www.freebsd.org

FreeBSD 13.2-RELEASE Release Notes

FreeBSD is an operating system used to power modern servers, desktops, and embedded platforms.
www.freebsd.org www.freebsd.org

Look at

src - FreeBSD source tree

cgit.freebsd.org cgit.freebsd.org

The default is now the number of physical CPUs in the system rather than 16.

So maybe your CPU is very limited (looks like it is 15 years old, dual core, two threads?)

The release notes mention a new loader tunable hw.vmm.maxcpu.

Maybe prod that to see if any change? So if the old default was 16, try bumping this new tunable to 16 - that might match the old behaviour?

(I've never used bhyve just looking at the source code).
 
After exploring all the helpful suggestions, it turns out that the CPU became unsupported for VM use after FreeBSD 12 and it's variants. Dropping all the same settings into a new box with a more modern CPU solved all the issues.
 
You must log in or register to reply here.
Back
Top