bhyve Debian 12 won't boot in UEFI mode in Bhyve

V

Vurefozu

I am creating this topic as a continuation of the bug issue - https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=278984

Hi.
If you install Debian 12 on bhyve from an iso image in UEFI mode, you will notice that the grub boot loader will not be found.

To fix this you need to do the following:
mkdir /boot/efi/EFI/BOOT/
# copy file - workaround for bhyve grub package #
# Pay attention to destination file bootx64.efi #
cp /boot/efi/EFI/debian/grubx64.efi /boot/efi/EFI/BOOT/bootx64.efi

Taken here: https://www.cyberciti.biz/faq/how-to-install-linux-vm-on-freebsd-using-bhyve-and-zfs/

I suggest fixing bhyve so that it tries to load not only /boot/efi/EFI/BOOT/bootx64.efi, but also /boot/efi/EFI/debian/grubx64.efi.
Click to expand...
 
https://ibb.co/c8cx9wZ

I'm trying to boot a Debian img file that I've created with qemu and that it boots if I use qemu. I imagine that I should create a new Debian vm from scratch if I want to boot it with bhyve,right ?
Click to expand...

https://wiki.freebsd.org/bhyve

Q: Can I use disk images from QEMU, VMware, or other virtualization platforms on bhyve?​


A: bhyve does not currently support reading foreign disk image formats directly, so one will have to convert existing VMDK, QCOW, QCOW2, and VDI disk images to a raw image using qemu-img. This can be accomplished by doing the following on a FreeBSD system:

$ pkg install qemu-tools
$ qemu-img convert -f vmdk -O raw vmware_image.vmdk bhyve_raw_image.raw
Click to expand...
 
My notes, not sure where from but: Debian UEFI FIX

1. Enter "exit" at the prompt get out of the UEFI Shell (you will come to BHYVE panel)
2. Choose Boot Maintenance Manager
3. (In the Boot Maintenance Manager ) Choose Boot From File
4. (In File Explorer) NO VOLUME LABEL, [PciRoot(0x0)/ Pci()] press Enter
5. (Still in the File Explorer) Choose <EFI>
6. (Still in the File Explorer) Choose debian
7. (Still in the File Explorer) Choose grubx64.efi
8. Then you will get into the system

How can you fix it:
1) Get access inside your Debian Guest VM.
2) as root: cd /boot/efi/
3) mkdir -p EFI/BOOT
4) cp EFI/debian/grubx64.efi EFI/BOOT/bootx64.efi
5) reboot
Click to expand...
 
Last edited:
There is no problem whatsoever with bhyve uefi.
It's not a bug but a lack of knowledge.

If you use pure bhyve, you must specify a var file in order bhyve recalls the var that the system has created/modified during installation or else.
bhyve(8)
Boot ROM device backends:

romfile[,varfile]
Map romfile in the guest address space reserved
for boot firmware. If varfile is provided,
that file is also mapped in the boot firmware
guest address space, and any modifications the
guest makes will be saved to that file.
Click to expand...

If you use vm-bhyve, you just have to set set uefi_vars at true. See vm(8), paragraph GUEST CONFIGURATION FORMAT.
 
Last edited:
Emrion said:
There is no problem whatsoever with bhyve uefi.
It's not a bug but a lack of knowledge.

If you use pure bhyve, you must specify a var file in order bhyve recalls the var that the system has created/modified during installation or else.
bhyve(8)


If you use vm-bhyve, you just have to set set uefi_vars at true. See vm(8), paragraph GUEST CONFIGURATION FORMAT.
Click to expand...
Thanks for clarifying that, for those that run into this issue in the future, following steps make the UEFI entries work permanently (thanks to Emrion's suggestion)

# Copy
cp /usr/local/share/uefi-firmware/BHYVE_UEFI_VARS.fd /root/debian12_VARS.fd

# Add copied file to bootrom path
bhyve -c 1 -m 2G -w -H -D \
-s 0:0,hostbridge \
-s 1:0,lpc \
-s 2:0,virtio-net,tap1 \
-s 3:0,virtio-blk,/dev/zvol/zroot/debian12 \
-s 5:0,fbuf,tcp=0.0.0.0:5901,w=1024,h=768,wait \
-s 6:0,xhci,tablet \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd,/root/debian12_VARS.fd \
debian12
 
Still doesn't work, even with UEFI vars file. I get this error:
Code: 
BdsDxe: failed to load Boot0006 "ubuntu” from HD(1,GPT,1FA2157F-2553-4DDE-ADBB-6DBBB12689A0,0x800,0x10000)/\EFI\ubuntu\shimx64.efi: Not Found
However, that file exists:
Code: 
# mount -t msdos /dev/zvol/zroot/ubuntu20p1 /mnt/tmp/
# ll /mnt/tmp/EFI/ubuntu/
total 4314
-rwxr-xr-x  1 root wheel     108 Apr  9 00:02 BOOTX64.CSV
-rwxr-xr-x  1 root wheel     126 Apr  9 00:02 grub.cfg
-rwxr-xr-x  1 root wheel 2594696 Apr  9 00:02 grubx64.efi
-rwxr-xr-x  1 root wheel  860824 Apr  9 00:02 mmx64.efi
-rwxr-xr-x  1 root wheel  960472 Apr  9 00:02 shimx64.efi
 
"/\EFI\ubuntu\shimx64.efi: Not Found"

Problem seems to be in that path. There is a forward slash "/", then a DOS / windows style path?
 
blackhole said:
"/\EFI\ubuntu\shimx64.efi: Not Found"
Problem seems to be in that path. There is a forward slash "/", then a DOS / windows style path?
Click to expand...
Well, if that's an issue, it is out of my control ― the installer added that Boot0006 entry to UEFI.
In the same box I have 4 other VMs, every time I had to copy an .efi file to /EFI/BOOT and it worked. Now with the "correct way" it doesn't.
 
I do have Debian-12 VM working. what is your template? If you wish try below template.

loader="uefi"
uefi_vars="yes"
graphics="yes"
xhci_mouse="yes"
graphics_res="1280x720"
graphics_listen="0.0.0.0"
graphics_port="5901"
cpu=2
memory=4G
network0_type="virtio-net"
network0_switch="public"
disk0_type="ahci-hd"
disk0_name="debian-disk0.img"
 
aragats said:
I do have several Debian/Ubuntu VMs working.
I use a script similar to one in chief's post above for all my VM's.
The problem is not how I launch the VM: Ubuntu installer works as expected and installs the OS, but then UEFI cannot start it.
Click to expand...
As this works for several people here, you must do something wrong. You don't give too much details on the way you use bhyve.

I have several Debian 11 VMs but not 12. Will try this evening if I have time, but there is no reason it doesn't work.
 
Just installed with the help of my framework https://github.com/Emrion/kvm
It works.

Command line for installation:
bhyve -AHP -c 1 -m 2g -s 0,hostbridge -s 1,nvme,/data/kvm/vm/deb12/disk -s 2,ahci-cd,/data/kvm/vm/deb12/debian-12.10.0-amd64-DVD-1.iso -s 31,lpc -s 29,xhci,tablet -s 30,fbuf,tcp=0.0.0.0:5900,wait -s 3,virtio-net,tapDEB12 -l bootrom,/data/kvm/BHYVE_UEFI.fd,/data/kvm/vm/deb12/UEFI_VARS.fd deb12

Command line to start it after installation:
bhyve -AHP -c 1 -m 2g -s 0,hostbridge -s 1,nvme,/data/kvm/vm/deb12/disk -s 31,lpc -s 29,xhci,tablet -s 30,fbuf,tcp=0.0.0.0:5900,wait -s 3,virtio-net,tapDEB12 -l bootrom,/data/kvm/BHYVE_UEFI.fd,/data/kvm/vm/deb12/UEFI_VARS.fd deb12
 
Thanks, Emrion!
In my case it's Ubuntu 20.04 LTS.
I ended up with dd'ing the entire disk image from a good Ubuntu 16 VM and rsync'ing Ubuntu 20's partition.
Since it's a production box I don't have much time to figure out what's going on. Will spend some time later.
 
Installed Ubuntu 20.04.6 LTS. It works...

Code: 
$ sudo efibootmgr -v
BootCurrent: 0006
Timeout: 0 seconds
BootOrder: 0006,0000,0002,0003,0004,0005
Boot0000* UiApp FvVol(7cb8bdc9-f8eb-4f34-aaea-3ee4af6516a1)/FvFile(462caa21-7614-4503-836e-8ab6f4662331)
Boot0002* UEFI bhyve-NVMe NVME-1-0      PciRoot(0x0)/Pci(0x1,0x0)/NVMe(0x1,58-9C-FC-20-CA-0F-00-01)N.....YM....R,Y.
Boot0003* UEFI PXEv4 (MAC:00A098616EA9) PciRoot(0x0)/Pci(0x3,0x0)/MAC(00a098616ea9,1)/IPv4(0.0.0.00.0.0.0,0,0)N.....YM....R,Y.
Boot0004* UEFI PXEv6 (MAC:00A098616EA9) PciRoot(0x0)/Pci(0x3,0x0)/MAC(00a098616ea9,1)/IPv6([::]:<->[::]:,0,0)N.....YM....R,Y.
Boot0005* EFI Internal Shell    FvVol(7cb8bdc9-f8eb-4f34-aaea-3ee4af6516a1)/FvFile(7c04a583-9e3e-4f1c-ad65-e05268d0b4d1)
Boot0006* ubuntu        HD(1,GPT,9772177c-3ebb-47ce-b0d8-5d63d28d8c8e,0x800,0x10d000)/File(\EFI\ubuntu\shimx64.efi)

Code: 
$ lsblk
NAME                      MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
loop0                       7:0    0 63,3M  1 loop /snap/core20/1828
loop1                       7:1    0 91,9M  1 loop /snap/lxd/24061
loop2                       7:2    0 49,9M  1 loop /snap/snapd/18357
nvme0n1                   259:0    0   10G  0 disk
  nvme0n1p1               259:1    0  538M  0 part /boot/efi
  nvme0n1p2               259:2    0  1,8G  0 part /boot
  nvme0n1p3               259:3    0  7,7G  0 part
    ubuntu--vg-ubuntu--lv 253:0    0  7,7G  0 lvm  /
 
You must log in or register to reply here.
Back
Top