bhyve Debian 13.0.0 Guest via Bhyve: Unable to get GRUB working

F

felsweg

Hi,

I wanted to give FreeBSD a shot, but also to have Debian 13.0.0 as a virtual machine using bhyve, especially the wrapper support scripts vm-bhyve to simplify the setup. Following the steps of the tutorial of `vm-bhyve` (and the other resources linked below), I ended up with a Grub based installation of Debian. The installation completes, but drops to the grub shell indicating that the linux kernel image cannot be found.

I am well aware, that I am not the first having this problem, as some other posts in this forum clearly indicate.

After trying the following suggested solution to configure GRUB to find the correct kernel image and load it, I am stuck at grub unable to find any usable filesystem.
Code: 
(hd0) (hd0,msdos5) (hd0,msdos1) (host)

Trying to read any filesystem always results in "unknown filesystem", such that set root=/... is not possible.

My current config includes

System: FreeBSD: 14 (latest stable)
Guest OS: Debian 13.0.0
packages installed: bhyve, vm-bhyve, grub2-bhyve, bhyve-firmware


The configuration i am using for the Debian VM is as follows:

Code: 
cpu=4
memory=8G
network0_type="virtio-blk"
network0_switch="public"
disk0_type="virtio-blk"
disk0_name="disk0.img"

# I assume this refers to (hd0,1)
grub_run_partition="1"
grub_run_dir="/boot/grub"

# I think thats not necessary in textmode
# as bhyve would default to that
graphics="no"

uuid="..."
network0_mac="..."

I have a strong suspicion, that something is off with the disk. Booting again with the installer iso and executing a shell, I can clearly see the boot directory. While i have not tried to modify the boot environment from the rescue shell itself, I wonder if there is straight way to install Debian/Linux and make booting it via Grub or better UEFI.

UEFI

Put here just for reference, in case UEFI is recommended for installation.

Setting the configuration to use `loader="uefi"` results in displaying the selection screen of the installer ( "install" "graphical install" ...), but after that the screen blanks.

I am working in text mode only and i am not using any x11 or other graphical environment.

Resources

Following resources were of no help either, unfortunately:
- https://github.com/churchers/vm-bhyve/blob/master/sample-templates/config.sample
- https://docs.freebsd.org/en/books/handbook/virtualization/#virtualization-host-bhyve
- https://github.com/churchers/vm-bhyve/wiki/Configuring-Grub-Guests
- https://blahg.josefsipek.net/?p=595
- https://forums.freebsd.org/threads/how-to-boot-debian-in-efi-mode-using-bhyve.81520/
- https://npulse.net/en/blog/125-bhyve-uefi-drops-into-efi-shell-linux-wont-boot-easy-workaround
- https://github.com/churchers/vm-bhyve
- https://www.duskopijetlovic.com/freebsd/bhyve/virtualization/2022/02/06/freebsd-vm-bhyve.html
- https://www.cyberciti.biz/faq/how-to-install-linux-vm-on-freebsd-using-bhyve-and-zfs/
- https://www.davidschlachter.com/misc/freebsd-bhyve-uefi-shell
- https://gist.github.com/ianthetechie/4e1e0b4cb48981629e5ad0e2cdd4e5ec
- https://i-bsd.com/freebsd-vm-bhyve-linux/
 
Enable uefi_vars="yes", so the UEFI boot settings are actually getting saved.

Code: 
loader="uefi"
graphics="yes"
graphics_res="1024x768"
eufi_vars="yes"

Then use a VNCviewer to access the console and install debian.

felsweg said:
I am working in text mode only and i am not using any x11 or other graphical environment.
Click to expand...
If I recall correctly the serial console is disabled by default. So you'll need to find a way to enable this first. Or else the only way to install Debian is through a graphics console with VNCViewer (which you could run on a Windows or MacOS computer, you don't have to run this locally on the VM host server).

I have two with VNC enabled:
Code: 
root@hosaka:~ # vm list -r
NAME            DATASTORE  LOADER     CPU  MEMORY  VNC           AUTO      STATE
case            default    bhyveload  4    4096M   -             Yes [4]   Running (2873)
jenkins         default    bhyveload  4    4096M   -             Yes [7]   Running (3691)
kdc             default    uefi       2    2048M   0.0.0.0:5900  Yes [1]   Running (1886)
lady3jane       default    bhyveload  4    16384M  -             Yes [3]   Running (2495)
errol           stor10k    bhyveload  2    4096M   -             Yes [9]   Running (4385)
fbsd-test       stor10k    bhyveload  2    4096M   -             Yes [13]  Running (5218)
fbsd-test-old   stor10k    bhyveload  4    4096M   -             No        Running (3921)
gl-runner-1     stor10k    bhyveload  4    4096M   -             Yes [12]  Running (5196)
kibana          stor10k    bhyveload  4    8192M   -             Yes [6]   Running (3671)
riviera         stor10k    bhyveload  2    4096M   -             Yes [11]  Running (63543)
sdgame01        stor10k    uefi       4    8192M   0.0.0.0:5901  Yes [8]   Running (3904)
tessierashpool  stor10k    bhyveload  2    4096M   -             Yes [5]   Running (3403)
wintermute      stor10k    bhyveload  4    4096M   -             Yes [10]  Running (4696)
And I can access these from any other computer on my network, don't have to run vncviewer locally (there is no X or wayland on the VM host itself).
 
SirDice

Installing Debian with VNC works like a charm. I just tested it with the following configuration:


Code: 
loader="uefi"
uefi_vars="yes"
cpu=4
memory=8G
network0_type="virtio-net"
network0_switch="public"

# changed by vm-bhyve
disk0_type="ahci-hd"
disk0_name="disk0.img"

# I guess the grub config can be discarded?
graphics="yes"
graphics_res="1024x768"

# I tried with this setting to enable the console via stdio
# but I assume this setting is plain wrong
bhyve_options="-l com1,stdio"

​

So, that implies I need to figure how to enable the serial console. The kernel module for the nullmode nmdm_load="yes" is enable and loaded.

Ultimatively, I want to go with cloud-init to have the installation process non-interactive. Also have the vm only reachable by ssh. But thanks to your help I could make the next step.

edit: I removed the grub_* configuration settings, as they are not needed when using uefi boot mode.
 
Have no experience with cloud-init, only installed Debian a couple of times by hand. And yes, those grub_* settings can be removed, they're ignored if you EUFI boot the VM.
 
Hi, I am glad I came across this post, because I struggled with the same thing a couple days ago, and while I am still struggling with the general problem, I know the solution to this specific one with what you are going through. I understand you may have already figured this out since then but I thought I'd share anyways.

So, from my understanding, modern ext4 has some newer features that were unavailable on previous versions of ext4. The features causes grub2-bhyve to get upset, not knowing what on Earth it is looking at. So a solution I found was to manually split up the virtual disk Debian will be on into a few partitions: '/boot', '/', and 'swap'.

I would give '/boot' a 1GB partition, 'swap' a 2GB partition, and '/' whatever is left over. The trick is to specify the use of ext3 on the '/boot' partition. I used ext4 on '/'. After that, it worked!

I am still struggling a bit though. I am trying to automate the loading of the kernel so when the server starts up, the VM can automatically start up without me. If you are struggling with that as well, I would appreciate any updates if you end up getting that figured out!
 
javamartini I haven't tested your proposal, but thanks for the input. It sounds reasonable. I am currently circumventing the problem by installing linux debian "natively" by using the cloud images in addition to cloud-init. This works good enough and doesn't require any VNC access, since the provided images are already bootable, while to be converted with qemu-img before that by vm-bhyve.

However, the cloud images come naturally with a reduced set of hardware support, so you would need to switch the kernel, if more features or additional firmware is required. I will post my setup later. But maybe it helps you setting up your virtual machines.
 
felsweg I'm glad to see you have found a solution!

I still am sort of looking for a solution for automating the booting of the VM if it wasn't set up using UEFI. I would have found it by now but the college semester is back on.

For those in the future, I probably will post it here so it saves others from the trouble.
 
javamartini said:
Hi, I am glad I came across this post, because I struggled with the same thing a couple days ago, and while I am still struggling with the general problem, I know the solution to this specific one with what you are going through. I understand you may have already figured this out since then but I thought I'd share anyways.

So, from my understanding, modern ext4 has some newer features that were unavailable on previous versions of ext4. The features causes grub2-bhyve to get upset, not knowing what on Earth it is looking at. So a solution I found was to manually split up the virtual disk Debian will be on into a few partitions: '/boot', '/', and 'swap'.

I would give '/boot' a 1GB partition, 'swap' a 2GB partition, and '/' whatever is left over. The trick is to specify the use of ext3 on the '/boot' partition. I used ext4 on '/'. After that, it worked!

I am still struggling a bit though. I am trying to automate the loading of the kernel so when the server starts up, the VM can automatically start up without me. If you are struggling with that as well, I would appreciate any updates if you end up getting that figured out!
Click to expand...
This didn't work for me. I had to rebuild sysutils/grub2-bhyve with this patch.


Code: 
--- grub-core/fs/ext2.c.orig    2025-09-30 01:30:51 UTC
+++ grub-core/fs/ext2.c
@@ -100,7 +100,10 @@ GRUB_MOD_LICENSE ("GPLv3+");
 #define EXT2_FEATURE_INCOMPAT_META_BG        0x0010
 #define EXT4_FEATURE_INCOMPAT_EXTENTS        0x0040 /* Extents used */
 #define EXT4_FEATURE_INCOMPAT_64BIT        0x0080
+#define EXT4_FEATURE_INCOMPAT_MMP               0x0100
 #define EXT4_FEATURE_INCOMPAT_FLEX_BG        0x0200
+#define EXT4_FEATURE_INCOMPAT_CSUM_SEED         0x2000
+#define EXT4_FEATURE_INCOMPAT_LARGEDIR          0x4000 /* >2GB or 3 level htree */
 
 /* The set of back-incompatible features this driver DOES support. Add (OR)
  * flags here as the related features are implemented into the driver.  */
@@ -114,8 +117,26 @@ GRUB_MOD_LICENSE ("GPLv3+");
  *                 journal because they will ignore the journal, but the next
  *                 ext3 driver to mount the volume will find the journal and
  *                 replay it, potentially corrupting the metadata written by
- *                 the ext2 drivers. Safe to ignore for this RO driver.  */
-#define EXT2_DRIVER_IGNORED_INCOMPAT ( EXT3_FEATURE_INCOMPAT_RECOVER )
+ *                 the ext2 drivers. Safe to ignore for this RO driver.
+ * mmp:            Not really back-incompatible - was added as such to
+ *                 avoid multiple read-write mounts. Safe to ignore for this
+ *                 RO driver.
+ * checksum seed:  Not really back-incompatible - was added to allow tools
+ *                 such as tune2fs to change the UUID on a mounted metadata
+ *                 checksummed filesystem. Safe to ignore for now since the
+ *                 driver doesn't support checksum verification. However, it
+ *                 has to be removed from this list if the support is added later.
+ * large_dir:      Not back-incompatible given that the GRUB ext2 driver does
+ *                 not implement EXT2_FEATURE_COMPAT_DIR_INDEX. If the GRUB
+ *                 eventually supports the htree feature (aka dir_index)
+ *                 it should support 3 level htrees and then move
+ *                 EXT4_FEATURE_INCOMPAT_LARGEDIR to
+ *                 EXT2_DRIVER_SUPPORTED_INCOMPAT.
+ */
+#define EXT2_DRIVER_IGNORED_INCOMPAT ( EXT3_FEATURE_INCOMPAT_RECOVER \
+                                      | EXT4_FEATURE_INCOMPAT_MMP \
+                                      | EXT4_FEATURE_INCOMPAT_CSUM_SEED \
+                                      | EXT4_FEATURE_INCOMPAT_LARGEDIR)
 
 
 #define EXT3_JOURNAL_MAGIC_NUMBER    0xc03b3998U
 
You must log in or register to reply here.
Back
Top