bhyve Arch Linux VM emulated with BHYVE : considerations.

ziomario

ziomario

Hello to everyone. I've tried to install Arch Linux on bhyve,but it didn't work. I mean,the image created is not able to boot when I use bhyve. I repeated the steps two times. The first time I've issued the commands using FreeBSD and creating the image with the tools and the tecniques provided by FreeBSD and Bhyve. The second time I've switched in Linux and I've repeated the steps creating a new image with qemu / kvm / virt-manager and then I tried to boot it with Bhyve. Below you can see the commands that I have issued. Take in consideration that those commands worked in Linux and with qemu-kvm. I mean,the image produced by these commands can boot in Linux with qemu-kvm-virt-manager but NOT in FreeBSD with bhyve. So,I suspect that there is some bug in the booting procedure of Bhyve. I've highlighted the commands that I have used for installing grub and the efibootmgr in Linux. I suspect (but Im not sure) that the Arch Linux is affected by the same problem that is affected Debian and "Jeff Sipek" talked about this problem in his website :

Josef “Jeff” Sipek » Installing Debian under FreeBSD's bhyve

blahg.josefsipek.net blahg.josefsipek.net

when he says :

This is not a Debian bug, but rather bhyve’s EFI support being incomplete. The easiest way around this is to copy the Debian binary into the standard location immediately after installation.

The difference is that he found an easy solution for the incomplete EFI support that prevents Debian from booting,but it's not easy to apply the same method for Arch. Or,at least,I tried,but it didn't work. So,I have a complete Arch installation stored inside an img file that can boot under Linux with qemu-kvm but not under FreeBSD with Bhyve. Someone can give me some suggestions to fix this problem ?

Code: 
loadkeys it
ip link --> enp1s0
ip addr --> 192.168.122.86/24 o dhcpcd
fdisk -l --> /dev/vda1 ; /dev/vda2 ; /dev/vda3
cfdisk /dev/vda
GPT
new / 512M / EFI system
new / 4G / Linux Swap
new / 195G / Linux System
mkfs.vfat -F32 /dev/vda1
mkfs.ext4 /dev/vda3
mkswap /dev/vda2
mount /dev/vda3 /mnt
mkdir -p /mnt/boot/efi
mount /dev/vda1 /mnt/boot/efi/
swapon /dev/vda2
reflector --verbose --country Italy --sort rate --save /etc/pacman.d/mirrorlist
pacstrap /mnt base base-devel linux linux-firmware openssh firefox nano
genfstab -U -p /mnt > /mnt/etc/fstab
arch-chroot /mnt /bin/bash
nano /etc/locale.gen
locale-gen
echo LANG=it_IT.UTF-8 > /etc/locale.conf
export LANG=it_IT.UTF-8
nano /etc/vconsole.conf --> KEYMAP=it ; EDITOR=nano
export EDITOR=nano
ln -s /usr/share/zoneinfo/Europe/Rome /etc/localtime
hwclock --systohc --utc
echo "arch-efi-kvm-bhyve" > /etc/hostname
pacman -S net-tools dhcpcd netctl
systemctl enable dhcpcd
pacman -S iwd wpa_supplicant wireless_tools dialog iw
systemctl enable iwd
passwd
useradd -m -G wheel -s /bin/bash marietto
passwd marietto
visudo uncomment (uncomment to allow members of group wheel to execute any command)

---> pacman -S grub efibootmgr os-prober
---> grub-install
---> grub-mkconfig -o /boot/grub/grub.cfg

exit
reboot
pacman -S xf86-video-intel or nvidia
pacman -S xorg-server xorg-xinit
pacman -S

GNOME : gnome gnome-extra gnome-initial-setup
KDE : plasma kde-applications xdg-user-dirs sddm
LXQT : lxqt breeze-icons sddm xdg-user-dirs
MATE : mate mate-extra xdg-user-dirs lightdm lightdm-gtk-greeter

systemctl enable lightdm
pacman -S networkmanager
systemctl enable NetworkManager
systemctl disable dhcpcd
pacman -S pipewire-pulse
reboot
 
I talked about suspicion, not security. Although this suspicion is supported by a similar experience that Jeff Sipek had and the fact that I repeated the process twice using two different virtualizers using the SAME image and in one of them the problem is not present. I think also that Jeff seems to be more experienced than me and if he says that the problem is caused by an incomplete EFI implementation on Bhyve,probably it is.
 
I think any physical MB you can get, has what the manufacturer considers to be true EFI. But they are all different.
 
Take also in consideration that the solution of Jeff works for Debian (and this confirms his theory). I had been able to boot Debian using his method. Probably it works on Arch also,with some other little modifications.
 
Maybe I'm missing some piece information and the particular point you are making - is there a specific reason why you prefer booting with bhyve's EFI? There's bhyve-grub which definitely works quite well, also with Arch.

I've been booting an Arch vm this way for some time now and it gets the job done.
 
I'm experimenting with the passing trhu of my graphic card. I'm not sure that it can work if I don't use an EFI bootloader. Can u confirm this ?
 
Hi my friends. yes,I tried to apply the Sipek solution but it didn't work. I will try again.
 
cmoerz said:
If I'm reading this EuroBSD con slide deck right, it appears you really do need UEFI. Looks like this graphics passthrough stuff ain't easy at all.
Click to expand...
No, you don't need UEFI unless you want to use X11/frame buffer in your Arch Linux guest installation. Otherwise it's totally runnable without it, and you can have the text output in the console via SSH.
 
That's nice. So,the next experiment is ready. I will reinstall arch linux using grub only and not uefi. As for this installation,it booted. The Sipek wordaround worked. I was confused when I have changed the boot files. Unfortunately,the VM freezed later in the boot process (the cursor hangs on the black screen). And Im sure that it's not caused by the passhtru arguments that I have added because I have removed all the lines related to it and it hangs the same. I presume that bhyve does not like the desktop manager MATE. In the past I found a similar problem with KDE / sddm. This is how I launch bhyve + the arch linux VM :

Code: 
bhyve -S -c 8 -m 8G -w -H \
    -s 0,hostbridge \
    -s 1,virtio-blk,/mnt/da2s1/OS/bhyve/arch/arch-kvm.img \
    -s 6,virtio-net,tap0 \
    -s 29,fbuf,tcp=0.0.0.0:5900,w=1440,h=900 \
    -s 30,xhci,tablet \
    -s 31,lpc \
    -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
    vm0

It does not complete the booting process neither using this :

Code: 
-s 1,ahci-hd,/mnt/da2s1/OS/bhyve/arch/arch-kvm.img \
 
I've removed mate and I've installed xfce4. Same problem. Even in safe mode it hangs as soon as I press on the boot entry. I don't know how to debug it directly with bhyve. But I can do it in Liinux with qemu-kvm.
 
Have you made sure you made a bootable .iso file? If you roll your own iso's, that's important to remember. I got burned by that back in the day.
 
Have you tried reducing number of CPUs, i.e. with -c 4?

I've had some strange issues in the past, where my bhyve VM didn't boot properly unless I stated sockets, threads etc. instead of just -c N.
 
I've installed ssh server on arch linux from linux-qemu-kvm and I have been able to connect to it on freebsd,but the cursor hangs on the black screen,without being able to load the desktop manager. Check the image attached and think to what problem there could be.
 

Attachments

  • Istantanea_2021-09-10_20-17-52.png
    Istantanea_2021-09-10_20-17-52.png
    253.4 KB · Views: 117
ziomario said:
what do u mean ? why ? because that solution worked for booting debian with EFI. It should work also for Arch Linux.
Click to expand...
Grub with EFI/GPT is working just fine under bhyve. Following the installation from https://wiki.archlinux.org/title/installation_guide

Code: 
bhyve -AHP \
-s 0:0,hostbridge \
-s 31:0,lpc \
-s 2:0,virtio-net,tap0 \
-s 3:0,virtio-blk,/home/versus/guest.img \
-s 4:0,ahci-cd,/home/versus/archlinux-2021.09.01-x86_64.iso -c 4 -m 1024M \
-s 29,fbuf,tcp=0.0.0.0:5900,w=800,h=600,wait \
-s 30,xhci,tablet \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
guest


Code: 
cfdisk
...
gpt
vda1 EFI System 512M
vda2 Linux swap 4G
vda3 Linux filesystem rest
...
mkfs.vfat -F32 /dev/vda1
mkswap /dev/vda2
mkfs.ext4 /dev/vda3
mount /dev/vda3 /mnt
swapon /dev/vda2
pacstrap /mnt base base-devel linux linux-firmware openssh firefox nano dhcpcd grub efibootmgr dbus xorg-server xorg-xinit xf86-video-fbdev xf86-input-evdev mate mate-extra xdb-user-dirs lightdm lighdm-gtk-greeter
genfstab -U /mnt > /mnt/etc/fstab
arch-chroot /mnt
locale-gen
nano /etc/locale.gen
...
uncoment en_US.UTF-8 UTF-8
...
echo 'LANG=en_US.UTF-8' > /etc/locale.conf
echo 'KEYMAP=euro' > /etc/vconsole.conf
echo 'myhost' > /etc/hostname
ln -sf /usr/share/zoneinfo/Europe/Sofia /etc/localtime
systemctl enable dhcpcd
systemctl enable lightdm
passwd
mkdir /tmp/efi
mount /dev/vda1 /tmp/efi
grub-install --target=x86_64-efi --efi-directory=/tmp/efi --bootloader-id=GRUB
grub-mkconfig -o /boot/grub/grub.cfg
exit
reboot
 

Attachments

  • Screenshot at 2021-09-10 23-44-50.png
    Screenshot at 2021-09-10 23-44-50.png
    413.5 KB · Views: 122
Bhyve doesn’t support EFI variables. Since the bootloader location is saved as EFI variable, it’s not possible to boot from another location than the default location. This is not a bug,but more a missing feature.
 
You must stop with those negative commentaries (replies) toward FreeBSD. If you can't configure it it doesn't mean that it's not working or it's having a bug. No one is forcing you to use the FreeBSD or Bhyve and if it doesn't fill your requirements or expectations then just don't use it.

Bhyve does support efivar otherwise it will not be able to boot anything. You can use bhyve interactive shell or EFI graphical shell to change/create the boot order, or use efibootmgr from inside *nix.


1631349879781.png
1631349925675.png


To manually create a GRUB boot entry using efibootmgr located at /dev/vda1

efibootmgr -c -d /dev/vda -p 1 -L "GRUB" -l "\EFI\GRUB\grubx64.efi"

1631350636004.png

1631350858337.png
 
You must log in or register to reply here.
Back
Top