bhyve Bhyve --> Guest Installs --> Devuan 4 (64 bit) & Win10 (64 bit) install/run, =>> Five Others FREEZE <<=

D

Dave-D

.I'm working on a FreeBSD testing server, trying to get devuan 5 running as a Bhyve guest.

Testing Server:
FreeBSD 13.2 amd64
ZFS file system
32GB memory

I was able to install Windows 10 Pro 64-bit as a Bhyve guest (UEFI) without any problem. Runs great.
I was able to install Devuan 4.0.0 amd64 (UEFI) as a Bhyve guest without any problem. Runs great.

Tried to install Devuan 5.0.0 amd64 (UEFI) (based on Debian 12 (Bookworm)) as a Bhyve guest,
install freezes, then spits out the following error message:

X64 Exception Type - 06(#UD - Invalid Opcode)
CPU Apic ID - 00000000 ! ! ! !

FXSAVE_STATE - 00000000BFF57550
! ! ! ! Can't Find Image Information ! ! ! !


Maybe the install can't find the boot loader?
 
I just verified integrity of Devuan 5 amd64 install file = OK.
So thats not the problem.
 
I just tried to install the following distros, as a point of reference, results as follows:

Current release of Alpine Linux (64-bit):
Created VM using default Alpine template.
Error: File '/boot/vmlinuz-vanilla' not found.
Error: You need to load the kernel first.

Current release of Arch Linux (64-bit):
Created VM using default Arch template.
Error: File '/arch/boot/x86_64/vmlinuz' not found.
Error: You need to load the kernel first.


I'm starting to think all of my problems (with these different distros) are similar, namely,
the installer can't find the required files to run.
 
hansw said:
Click to expand...

No I didn't.
I used the following commands with the default template, then tried various modifications of the default template.
as follows:
vm create -t devuan -s 20G devuan64
vm install -f devuan64 downloaded_devuan.iso

This worked fine with Devuan 4 amd64.
Immediately failed with Devuan 5 amd64, and all the others mentioned.
All failed at the same point, likely couldn't find the files it needed to start the install.

The link you posted is more than likely my answer, at least for devuan (debian w/o system-d)

Can we somehow put all of the below (or at least the parts needed) into a default install template
so its easy for anyone to use to install debian or devuan VMs?

from your link:

bhyve -c 1 -m 1G -w -H \
-s 0,hostbridge \
-s 3,ahci-cd,/tmp/debian-10.7.0-amd64-netinst.iso \
-s 4,virtio-blk,/dev/zvol/zroot/debianvm \
-s 5,virtio-net,tap0 \
-s 29,fbuf,tcp=0.0.0.0:5900,w=800,h=600,wait \
-s 30,xhci,tablet \
-s 31,lpc -l com1,stdio \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
debianvm


Thanks hansw for taking time to help.
.
 
I've been messing around with this, but still coming up with nothing.

Devuan v4.0.0 - amd64 installs and runs great.

Devuan 4.0.0 - 32bit install freezes with the following error:

devuan4-32.png


Devuan 5.0.0 - 64bit install freezes with the following error:

devuan5-64.png


Been messing around with the template file for hours,
never getting any different result.

Searched the web and forums, not
coming up with any templates that work.

I find it strange that Devuan 4.0.0 64-bit installs and runs great,
but...
Devuan 4.0.0 32-bit will not install,
Devuan 5.0.0 32-bit will not install,
Devuan 5.0.0 64-bit will not install.

Also,
Alpine Linux 64-bit (most recent) will not install (same basic problem.)
Arch Linux 64-bit (most recent) will not install (same basic problem.)

Devuan 5.0.0 (32/64) was just released last month,
is based on Debian 12, without system-d.
 
Weird, do you have cbsd installed too? If so you might want to see if they succeed at it. I know I was unable to install debian until I move the efi file
 
hansw said:
Weird, do you have cbsd installed too? If so you might want to see if they succeed at it. I know I was unable to install debian until I move the efi file
Click to expand...

No. Is CBSD the proprietary Bhyve front-end?
I've only tried vm-bhyve so far.
Was thinking about trying some other front ends to see what I might learn.
Maybe some of those front ends have "knowledge" coded in as to the different (up to date) locations and what is needed to make each distro install.

I did see some free front ends I could try.
 
_al said:
hansw said:

... I know I was unable to install debian until I move the efi file
Click to expand...
How do you find out where the efi file is supposed to go, what files to include and where, etc?

I did read somewhere that different distro's are moving the efi file around which makes it impossible for bhyve guest install to find.
But then shouldn't that location be coded into the installer so that bhyve could discover that location and respond accordingly?
 
Dave-D said:
How do you find out where the efi file is supposed to go, what files to include and where, etc?
Click to expand...
As soon as I couldn't start Debian after installation, I searched the internet for a solution to the problem, and found instructions on this link. Tried. And everything worked out.;)
 
The same picture with CBSD ( with bhyve firmware and ReFIND ). However, if you try to boot the 'devuan_daedalus_5.0.0_amd64_server.iso' image as a disk (not ISO/CD), then everything fine ( e.g. options `iso2img="1"` in CBSD: https://github.com/cbsd/cbsd/blob/develop/etc/defaults/vm-linux-Devuan-x86-5.conf#L38 ). Same trick used to boot Linux Tails with bhyve.

Also needs to mount ISO (via `cbsd bconfig` -> vm_iso_path ) so that the installation can find the required ISO source by label

MB related: https://git.devuan.org/devuan/installer-iso/commit/54b83f0ee1a198884e9f848a5762e5da16e27f8a

1694028370278.png



1694028722590.png

1694028739206.png
 
I remember having had problems with apic with Linux quite often ... maybe try to boot your kernel with the "noapic" boot parameter. Also, there was/is a bug in debian, on the EFI partition, copy the efi file:
cp EFI/debian/grubx64.efi EFI/BOOT/bootx64.efi
 
Just for kicks, I just tried to install a bunch of different distros.
Most installs fail with the same basic type of error.
Looks like they can't find the boot partition/loader.

Devuan 5-64bit won't install, but the desktop-live .iso boots to a live desktop,
then after clicking on the "install" icon, goes into some jazz about missing gpt partition tables/etc.
Some of the results I'm getting don't make a lot of sense.

Conclusion:

Whatever the o/s, whether *BSD, linux or windoz, if I can get it to install, it runs great.
Overall, I'm pretty impressed with Bhyve.
Seems downright amazing.

I think the main problem is configuration.

The best solution may be to use a Bhyve management program that auto-configures as much as possible,
or at least makes it easy enough to get things installed and running in a reasonable amount of time.

Options I see so far are:
​
VM-Bhyve (already been using)​
​
virt-manager​
​
iohyve​
​
runhyve​
​
cbsd (already mentioned here)​
​
bastille​

Any additions / thoughts / suggestions / experiences re: the above list or this whole "auto-configure" approach in general?
 
Dave-D said:
How do you find out where the efi file is supposed to go, what files to include and where, etc?

I did read somewhere that different distro's are moving the efi file around which makes it impossible for bhyve guest install to find.
But then shouldn't that location be coded into the installer so that bhyve could discover that location and respond accordingly?
Click to expand...
I simply bumped into the problem trying to install Debian on it. Went looking for solutions and found this post Nixcraft made. It worked for me.
There are more frontends yes, tried some but I like cbsd because of the toolset it offers.
 
  • Like
Reactions: Ole
hansw said:
I like cbsd because of the toolset it offers.
Click to expand...

cbsd looks amazing.

FreeBSD --> cbsd --> jail mgt / vm mgt (Xen/Bhyve/QEMU/NVMM)

See this 4-part article:

eerielinux.wordpress.com

Exploring the CBSD virtual environment management framework – part 1: Introduction and installation

[New to Gemini? Have a look at my Gemini FAQ.] This article was bi-posted to Gemini and the Web; Gemini version is here: gemini://gemini.circumlunar.space/users/kraileth/neunix/2022/cbsd_pt1.gmi On…
eerielinux.wordpress.com eerielinux.wordpress.com
 
There is no file to copy if you set a BHYVE_UEFI_VAR.fd for the VM. During the install, debian writes where to boot in the UEFI vars. You find a copy of that file in sysutils/uefi-edk2-bhyve.

From bhyve(8):
Run a UEFI virtual machine with a VARS file to save EFI variables. Note
that bhyve will write guest modifications to the given VARS file. Be
sure to create a per-guest copy of the template VARS file from /usr.

bhyve -c 2 -m 4g -w -H \
-s 0,hostbridge \
-s 31,lpc -p com1,stdio \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CODE.fd,BHYVE_UEFI_VARS.fd
uefivm
Click to expand...
 
hansw said:
I like cbsd because of the toolset it offers.
Click to expand...
CBSD is well supported ;))

Just created Devuan 5.0 cloud image for `bhyve`: https://mirror2.bsdstore.ru/cloud/Devuan-x86-5.0.raw.xz

Checkout cloud-init instruction for `vm-bhyve`: https://github.com/churchers/vm-bhyve#using-cloud-init

As for CBSD, profiles just updated, so Devuan VM can be run instantly from the library via `cbsd bconstruct-tui` or (change `ci_ip4_addr`/`ci_gw4` to the correct ones for you):
cbsd bcreate jname=dev vm_ram=4g vm_cpus=2 vm_os_type=linux vm_os_profile=cloud-Devuan-x86-5 imgsize=20g ci_ip4_addr=10.0.100.88 ci_gw4=10.0.100.1 runasap=1
cbsd blogin

As for the original ISO issue, this will require further investigation. Any information would be helpful, feedback is welcome.
 
Related Question:

What are the chances that any of the problems I'm experiencing are being caused by hardware,
namely, the motherboard, and specifically, the motherboards "unique implementation" of virtualization? Or perhaps any motherboards virtualization implementation is standardized?

I'm testing with a slightly older Supermicro Dual-Intel CPU Server Board that supports virtualization.
(I believe VX-T and VX-d). All of this is new to me, so I just bought an older server board to throw a server together so I couild try it all out before spending money.

So, is it safe to say that since I *was* able to install 1.) a Windows 10 64-bit VM, and 2.) a Devuan 4 64-bit VM, both VM's installations went great and both ran without any issues, can I assume with any level of confidence that other distro's I'm having trouble installing (see all previous posts), would more than likely be caused by software and/or configuration issues?

In other words, is the motherboard basically an "all or nothing" when it comes to getting virtualization working? i.e. if one VM works, others should work as well, as long as correctly configured?

Basically, I'm wondering if I can pull the trigger and get parts ordered. I can fool around with VM implementation over the next month or two...

Note: I'm asking for your EXPERIENCES, *not* a personal guarantee.
 
_al said:
My experience - https://forums.freebsd.org/threads/...husiast-triple-boot-desktop.89521/post-616072
(post #5).
Click to expand...

Interesting. Thank you.

Did you run into any hardware-related issues with your setup?

I'm thinking that if I was going to have any issues with the motherboards virtual implementation that it would have popped up by now.

I'm thinking that virtualization (Intel VT-x and VT-d / etc.) is a standard. So the hardware end is probably more of an "all or nothing" proposition
(as long as you have a virtualization-capable CPU to go with the motherboard.)

In all the thrashing around I've been doing over this last week, I haven't seen any hardware-related errors yet.
Everything seems configuration-related.
My previous Supermicro motherboard had virtualization issues, but that was immediately obvious when after
intalling bhyve and vm-bhyve, I could not run bhyve without immediately kicking out a hardware-related issue.
Basically an immediate no-go from the very start. It seems the motherboard had VT-x, but not VT-d.

I've been messing around with FreeBSD 13.2 64bit --> CBSD --> Bhyve / QEMU.
So far, much better. Much more advanced tool set. I feel like I'm getting somewhere.

Also, with this arrangement, I can run an O/S as old as Windows XP if I want to.

I'll keep messing around and post my findings back to the board.
 
Findings:

CBSD seems to have a nice toolset, but seems in need of organization and documentation to make it all more usable.
It's jail-oriented heritage seems obvious, many corresponding "b___" commands (for bhyve) are not listed in the -help output,
have to dig for information and take a lot of notes, which makes everything a lot harder to use.

CBSD does show a lot of promise, and with some work could be totally amazing.

So, I'm down to:

vm-bhyve

iohyve

or

chyves (an iohyve total re-write)
 
Dave-D said:
Did you run into any hardware-related issues with your setup?
Click to expand...
No. Everything went well. I do not use cbsd or vm though, I use bhyve directly. But I don't think this should have any affect on the operation of the VM. (maybe in the future I'll switch to vm or cbsd).
 
You must log in or register to reply here.
Back
Top