bhyve Bhyve --> Guest Installs --> Devuan 4 (64 bit) & Win10 (64 bit) install/run, =>> Five Others FREEZE <<=

_al said:
No. Everything went well. I do not use cbsd or vm though, I use bhyve directly. But I don't think this should have any affect on the operation of the VM. (maybe in the future I'll switch to vm or cbsd).
Click to expand...
I've decided to start out using vm-bhyve.
I already have two VM's running, I just have to figure out how to fix the installation boot problem(s),
so that I can run the up-to-date versions I need.
Possible solutions were mentioned in links posted above.
So I'll play around with that and try to solve my pre-install boot problems.

"Keep it simple" to start, can get more advanced at a latter date, if that is (very likely) required.
 
  • Like
Reactions: _al
Dave-D said:
Is this the line that shows the installer where to find the boot loader?

-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd
Click to expand...
'bootrom' option specifies the name of the boot ROM to use for starting the virtual machine.
 
_al said:
'bootrom' option specifies the name of the boot ROM to use for starting the virtual machine.
Click to expand...

I'm starting to catch on. You're using UEFI from your local computer.

Appreciate all the patience w/newbies like myself.

I'll keep messing around and see if I can make it work using vm-bhyve.

Otherwise, maybe use bhyve directly.
 
I'm getting *very* frustrated.

Nothing installs. Except for the two mentioned, Devuan v4-64bit, and Windows 10-64-bit.
Everything else bombs. Alpine linux, devuan linux (even the desktop-live flavor), NetBSD, Windows 7, etc.

The *BSD's should install, as they are not supposed to have the UEFI issues.
Still, nothing.

Does everyone else have all these problems installing distros, or is it just me?

It should not be this hard.
Something is drastically wrong.

But what?

Maybe the templates are seriously out of date?
Maybe I'm doing something terrible wrong?

Even tried the command line approach, using bhyve directly, without vm-bhyve.
Same thing. Same types of errors.

It's always some kind of error likely caused by not finding the uefi files,
Server freezes, o/s crash, total lockup, have to reboot.

Repeat, repeat, repeat. Again, again, again.

If anyone has a problem with pride,
get them to mess around with computers.
Computers will help to keep you humble.
 
Dave-D said:
Server freezes, o/s crash, total lockup, have to reboot.
Click to expand...
Does /var/log/messages tell you anything useful?

I was able to complete an offline installation of Alpine Linux 3.18.3. I also had to use a BHYVE_UEFI_VARS.fd file (as pointed out earlier by Emrion) to make sure the vm can boot after installation.

Steps:
  1. truncate -s 5G /tmp/alpine.img
  2. cp /usr/local/share/uefi-firmware/BHYVE_UEFI_VARS.fd /tmp/alpine-uefi-vars.fd
  3. bhyve -k /tmp/alpine.conf
/tmp/alpine.conf:
INI: 
name=alpine
cpus=1
cores=1
threads=1
sockets=1
memory.size=1024M
acpi_tables=true

x86.vmexit_on_hlt=true
x86.vmexit_on_pause=true

pci.0.0.0.device=hostbridge

#pci.0.2.0.device=virtio-net
#pci.0.2.0.backend=tap0

pci.0.4.0.device=virtio-blk
pci.0.4.0.path=/tmp/alpine.img

pci.0.6.0.device=ahci
pci.0.6.0.port.0.type=cd
pci.0.6.0.port.0.path=/tmp/alpine-extended-3.18.3-x86_64.iso

pci.0.29.0.device=fbuf
pci.0.29.0.wait=true

pci.0.30.0.device=xhci
pci.0.30.0.slot.1.device=tablet

pci.0.31.0.device=lpc
lpc.bootrom=/usr/local/share/uefi-firmware/BHYVE_UEFI.fd
lpc.bootvars=/tmp/alpine-uefi-vars.fd

After the installation completes, I type reboot at the alpine:~# prompt. Next, I comment out the lines starting with pci.0.6.0 in /tmp/alpine.conf. Then, I recreate the vm by running bhyvectl --vm=alpine --destroy, followed by bhyve -k /tmp/alpine.conf.

Installation complete:
alpine-linux-installation-complete.png


Booting from hdd:
alpine-linux-boot-from-hdd.png
 
huggablemonad said:
I was able to complete an offline installation of Alpine Linux 3.18.3. I also had to use a BHYVE_UEFI_VARS.fd file (as pointed out earlier by Emrion) to make sure the vm can boot after installation.

Steps:
  1. truncate -s 5G /tmp/alpine.img
  2. cp /usr/local/share/uefi-firmware/BHYVE_UEFI_VARS.fd /tmp/alpine-uefi-vars.fd
  3. bhyve -k /tmp/alpine.conf
Click to expand...

My problem is, I can't get the install to boot before installation.

Not sure how to fix it.

But strongly suspect I need a more detailed .conf file.
Looks like that might take a lot of research, study, testing.

If a more advanced .conf file is, in fact, required, then I'm not sure why
there isn't a more advanced .conf file available, especially for debian,
which is pretty popular.

The devuan I'm trying to install is basically Debian, minus the system-d stuff, which
shouldn't matter for the .conf file.

Perhaps the bhyve-provided templates are very basic, and just a basic starting point?

Also, my *hard crashes* are suspect to me, but maybe not, since bhyve probably runs "close to the hardware,"
where any code catastrophe is "extreem or catastrophic."

???
 
Have you looked at bhyve_config(5) ? To be honest, I myself didn't use too many configuration options when installing/running Debian 11 (I have not tried installing Debian 12, which has a 6.x.x kernel, maybe I will encounter the same problems when installing it on my machine)
 
_al said:
Have you looked at bhyve_config(5) ? To be honest, I myself didn't use too many configuration options when installing/running Debian 11 (I have not tried installing Debian 12, which has a 6.x.x kernel, maybe I will encounter the same problems when installing it on my machine)
Click to expand...

If you can find the time, could you please try to install Debian 12 and let me know how you come out?
It won't take long, either the installation routine will open the initial screen, or it will bomb out before the installation screen gets a chance to load.
I used Devuan 5.0.0 "netinstall" -
I believe the Debian equivalent is: debian-12.1.0-amd64-netinst.iso
And then, if it does install, could you post the .conf file you used?

I could not install Devuan 4-32, but could install Devuan 4-64 (Both based on Debian Bullseye (11.1) with Linux kernel 5.10).
I could not install Devuan5-32 or Devuan5-64 (Both based on Debian Bookworm (version 12) with Linux kernel 6.1)
 
I tried. Everything installed without problems. How I did it. I created an deb12_inst.sh script :

[lanin@freebsd ~/v/deb12]$ cat deb12_inst.sh
sudo bhyve -c 8,sockets=2,cores=2,threads=2 -m 16G -H -w -S \
-s 0,hostbridge \
-s 3,ahci-cd,/mirlo/iso/debian-12.1.0-amd64-DVD-1.iso \
-s 4,ahci-hd,/mirlo/bsd/vms/deb12.img \
-s 5,virtio-net,tap0 \
-s 7,passthru,94/0/0 \
-s 29,fbuf,tcp=0.0.0.0:5900,w=1920,h=1200,wait \
-s 30,xhci,tablet \
-s 31,lpc \
-l com1,stdio \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
deb12

and ran it in the terminal.
In the second terminal I ran the command vncviewer :0.
At the last installation step, I clicked "Go Back" and switched to the shell (pictures deb12_01, deb12_02). Entered the required commands (picture deb12_03), then exited the shell and completed the installation. After exiting VM I destroyed it sudo bhyvectl --vm=deb12 --destroy
Next, I deleted line '-s 3,ahci-cd,/mirlo/iso/debian-12.1.0-amd64-DVD-1.iso \' from the script above, and rename it to deb12_run.sh.
In first terminal I ran deb12_run.sh, in second - command vncviewer :0

Final picture - deb12_04.
deb12_01.png


deb12_02.png

deb12_03.png

deb12_04.png
 
(in addition to above)
Before install I created deb12 image truncate -s 300G deb12.img
My system config files are:
/etc/rc.conf

...
ifconfig_igb0="DHCP"
...
cloned_interfaces="bridge0 tap0 tap1"
ifconfig_bridge0="addm igb0 addm tap0 addm tap1 up"
...


/boot/loader.conf

...
vmm_load="YES"
vmm_name="/boot/kernel/vmm.ko"
...
if_bridge_load="YES"
if_tap_load="YES"
pptdevs="94/0/0 216/0/0"
...
 
_al said:
I'll try to install Debian 12 today and post the results
Click to expand...

I duplicated your install, changing what was required to run on my server.
The install ran on the server itself, did not need to access it from a client.
Did not carry out the install.
The main thing was that the install started.

I did a web search, typing "Bhyve BdsDxe" into DuckDuckgo search engine.
Apparently this is a common problem over the last couple years.
Claimed to be a known bug, having something to do with Bhyve not being able to determine where to find the boot loader.

Results seem to vary with the distro and the versions / released files of that distro.

I wonder what would happen if you try to install the same .iso I've been trying to install.
See devuan.org: devuan_daedalus_5.0.0_amd64_netinstall.iso

No need to run through the whole install.
Just see if the installer will load.

Otherwise, I've decided,
I can get by with the two that I can install: Devuan 4.0.0-64bit and Windows 10-64bit.

I thought about trying other hypervisors.
But I would rather stick with Bhyve.
I'm sure this problem will get fixed, as the devs are able.
 
Dave-D said:
I wonder what would happen if you try to install the same .iso I've been trying to install.
See devuan.org: devuan_daedalus_5.0.0_amd64_netinstall.iso

No need to run through the whole install.
Just see if the installer will load.
Click to expand...
The installation started without problems, and everything installed well. I did exactly what is written in post #37. For installation I used the same two files - one to launch bhyve, the other to launch TigerVNC (~/v/d5/d5_i, ~/v/d5/d5_c - see picture d5_05). After the line "fbuf frame buffer base: ...." appeared in the ~/v/d5/d5_i window (picture d5_01), I launched TigerVNC (picture d5_02). At this moment two lines appeared in the ~/v/d5/d5_i window (after a couple of dozen empty lines) : "BdsDxe:..." (picture d5_03). Despite this, the installation goes well (picture d5_04). Final picture - d5_05 (bhyve is launched by the script ~/v/d5/d5_r).
d5_01.png

d5_02.png

d5_03.png

d5_04.png

d5_05.png
 
_al said:
The installation started without problems, and everything installed well. I did exactly what is written in post #37.
Click to expand...

More than likely this proves the problem is at my end.
(I did notice you used the 5.0.1 release, I used the 5.0.0 release, but more than likely not relevant.)

Question about the following line of your script:
-s 7,passthru,94/0/0 \
Is this for PCI Passthru?
I'm not sure if PCI Passthru is working on my system.

What are the chances that the problem *is* hardware related?

Two initial possibilities:

1. Here's an interesting question:
Since hypervisors like Bhyve run so "close to the hardware,"
could my problems finding the boot files be related to
my motherboards BIOS boot settings?

2. Hardware
I'm testing on an older (used) server motherboard, as follows:
Brand/Model = Supermicro X9DRD-iF, originally released about 8 years ago.
Board supports Intel VT-x and VT-d, if available on the CPU. (both enabled in BIOS).
CPU: Dual Intel Xeon X5-2630 V2. (Has Intel VT-x and VT-d)
Memory: 32GB ECC

Didn't see any VT-related error messages at any time.
My only problem so far, with the majority of attempts, is not being able to find the boot files.
Only Devuan 4-64bit and Windows 10-64 bit would install.
 
I forgot to delete it, I made a copy-paste from my old script. Delete this line. And try again.

But I don't think it will affect the installation/booting of the VM unless pptdevs are specified in loader.conf.
 
_al said:
I forgot to delete it, I made a copy-paste from my old script. Delete this line. And try again.
Click to expand...

Will do.
You don't think any of my "possible hardware issues" deserve attention?

I really appreciate your hanging in there with me.
It means a lot.
Thanks, _al !
 
  • Thanks
Reactions: _al
Dave-D said:
You don't think any of my "possible hardware issues" deserve attention?
Click to expand...
Quite possible....I'm not very good with hardware. I think it’s better to look in PRs for something related to VT-d on dual-processor systems.
 
_al said:
Quite possible....I'm not very good with hardware. I think it’s better to look in PRs for something related to VT-d on dual-processor systems.
Click to expand...

I think the virtualization part of the hardware is more of a go/no-go situation.
I started messing around with FreeBSD / Bhyve on an older Supermicro X7 series board (about 15+ years old),
but had problems right away. Installed Bhyve but couldn't get it to run. So the problems were immediate and obvious.
Turns out the board had support for VT-x, but did *not* have support for VT-d.
I'm sure that Intel VT-x and Intel VT-d are both "standards / standardized" to ensure industry stability via compliance to a "provided standard."
If not for that, you would everyone having "their own views of how virtualization should work," which would result in industry chaos.

However, the BIOS boot settings might be one possibility, I've heard of stranger things.

Would you mind checking your BIOS to see:
1. What boot devices you have enabled, and
2. in what order?

Thanks again, _al !
 
I have several SATA HDDs. One of them has Grub installed, through which I run FreeBSD, Debian, Windows. This HDD is listed first in the "boot devices' list". I also have a DVD reader which follows the HDD in the "boot devices' list". I don't have network boot enabled.

I use my machine rather as a workstation, not as a server. I sit in front of it and work on two or more VMs simultaneously...

Edit.
In BIOS: CSM enabled, UEFI and Legacy (both allowed), UEFI be tried first.
 
_al said:
In BIOS: CSM enabled, UEFI and Legacy (both allowed), UEFI be tried first.
Click to expand...

Another web search....
Bhyve Guests not finding the boot files may also be caused by misc. UEFI vs. Legacy BIOS issues, and possibly related settings.
It seems that some ended up installing Guests using Legacy boot, avoiding UEFI.

Are you using Legacy or UEFI boot for FreeBSD Host?
Are you using Legacy or UEFI boot for Guests?

I'm attempting to use UEFI for everything,
UEFI for FreeBSD (Host),
UEFI for each installed guest.

I'll try to find time to do some more experimenting.
 
You must log in or register to reply here.
Back
Top