bhyve bhyve i386 OS on amd64 host?

rowan194 · Jan 2, 2024

Can an amd64 host support an i386 OS? I know the CPU itself can support both (as well as, technically, 8086 real mode), but unsure whether it can freely switch between 32 and 64 bit, and if so, whether bhyve (or FreeBSD as the host OS) supports that?

I've been searching, but I cannot find a clear answer to this myself.

I tried to boot an old i386 image as a test, but it doesn't work:

Code:

# bhyveload -m 256 -d IBM8.4-ad1.image-THROWAWAY vm0
Consoles: userboot

FreeBSD/amd64 User boot lua, Revision 1.2
Consoles: userboot

FreeBSD/amd64 User boot 4th, Revision 1.2
Segmentation fault (core dumped)

zirias@ · Jan 2, 2024

I don't know whether it works, I would expect it should work. But then, which OS? FreeBSD? Some very old version?

bhyveload(8) can only boot FreeBSD, and I'd expect problems with very old versions, maybe also with booting an i386 version (but then, you could try an i386 userboot.so with the -l switch).

Have you tried booting with UEFI and/or UEFI-CSM?

Phishfry · Jan 2, 2024

I am running bhyve on AMD64 and have i386 VM's.

Here is an example snipped from my startup script.

Code:

bhyveload -m 4G -S -d /vm/freebsd-vm4.img -c /dev/nmdm4A freebsd-vm4 &
sleep 5
bhyve -S -c 10 -m 4G -A -H -P -s 0:0,hostbridge -s 4:0,ahci-hd,/vm/freebsd-vm4.img -s 5:0,passthru,133/0/3 -s 10:0,passthru,13/0/0 -s 31:0,lpc -l com1,/dev/nmdm4A freebsd-vm4 &

Phishfry · Jan 2, 2024

My other virt box:

Code:

bhyveload -S -m 4G -c /dev/nmdm0A -d /vm/freebsd/freebsd-i386a.img freebsd-i386a &
sleep 15
bhyve -S -c 8 -m 4G -AHP -s 0:0,hostbridge -s 1:0,lpc -s 5:0,ahci-hd,/vm/freebsd/freebsd-i386a.img -s 7:0,passthru,15/0/0 -l com1,/dev/nmdm0A freebsd-i386a &

rowan194 · Jan 2, 2024

zirias@ said:
I don't know whether it works, I would expect it should work. But then, which OS? FreeBSD? Some very old version?

That's possibly the key. I just checked, and the image is really old. Like 20+ years. I had assumed that bhyve could boot anything. I will freely admit I'm stumbling around blind.

I haven't been able to boot something more current (12.4-R and 14.0-R) either, which is why I was trying the only other image I could find!

Phishfry · Jan 3, 2024

rowan194 said:
bhyveload -m 256 -d IBM8.4-ad1.image-THROWAWAY vm0

Notice here you don't tell it where to send console.

Use -c and send to nmdm (null modem console viewable on host with cu) or use stdio output.

I would also annotate your -m memory usage.
Is that 256M or 256G?

Phishfry · Jan 3, 2024

bhyveload(8)

man.freebsd.org

bhyveload can only load FreeBSD as a guest.

rowan194 · Jan 3, 2024

The image I tried to boot is FreeBSD, but really, really old (4.x, I think). It was really just me remembering that I happened to have a raw image of an old IDE drive sitting on my NAS, and trying it.

But to regain focus: what's the minimum version that is likely to work with bhyve? That's my main use case - occasional use of an older install of FreeBSD, with applications that are unlikely to compile cleanly on more current versions. I want to retire the hardware. It's running 9-STABLE.

Phishfry · Jan 3, 2024

I am not sure what the earliest FreeBSD that runs under bhyve is.

Here is the first forum post inquiring about it.

BHyVe

Not sure if this is the correct place for this question, but over the weekend I'm going to be testing BHyVe on my Intel i3 server and there isn't much information available on BHyve. Does anyone know the limit of the number of FreeBSD guests (since that is all that's supported at the moment)...

forums.freebsd.org

So 10-CURRENT is when it was introduced. I bet 9-STABLE could be ran.

cy@ · Jan 3, 2024

Phishfry said:
I am running bhyve on AMD64 and have i386 VM's.

Here is an example snipped from my startup script.

Code:

bhyveload -m 4G -S -d /vm/freebsd-vm4.img -c /dev/nmdm4A freebsd-vm4 & sleep 5 bhyve -S -c 10 -m 4G -A -H -P -s 0:0,hostbridge -s 4:0,ahci-hd,/vm/freebsd-vm4.img -s 5:0,passthru,133/0/3 -s 10:0,passthru,13/0/0 -s 31:0,lpc -l com1,/dev/nmdm4A freebsd-vm4 &

This works because the CPU "knows" whether it's in 16-, 32- or 64-bit mode and whether its in real or protected mode through bits set in machine state registers. All bhyve has to do is set the state of the CPU for the VM while the VM's PID is active.

rowan194 · Jan 3, 2024

Host: 12.4-RELEASE-p5

Guest 9-STABLE image shows the loader screen, but just quietly exits back to the host shell prompt (no error, no coredump) at "Booting..."

I can't find much information about the "raw" format. Is it literally a disk dump, or something proprietary? I imaged a physical disk this way:

Code:

dd if=/dev/da0 of=/nas01/backups/c1au-boot-tempimage.img bs=1m

And here's what happens when I try to invoke the VM using that image:

Code:

root@nas01:~ # bhyveload -m 512M -c stdio -d /nas01/backups/c1au-boot-tempimage.img vm0
Consoles: userboot

FreeBSD/amd64 User boot lua, Revision 1.2
Consoles: userboot

FreeBSD/amd64 User boot 4th, Revision 1.2
Loading /boot/defaults/loader.conf
|

  ______               ____   _____ _____
 |  ____|             |  _ \ / ____|  __ \
 | |___ _ __ ___  ___ | |_) | (___ | |  | |
 |  ___| '__/ _ \/ _ \|  _ < \___ \| |  | |
 | |   | | |  __/  __/| |_) |____) | |__| |
 | |   | | |    |    ||     |      |      |
 |_|   |_|  \___|\___||____/|_____/|_____/    ```                        `
                                             s` `.....---.......--.```   -/
 +------------Welcome to FreeBSD-----------+ +o   .--`         /y:`      +.
 |                                         |  yo`:.            :o      `+-
 |  1. Boot Multi User [Enter]             |   y/               -/`   -o/
 |  2. Boot [S]ingle User                  |  .-                  ::/sy+:.
 |  3. [Esc]ape to loader prompt           |  /                     `--  /
 |  4. Reboot                              | `:                          :`
 |                                         | `:                          :`
 |  Options:                               |  /                          /
 |  5. [K]ernel: kernel (1 of 2)           |  .-                        -.
 |  6. Configure Boot [O]ptions...         |   --                      -.
 |                                         |    `:`                  `:`
 |                                         |      .--             `--.
 |                                         |         .---.....----.
 +-----------------------------------------+


/boot/kernel/kernel text=0x679e22 data=0xab420+0xe4fc8 syms=[0x8+0xca380+0x8+0xe3fb1]
/boot/kernel/zfs.ko size 0x22b048 at 0xdb9000
loading required module 'opensolaris'
/boot/kernel/opensolaris.ko size 0x5040 at 0xfe5000
/boot/kernel/geom_mirror.ko size 0x20670 at 0xfeb000
/boot/zfs/zpool.cache size=0x13a4
Booting...
root@nas01:~ #

zirias@ · Jan 3, 2024

rowan194 said:
But to regain focus: what's the minimum version that is likely to work with bhyve?

I think any version would run, as long as it has either virtio drivers or drivers for "real" hardware bhyve emulates. My doubt was about bhyveload(8) being able to boot it, because it uses a special version of loader(8) (/boot/userboot.so from the host). You could try using the userboot.so from the guest (which works for me to boot 15-CURRENT from a 13-RELEASE host), but it won't exist in versions before bhyve was introduced. Or you could try to boot with UEFI/UEFI-CSM instead, which should work for any guest system.

If you're just starting with bhyve, I suggest having a look at sysutils/vm-bhyve, I find it very helpful to manage bhyve vms.

rowan194 · Jan 3, 2024

The 9.x stable disk does have userboot.so, but trying to use that bombs out:

Code:

root@nas01:~ # bhyveload -m 512M -c stdio -l /root/userboot-STABLE9.so -d /nas01/backups/c1au-boot-tempimage.img vm0
Abort (core dumped)

I was hoping for a simple one-liner to boot a headless server with a basic network interface and block storage, but it looks like it's going to be a lot more complicated than that...

_martin · Jan 3, 2024

rowan194 said:
I was hoping for a simple one-liner to boot a headless server with a basic network interface and block storage, but it looks like it's going to be a lot more complicated than that...

Not exactly what you were looking for but if the goal is to run "something" you could use VirtualBox for it (note though you can't mix hypervisors then, so either bhyve or vbox).

Phishfry · Jan 4, 2024

Sorry I am not familiar with userboot.so and I see no documentation.

What is it providing here? I have never used it.

zirias@ · Jan 4, 2024

Phishfry said:
Sorry I am not familiar with userboot.so and I see no documentation.

What is it providing here? I have never used it.

If you ever used bhyveload(8) to boot a FreeBSD guest, you have used it. It's the loader(8) code in the special form to be launched from userspace. Of course this way, you can only boot FreeBSD guests.

Once again, a bhyve guest has more booting options:

sysutils/grub2-bhyve offering booting with GRUB, mostly useful for Linux guests
sysutils/edk2-bhyve offering an UEFI implementation to boot using the guest's own EFI bootloader
sysutils/uefi-edk2-bhyve-csm same with a CSM, so a legacy BIOS bootloader can be used

The latter two options are "generic" and should be able to boot (almost) any guest.

zirias@ · Jan 4, 2024

rowan194 said:
The 9.x stable disk does have userboot.so, but trying to use that bombs out:

Code:

root@nas01:~ # bhyveload -m 512M -c stdio -l /root/userboot-STABLE9.so -d /nas01/backups/c1au-boot-tempimage.img vm0 Abort (core dumped)

Is this an i386 guest as well? I never tried something like that, so it's guesswork here, but I assume a 64bit bhyveload(8) binary won't be able to use a 32bit userboot.so.

rowan194 said:
I was hoping for a simple one-liner to boot a headless server with a basic network interface and block storage, but it looks like it's going to be a lot more complicated than that...

I really suggest you use UEFI (or CSM) to boot. Make sure your guest has a working bootloader on its virtual disk, either an EFI loader in an EFI partition, or the MBR bootcode plus the "second stage" in a freebsd-boot partition. Then of course you can launch the virtual machine with a "one-liner", but using the bhyve(8) command directly (with lots of flags). Or you use sysutils/vm-bhyve wich simplifies usage a lot.

rowan194 · Jan 5, 2024

I managed to get FreeBSD-9.2-RELEASE-amd64-memstick.img (from freebsd.org) to start without needing to specify a separate userboot.so

The image I actually want to boot uses a custom kernel, and in the past I've been fairly aggressive with commenting out unused stuff - including virtio, which I guess is the reason this is failing.

Will take some fiddling around with hardware to be able to boot the disk via bare metal again, but I'll see if recompiling GENERIC fixes it, and allows me to boot via VM image instead.

Edit: Copying GENERIC from 9.2R part works, but dies at the root mount, possibly because I had to rename the image zroot in order to import it and modify the file contents. Unsure if it can be renamed back to zroot on export? Seems I may need to boot in rescue mode (eg on a UFS boot?) in order to do this. Bare metal issue again...

Or perhaps error 45 is some completely different issue, like a 9.2R kernel trying to mount a 9.3+ ZFS file system?

Code:

Trying to mount root from zfs:zroottmp/ROOT/default []...
Mounting from zfs:zroottmp/ROOT/default failed with error 45.

Edit 2: Found some spare hardware, recompiled GENERIC, 9-STABLE now boots inside a VM.

Code:

sh /usr/share/examples/bhyve/vmrun.sh -d /dev/da0 vm0

One possibly odd thing I did was use the disk itself as the device, rather than an image of the disk. I thought this may cause problems, since the host can see things like gmirror and zroot on the attached guest disk, but once bhyve is running the device seems to be at least partly locked out on the host? I can see the device, I can 'hd' it, but 'zpool import' doesn't show anything, and 'gpart show da0' says no such geom. The latter works fine inside the VM.