bhyve Running OpenBSD 7.4 on FreeBSD bhyve - but how? (Lots of hairs lost)

K

kalleboy

Hello dear fellows.

For God's sake, for Santa Claus's sake, I've been reading since hours, tutorials about running OpenBSD under FreeBSD's bhyve.. None of them works.

One of them recommends grub2-bhyve, other one suggests UEFI, another one suggests grub, one more article recommends bhyveload, tried them all - none of them works. Mostly, it doesn't boot at all, sometimes it boots but reboots back to the typical OpenBSD boot> loading screen..

Found out myself on FreeBSD Bugs' page as well, somehow: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=273814

Perhaps OpenBSD (as of 7.4) is not supported anymore? The default bhyve template obviously doesn't work neither. I've also tried this one: https://github.com/churchers/vm-bhyve/blob/master/sample-templates/openbsd.conf

And didn't help.

Tried cd74.iso, didn't work. Tried install74.img, didn't work.

Tried bhyve-devel package as well, nothing changed. Have lost too many hairs.

Is there any TRUE solution for this, please? Could someone kindly give it a quick try and report here please?

FreeBSD version: 14
 
I've been running several OpenBSD VMs on bhyve for years now - there's really absolutely nothing special about it, it 'just works'™
OpenBSD also boots fine via UEFI and has been for several years now - so there's absolutely no need to mock around with grub, which is really only needed for various (ancient) linux variants that won't come up using the UEFI loader.

At the OpenBSD loader you only have to manually 'set tty com0' at the very first boot of the installer to get console output redirected (same as on other headless systems) and that's it.
I seem to have mostly .img installer files, but also some .iso on the hosts I just checked, so both should work fine with bhyve. I use vm-bhyve and set up the VMs via vm install <vmname> <installerimage> and absolutely nothing special (e.g. bhyve options) in the template.

I have the -H and -w options set on some of the OpenBSD VMs, but I really have to review if this is still necessary (especially -w as it is intended for debugging). The more recently installed VMs don't have those options set, and I also don't have them in the current templates, so they don't seem to be required (any more).
 
Thanks for your prompt reply sko, but have you tried installing OpenBSD 7.4 under bhyve FreeBSD 14? It just doesn't work.
 
Okay, I've done installing OpenBSD now under FreeBSD bhyve, the installation went fine, however after a reboot, the installer started again, I stopped vm, started back.

Then I now get; "BdsDxe: failed to load Boot0001 "UEFI Misc Device" from PciRoot(0x0)/Pci(0x4,0x0): Not Found"

and then it says ">>Start PXE over IPv4." and then IPv6.

Any clue please?
 
kalleboy said:
Okay, I've done installing OpenBSD now under FreeBSD bhyve, the installation went fine, however after a reboot, the installer started again, I stopped vm, started back.

Then I now get; "BdsDxe: failed to load Boot0001 "UEFI Misc Device" from PciRoot(0x0)/Pci(0x4,0x0): Not Found"

and then it says ">>Start PXE over IPv4." and then IPv6.

Any clue please?
Click to expand...
Probably uefi boot entry is forgotten during reboot. If you use vm-bhyve port, set uefi_vars="yes" in the config of the VM. (only works on FreeBSD 14 and higher)
Otherwise, try to adjust boot file to /boot/efi/efi/boot/bootx64.efi, where /boot/efi is most likely the mount location and /efi/boot/bootx64.efi is the file location on the efi parititon.
 
After a successful OpenBSD 7.4 installation, and reboot, this appears in vm console:

bsdshot.png


And this is the config of my VM:

Bash: 
loader="uefi"
uefi_vars="yes"
cpu=1
memory=512M
network0_type="virtio-net"
network0_switch="public"

disk0_type="virtio-blk"
disk0_name="disk0.img"

bhyve_options="-w"

graphics="no"
xhci_mouse="no"

#disk1_type="ahci-hd"
#disk1_name="install74.img"
#This is installation image, commented both above out after the first installation and before the first boot

uuid="0750093e-a0a1-11ee-8dc5-ac9e17835d7b"
network0_mac="58:9c:fc:01:7a:1d"
 
I just did this successfully (on 14.0-STABLE). I just used bhyve. Using something like this:
Code: 
$ fetch  https://cdn.openbsd.org/pub/OpenBSD/7.4/amd64/install74.img
$ truncate -s 16G obsd.img
$ cat > run <<EOF
exec bhyve -c 1 \
        -s 0,hostbridge \
        -s 4,ahci-hd,install74.img \
        -s 5,nvme,obsd.img \
        -s 10,e1000,tap6,mac=aa:bb:cc:dd:ee:ff \
        -s 11,fbuf,tcp=0.0.0.0:5900 \
        -s 20,xhci,tablet \
        -s 31,lpc \
        -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
        -m 2G -H -w \
        -l com1,/dev/nmdm0B \
        obsd
exit 0
        -s 4,ahci-hd,install74.img \
EOF
$ nohup sh -x run

It appears install works only on a vnc console (not serial). sd0 is install74.img, sd1 is where you want to install. Make sure you use gpt for the disk format. Once the install is finished and you shutdown the vm, remove the -s 4,ahci-hd,install74.img \ line before rebooting.
 
Hi bakul Thanks for the tips and commands.

I don't have any GUI on my FreeBSD box, (a local network), and my Windows VNC client doesn't want to connect to the VNC on FreeBSD. Is there any way to do it through serial console?
 
Sorry for the late reply,

kalleboy said:
Thanks for your prompt reply [FONT=monospace]sko[/FONT], but have you tried installing OpenBSD 7.4 under bhyve FreeBSD 14? It just doesn't work.
Click to expand...
Yes, I also recently did a fresh installation of 7.4 on bhyve. However, I don't have any 14.0-RELEASE hosts running bhyve in production, only 13.2-RELEASE (could test on my laptop though). But what you have initially described sounds suspiciously like what happens if you don't redirect the default tty on a headless system: OpenBSD will just reboot if it can't open any tty to send STDOUT/STDERR to.
As said: enter "set tty com0" at the loader, then boot (press 'enter') and you can install via the bhyve serial console.

kalleboy said:
Okay, I've done installing OpenBSD now under FreeBSD bhyve, the installation went fine, however after a reboot, the installer started again, I stopped vm, started back.
Click to expand...
Did you specify GPT partition scheme and EFI boot during setup? The installer will complain that this might not boot on all systems (i.e. old cruft that only knows MBR), but you can ignore that message. I really don't know why that warning is still in place after so many years...
And RE the installer loading again after restart: bhyve won't detach the install medium during a reboot; the VM has to be stopped and re-started to boot from its disk image instead of the manually attached installer image. (I wanted to raise a PR about that for a long time...)
I suspect from your config file excerp that you are also using vm-bhyve? Then you don't need to attach the install media via config; just use vm install <vmname> <installerimage> (the image has to be imported via vm iso beforehand); see vm(8). You can directly import isos/images from an url with that command.

bakul said:
It appears install works only on a vnc console (not serial).
Click to expand...
There is no need for fiddling around with VNC; OpenBSD installs/boots/works just fine over serial console. But again: you have to manually redirect the default tty to com0 at first boot of the installer (or write it to the /etc/boot.conf on the install media).
The installer should ask you at some point if tty should be redirected to com0 by default - if you answer this with 'yes' it will put that 'set tty com0' line into /etc/boot.conf of the new installation.


Here's a config file from the 7.4 buildhost-VM I recently installed:
Code: 
# cat obsd-buildhost.conf
loader="uefi"
cpu=8
memory=64GB
bhyve_opts="-H -w"
network0_type="virtio-net"
network0_switch="mgmt40g"
disk0_dev="zvol"
disk0_type="nvme"
disk0_name="disk0"
uuid="55990b01-2f7f-11ee-8964-0cc47ac07a2e"
network0_mac="58:9c:fc:02:2f:8e"

As said: those "-H -w" options shouldn't be necessary; I have to dig into my notes to see why I put them into my template in the first place...
Regarding disk_type: bhyves 'nvme' emulation is faster by a huge margin than virtio - I'm using nvme for all VMs except some old linux applinaces.

edit: I just confirmed that this VM also boots with that "bhye_opts" line commented out.
 
sko thanks so much, these were the solution! Especially the GPT selection during installation.

1-"Did you specify GPT partition scheme and EFI boot during setup? The installer will complain that this might not boot on all systems (i.e. old cruft that only knows MBR), but you can ignore that message. I really don't know why that warning is still in place after so many years..."


2-"And RE the installer loading again after restart: bhyve won't detach the install medium during a reboot; the VM has to be stopped and re-started to boot from its disk image instead of the manually attached installer image. (I wanted to raise a PR about that for a long time...)"
I've installed and it boots fine now, thanks.

The only remaining things in my mind are;

I'd like my VMs and host machine to talk each other through a virtual network/virtual IP range. To do this, I did:

Code: 
vm switch create -a 192.168.2.1/24 public
vm switch add public re0 (my host's physical NIC)

And then, I set a static IP in my OpenBSD vm for itself: 192.168.2.2

Both host and my OpenBSD vm ping each other fine (in VM, I get reply from the host 192.168.2.1, and in host I get reply from VM 192.168.2.2)

I don't know though, if that's the correct way to do it? Without having any bridge at all?

Final question; what would you recommend for my OpenBSD VM to become online - in terms of DNS resolution?

Currently, it doesn't resolve any hostname. Even though my ISP's nameserver IPs are in resolv.conf file.

Should I install unbound in my host machine, or?

Many thanks! Regards.
 
kalleboy said:
I'd like my VMs and host machine to talk each other through a virtual network/virtual IP range. To do this, I did:
[...]

I don't know though, if that's the correct way to do it? Without having any bridge at all?

Final question; what would you recommend for my OpenBSD VM to become online - in terms of DNS resolution?

Currently, it doesn't resolve any hostname. Even though my ISP's nameserver IPs are in resolv.conf file.

Should I install unbound in my host machine, or?
Click to expand...

All of that mainly depends on your overall setup.

If this is a small-ish, single host with a single public IP (e.g. in a datacenter):
I usually have a single bridge configured in rc.conf (and set the 'type' to manual in vm-bhyve system.conf) to which I connect all the VMs and jails of that host and run an unbound (root] resolver in a jail that is used by all the VMs and jails on that host.
The host itself *could* also use that DNS, but until that jail is up, the host can't resolve anything. Especially during release-upgrades this can be problematic, so I usually either use a public DNS (e.g. quad-9) or local-unbound on the host.
If this host is supposed to generate a lot of DNS-requests (e.g. a mailserver that constantly checks for SPF, DKIM and other records) I'd highly recommend this setup, possibly even with some adjustments to the caching/TTLs. Otherwise this host might end up hitting some rate-limiting on public resolvers! (And it is just not polite to put such a load on public infrastructure if it can be easily avoided)

For more complex networks, I use distinct bridges for every vlan I want to connect jails/VMs to - this is true for local VLANs as well as those that carry public traffic. The host itself is usually only connected via a dedicated mgmt interface to the locked-down mgmt VLAN; jails and VMs can be attached to any VLAN, even the public one. So the host can talk at least to the jails/VMs that are also in the mgmt network (and usually also the DMZ) without any special/local configuration.
For such setups one (or more) local DNS servers which listen in different networks and are more practical. I prefer BIND with different views and ACLs, but unbound could handle this too. Those nameservers have interfaces in all networks except the public one - the VMs/jails that are directly connected to the internet either are also connected to one of the local networks (e.g. to access the database servers) and can use that DNS, or otherwise can reach at least the DNS in the DMZ (which has an RFC1819 IP, but as all of this happens within our AS they are of course still reachable/routable...).
 
Thank you so much sko for super informative, great answers and ideas, much appreciated!

Your first example was exactly my fact, small network with few VMs talking to each other (and to the public World). And for this, I did as follows, which runs;

Code: 
vm switch create -a 192.168.2.1/24 public
vm switch add public re0
vm switch list

NAME    TYPE      IFACE      ADDRESS         PRIVATE  MTU  VLAN  PORTS
public  standard  vm-public  192.168.2.1/24  no       -    -     re0

And then;

in HOST sysctl.conf:
net.inet.ip.forwarding=1

in HOST pf.conf:
nat on $ext_if from {192.168.2.1/24} to any -> ($ext_if)

in OPENBSD VM:

testvm# cat /etc/hostname.vio0
inet 192.168.2.2 255.255.255.0
testvm# cat /etc/mygate
192.168.2.1

and also the "/etc/resolv.conf" in vm has my ISPs DNS servers.

With a setup above (I didn't add anything in my rc.conf file), my VMs are able to ping each other, ping public Internet, ping host (hosts's pseudo network IP: 192.168.2.1) and so on.

Do you think the setup above is correctly implemented?

If all seems fine, then later, I'm going to apply some restrictive PF rules for each VM's services.

Much regards.
 
kalleboy said:
Do you think the setup above is correctly implemented?
Click to expand...
looks fine to me.
You could also just bridge the local network (i.e. add the host's physical interface to the bridge) and have the VMs directly connected to that network, so there's no need to NAT anything or put port-forwardings in place if you want to reach the VMs from your LAN.
But depends on your actual use case if this gives any advantages or is even wanted.

Regarding sysctls; also keep in mind there are those 3 that might be necessary:
Code: 
net.link.bridge.pfil_onlyip=0  # we need to be able to pass non-ip-packets on bridges (CARP!)
net.link.bridge.pfil_bridge=0  # disable packet filter on the bridge interface
net.link.bridge.pfil_member=0  # disable packet filter on the member interface
 
Not sure this is worth adding here, but an issue I ran into the other day--I went to install OpenBSD in vm-bhyve and used their default template which has grub. It gives you a console, but when I hit enter, I got a message that it couldn't find 6.2 (I think, maybe 6.4). Anyway, in this case, the solution was to hit e to edit the grub line, and change 6.2 to 7.4, (or whatever version you're planning to install. I'm putting that here for anyone who lands on this thread while searching why their openbsd install isn't going right.
 
scottro said:
Not sure this is worth adding here, but an issue I ran into the other day--I went to install OpenBSD in vm-bhyve and used their default template which has grub.
Click to expand...

The default vm-bhyve template for OpenBSD uses UEFI:
github.com

vm-bhyve/sample-templates/openbsd.conf at master · churchers/vm-bhyve

Shell based, minimal dependency bhyve manager. Contribute to churchers/vm-bhyve development by creating an account on GitHub.
github.com github.com

Generally, for anything but some ancient/exotic linux distros that don't support UEFI, you should never use grub because it blows up and fails to boot at every chance it gets - be it updates, changed drive configuration or a new moon phase...

If in doubt, just use uefi and it will usually 'just work'™.
(the Linux VM will still use grub, but mostly they figured out how to upgrade grub entries during updates, so it will keep booting after e.g. kernel upgrades which won't work if the entries are managed 'externally' via vm-bhyve configuration)
 
In what I had included with my vm-byve package, from pkg install vm-bhve, it used grub. I tried with uefi, but got a failure, I don't remember what it said, because I changed to grub and it worked without problem. I'll try with the default template at churchers, and report back.
 
TBH I haven't looked at those example configs for a long time, as I'm using my own repo for a host-agnostic base vm-bhyve configuration and my own templates...
The openbsd.conf from the 1.5 version tag is missing this update:
github.com

Fix and update OpenBSD template · churchers/vm-bhyve@e56d2ea

OpenBSD does not support kopenbsd from grub anymore: using uefi. Increaced minimum ram requirement. Added graphics: com0/1, stdio are currently not working.
github.com github.com
Might be worth a PR to add this patch to the port,or better yet, given the maintainer is also the developer, he could just bump the upstream version.
 
I tried with the churchers template and also sko's config but had no luck. It would boot, but then vm-log showed it just stopped. This isn't an important thing for me, and the grub version gives me what I need for a small openbsd install, so I don't want to take anyone's time. I suspect I'm missing something, but it's not important enough to me to troubleshoot at this time.
 
Well, to update, I tried with a modified template from churcher's which worked with vncviewer. I did find that when it asked which disk to install to, I had to pick disk1, rather than the default 0, as it seemed to otherwise be trying to install on the default image. The config I used loader="uefi" cpu=1 memory="2048" network0_type="virtio-net" network0_switch="public" disk0_type="virtio-blk" disk0_name="disk0.img" bhyve_options="-w" graphics="yes" xhci_mouse="yes" graphics_res="1600x900" graphics_wait="no" # Uncomment two lines below and download installXX.img, place in vm folder. # Replace XX with OpenBSD version. # You can delete the two lines below when installation is done #disk1_type="virtio-blk" #disk1_name="install74.img" uuid="f13a0f74-baa3-11ee-abcc-b8ca3abc453f" network0_mac="58:9c:fc:0d:7a:20"

I tried with the two lines uncommented, but it actually made no difference. Whether I had those disk1 lines commented or not, it still saw disk0 as the install image and I had to shutdown and
then run vm start to get to it. If I chose the default reboot, even with those lines commented out, it restarted the installation. I don't really need or want the GUI but I'm having issues when I leave those out. As it's more for playing around, I'm happy with what I have, though I'd like to succeed without the GUI. When I have more time, I'll come back with the errors I had, and what I've tried to do (and failed) to fix it.

Meanwhile, finally remembered to give a thanks to sko
 
Ok, after several tries and deciding what I wanted, which was basically ssh or console, but having the X sets, because some stuff I use requires them, I have an openbsd.conf that works for me.
loader="uefi" cpu=1 memory="2048" network0_type="virtio-net" network0_switch="public" disk0_type="nvme" disk0_name="disk0.img" graphics="yes" uuid="1c3c693a-bae8-11ee-afd1-b8ca3abc453f" network0_mac="58:9c:fc:03:16:45"
I leave the graphics=yes line because for whatever reason the install goes more smoothly with vncviewer. However, during the install process, as sko has said, it asks if you want default console to be com0 with a default of no that i changed to yes.

I found after some playing around with various configs, that, at least on my installs, I had to specify sd1 rather than the default sd0 for the disk, but after that I took defaults which I think were gpt and autopartition.
Originally I chose to not install any of the xwhtever.tar.gz sets but as mentioned, when I wanted to put some things in,I discovered I was missing dependencies, so I wound up again taking defaults and letting all the xwhater's be installed. I'm not sure how much space I lost to that, but it isn't enough to be a problem.

Another idiosyncrasy that I found was that if I chose reboot at the end of the install, the reboot would go right back to the installer image. I wound up doing a vm poweroff openbsd and vm start openbsd--I am sure I'm missing an obvious thing, Trying sysctl hw.allowpowerdown=1 and then doing halt -p just gives me a message to press any key to reboot. This is more an annoyance than anything else as it only matters after the original install.
 
You must log in or register to reply here.
Back
Top