bhyve Red Hat not supported by cbsd?

B

bxbzq

Hi, I'm looking to install Red Hat bhyve guest using cbsd, but it is not there in the list of when running # cbsd bconstruct-tui. Do I get it right that Red Hat is not supported by cbsd yet?
 
In `cbsd bconstruct-tui` or ( `cbsd get-profiles --help` ) you see the profiles that come with CBSD. This does not mean that you are limited only to them.
You can register custom ISO via `cbsd media` or create your own profile ( the best option ):

"Contrib" CBSD profiles are located in the `~cbsd/etc/defaults/` directory ( vm-XXXX-YYY.conf files).
For user's profiles please use `~cbsd/etc/` directory.

For example, you can take Linux Rocky9 (or Alma Linux, or Oracle) profile as an example and change it for Red Hat, e.g.:

Code: 
cat ~cbsd/etc/vm-linux-RedHat-9-x86_64.conf <<EOF
vm_profile="RedHat-9-x86_64"
vm_os_type="linux"

long_description="RedHat Linux: 9.1"

fetch=1

# Official resources to fetch ISO's
iso_site="\
https://download.rockylinux.org/pub/rocky/9.1/isos/x86_64/ \
"

cbsd_iso_mirrors=

iso_img="RedHat-9.1-x86_64-dvd.iso"

register_iso_name="cbsd-iso-${iso_img}"
register_iso_as="iso-${vm_profile}"

default_jailname="redhat"

vm_package="small1"
imgsize="12g"

bhyve_rts_keeps_utc="1"

vm_vnc_port="0"
vm_efi="uefi"

xen_active=1
bhyve_active=1
qemu_active=1

clonos_active=0
sha256sum="0"

virtio_rnd="1"
EOF

After that you will be able to use it via `cbsd bconstruct-tui` or CBSDfile or:


cbsd bcreate jname=rhel1 vm_os_type=linux vm_os_profile=RedHat-9-x86_64 vm_ram=1g vm_cpus=1 imgsize=20g runasap=1


Save your RedHat.iso image as `~cbsd/src/iso/cbsd-iso-RedHat-9.1-x86_64-dvd.iso`(or create symlink )

PS: AFAIK, RedHat doesn't distribute ISOs via a direct permalink, so we can't let CBSD download them like other distributions.
 
Thanks Ole. Good to meet you. Red Hat is now in the list.

Looking for detailed tutorial to set up jail, bhyve and a Linux distro like Red Hat using cbsd, for newbies like me. Found this one, https://www.bsdstore.ru/en/bhyve_cbsd_howto.html, but it is a very brief intro, and I'm already stuck in # cbsd blogin redhat. Can you point me to the right direction?
 
By default (without extra configuration), the `cbsd blogin` command works only with cloud profiles/images. Cloud profiles is a pre-installed OS that is immediately ready for use without instalation stage, where the network, passwords, accounts and access keys are configured via a cloud-init package. The most distributions generate a cloud image along with ISO image. Since the CBSD generates settings for cloud-init, the `cbsd blogin` command with default config can use the correct key to login into bhyve VM.

Since you are using an ISO image, you must install the system in the usual way using the RedHat installer. Use VNC console to do this.

Back to the `cbsd blogin` command. After installing and configuring the network in the guest, you can customize the `cbsd blogin` per VM via blogin.conf config file and use it without 'cloud-init`.

Default blogin.conf configuration file located in ~cbsd/etc/detaults/ directory. You can overwrite globally for all virtual machines `cbsd blogin` behavior using `blogin.conf` in ~cbsd/etc/ directory.

Moreover, you can only override the behavior for specific virtual machines if you drop this configuration in the virtual machine settings directory: ~cbsd/jails-system/<VMNAME>/etc/

priority from top to bottom (last one - wins):
Code: 
~cbsd/etc/defaults/blogin.conf
~cbsd/etc/blogin.conf
~cbsd/jails-system/<VMNAME>/etc/blogin.conf
In `blogin.conf` file you can use `facts` for VM ($hwaddr, $ip4_addr and so on to generate correct SSH string or any other action (lauch RDP client), e.g.: ssh bxbz1@$ip4_addr
 
bxbzq said:
Thanks Ole. Good to meet you. Red Hat is now in the list.

Looking for detailed tutorial to set up jail, bhyve and a Linux distro like Red Hat using cbsd, for newbies like me. Found this one, https://www.bsdstore.ru/en/bhyve_cbsd_howto.html, but it is a very brief intro, and I'm already stuck in # cbsd blogin redhat. Can you point me to the right direction?
Click to expand...

Installing Linux on bhyve requires you to move the efi boot files before the reboot of the install session. See the cyberciti.biz link. Done it a few times now, works like a charm
 
The parent os is Ghostbsd stable, the bhyve instance with Freebsd is running 14.

pkg info cbsd
cbsd-13.2.0
Name : cbsd
Version : 13.2.0
Installed on : Sat Apr 22 19:41:02 2023 CEST
Origin : sysutils/cbsd
Architecture : FreeBSD:13:amd64
Prefix : /usr/local
Categories : sysutils
Licenses : BSD2CLAUSE
Maintainer : olevole@olevole.ru
WWW : https://www.bsdstore.ru
Comment : Yet another FreeBSD Jail, XEN and Bhyve Management Utility
Options :
BHYVE_GOP : off
Shared Libs required:
libssh2.so.1
libsqlite3.so.0
libedit.so.0
Annotations :
FreeBSD_version: 1301510
repo_type : binary
repository : GhostBSD
Flat size : 15.5MiB
Description :
Yet another FreeBSD jail and bhyve management tool.

Ghost is using a lot that Freebsd is using,
 
Ole said:
"Contrib" CBSD profiles are located in the `~cbsd/etc/defaults/` directory ( vm-XXXX-YYY.conf files).
For user's profiles please use `~cbsd/etc/` directory.
Click to expand...
Thanks Ole for this detailed information.

I created a profile for fnos-a NAS os and put it in `~cbsd/etc/` directory, but after executing command cbsd construct-tui, the new profile doesn't show up, do I need to run the initenv again? Hereby is the profile:

Code: 
% more vm-linux-fnOS-0.9.26.conf
# don't remove this line:
vm_profile="fnOS-0.9.26"
vm_os_type="linux"
# this is one-string additional info strings in dialogue menu
long_description="fnOS NAS: 0.9.26"
# custom settings:
fetch=1
# Official resources to fetch ISO's
iso_site="\
https://download.liveupdate.fnnas.com/x86_64/trim/fnos-0.9.26-1118.iso \
"
# Official CBSD project mirrors ( info: https://github.com/cbsd/mirrors )
#cbsd_iso_mirrors="https://mirror.convectix.com/iso/ #https://raw.githubusercontent.com/cbsd/mirrors/refs/heads/main/cbsd-iso.txt"
iso_img="fnos-0.9.26-1118.iso"
# register_iso as:
register_iso_name="cbsd-iso-${iso_img}"
register_iso_as="iso-${vm_profile}"
default_jailname="fnos"
# https://help.fnnas.com/articles/fnosV1/start/install-os.md
imgsize="2.3g"
#imgsize_min="8g"
vm_ram="5g"
# VNC
vm_vnc_port="0"
vm_efi="uefi"
# disable profile?
xen_active=1
bhyve_active=1
qemu_active=1
# Available in ClonOS?
clonos_active=1
md5="48c0bd5bd889ceb271a669de5a05033e"
# disable profile?
xen_active=1
bhyve_active=1
qemu_active=1
# Available in ClonOS?
clonos_active=1
md5="48c0bd5bd889ceb271a669de5a05033e"
#iso_img_dist_size="1971412992"
# enable birtio RNG interface?
virtio_rnd="1"
# firmware settings
cd_boot_firmware="bhyve"
hdd_boot_firmware="refind"
# vm_post_message in single line
vm_post_message="You need to add at least one additional disk, e.g: \n\n% cbsd bhyve-dsk mode=attach jname=${jname} dsk_controller=virtio-blk dsk_size=10g"
 
jazzi said:
the new profile doesn't show up, do I need to run the initenv again? Hereby is the profile:
Click to expand...
No, profiles are scanned every time you run `bconstruct-tui`.. The profile works for me (the only thing is that you made a mistake in the iso_site= parameter - needs URL without filename):
iso_site="https://download.liveupdate.fnnas.com/x86_64/trim/"
Click to expand...

Code: 
% cbsd get-profiles src=iso |grep fnO
fnOS-0.9.26                  linux       fnOS NAS: 0.9.26

% cbsd bcreate jname=fnos1 vm_os_type=linux vm_os_profile=fnOS-0.9.26 vm_ram=5g vm_cpus=2 imgsize=3g
% cbsd bhyve-dsk mode=attach jname=fnos1 dsk_controller=virtio-blk dsk_size=20g
% cbsd bstart
...

The profile also appeared in TUI.

1757833724024.png



I added profile into upstream ( with minor changes ), so you can do "update profiles" button in TUI or fetch manually:

Code: 
fetch -o ~cbsd/etc/vm-linux-fnOS-0.9.conf https://raw.githubusercontent.com/cbsd/cbsd-vmprofiles/refs/heads/master/vm-linux-fnOS-0.9.conf

PS: perhaps you need to delete the incorrect ISO ( check out: `ls -la ~cbsd/src/iso/ | grep -i fnos` ), since your profile does not contain a checksum and if the ISO was downloaded incorrectly - CBSD will not be able to check it, you need sha256= (added in my version)

PPS: Also, i'm not sure if we need to use 'refind' firmware in profile. Maybe 'bhyve' will work fine -- I haven't had time to check yet
PPS: What is the default password ;-) ? Maybe it makes sense to add in profile post-message.

1757834035495.png

1757834735400.png
1757835231718.png
 
RHEL10 and clones are very easy to install in vm-bhyve. I have a page https://srobb.net/vm-bhyve.html. But with RH, both 9 and 10, I was able to use the boot iso, then run vm install RH <path to iso> and it was simple. Some of the later 9.x versions had problems at one point due to some glibc issue (I've forgotten as it's been long fixed), but other wise, RH and clones were problem free in vm-bhyve.
 
Ole said:
PPS: What is the default password ;-) ? Maybe it makes sense to add in profile post-message.
Click to expand...
There is no default password, the password will be set through the installation.
Thanks Ole, I see the profile but unfortunately there is another problem, the vm can not be started, the status shows it's OFF and could not open the backing file, but the ls command shows the file is there, I run the following command to see why:

Code: 
bhyve: Could not open backing file: /usr/cbsd/vm/fnos/dsk1.vhd: No such file or directory

Code: 
sudo /usr/local/cbsd/share/bhyverun.sh -c /usr/cbsd/jails-system/fnos/bhyve.conf

ifconfig: interface tap2 does not exist

[debug] env LIB9P_LOGGING=/usr/cbsd/jails-system/fnos/cbsd_lib9p.log /usr/bin/nice -n 1 /usr/sbin/bhyve  -c 1 -m 4294967296 -H -A -U a20caad7-91e1-11f0-9cfa-70106fc7d856 -s 0,hostbridge   -s 4,virtio-blk,/usr/cbsd/vm/fnos/dsk1.vhd,sectorsize=512/4096  -s 1,ahci-cd,/usr/local/cbsd/upgrade/patch/efirefd.fd,ro  -s 5,virtio-net,tap2,mtu=1500,mac=00:a0:98:0a:87:9f  -s 6,virtio-rnd  -s 7,fbuf,tcp=192.168.31.240:5901,w=1024,h=768,password=123454 -s 30,xhci,tablet  -s 31,lpc -l bootrom,/usr/local/cbsd/upgrade/patch/efi.fd,/usr/cbsd/jails-system/fnos/BHYVE_UEFI_VARS.fd,fwcfg=qemu -l com1,stdio     fnos

bhyve exit code: 4. exit_action settings: 0, exit_action_mode ret: 0: must stoppped

See /tmp/tmp.6V8U37h7ND for details
Mon Sep 15 11:15:45 CST 2025
cmd: env LIB9P_LOGGING=/usr/cbsd/jails-system/fnos/cbsd_lib9p.log /usr/bin/nice -n 1 /usr/sbin/bhyve  -c 1 -m 4294967296 -H -A -U a20caad7-91e1-11f0-9cfa-70106fc7d856 -s 0,hostbridge   -s 4,virtio-blk,/usr/cbsd/vm/fnos/dsk1.vhd,sectorsize=512/4096  -s 1,ahci-cd,/usr/local/cbsd/upgrade/patch/efirefd.fd,ro  -s 5,virtio-net,tap2,mtu=1500,mac=00:a0:98:0a:87:9f  -s 6,virtio-rnd  -s 7,fbuf,tcp=192.168.31.240:5901,w=1024,h=768,password=123454 -s 30,xhci,tablet  -s 31,lpc -l bootrom,/usr/local/cbsd/upgrade/patch/efi.fd,/usr/cbsd/jails-system/fnos/BHYVE_UEFI_VARS.fd,fwcfg=qemu -l com1,stdio     fnos
-----

bhyve: Could not open backing file: /usr/cbsd/vm/fnos/dsk1.vhd: No such file or directory
Could not open backing file: No such file or directory
Device emulation initialization error: No such file or directory
Sleep 1 seconds...
Erase is delete.
Warning: unable to determine bhyve pid for: fnos
Killed
 
jazzi said:
I run the following command to see why:
Click to expand...
Is this behavior observed with any VM or FNOS only? In fact, it is a symlink to zvol (if you use zfs), so ZFS must be mounted ( `zfs list |grep fnos` ). Or it can be a regular file (UFS, NFS, GlusterFS, S3/fusefs), you can recreate it:
Code: 
rm -f /usr/cbsd/vm/fnos/dsk1.vhd
truncate -s5g /usr/cbsd/vm/fnos/dsk1.vhd
 
Ole said:
Is this behavior observed with any VM or FNOS only?
Click to expand...
Yes, it's FNOS only. I tried Linux Mint and no problem at all, hereby is the status:

Code: 
$ sudo cbsd bls

JNAME      JID    VM_RAM  VM_CURMEM  VM_CPUS  PCPU  VM_OS_TYPE  IP4_ADDR        STATUS  VNC
fnos       0      4096    0          1        0     linux       192.168.31.241  Off     192.168.31.240:5901
linuxmint  36436  512     0          1        0     linux       192.168.31.242  On      192.168.31.240:5902

And yes I use ZFS, hereby is the status:

Code: 
$ zfs list | grep fnos
zroot/cbsd/fnos                               20.0G  82.2G   276K  /usr/cbsd/vm/fnos
zroot/cbsd/fnos/dsk1.vhd                        56K   102G    56K  -

By the way, FNOS is Debian based os and my host OS is FreeBSD 14.3.
 
jazzi said:
By the way, FNOS is Debian based os and my host OS is FreeBSD 14.3.
Click to expand...
Yep, so this thread can probably be closed as "solved" (since I don't know of any open issues with RHEL and CBSD based distros at this point).

jazzi as for FNOS issue, try to recreate the symbolic link yourself (or delete the zvol and create the disk via `truncate -s5g dsk1.vhd` in ~cbsd/jails-data/fnos-data/ directory ):
( check that such path exists in dev /dev/zvol/*/*/fnos/... )

Code: 
ln -sf /dev/zvol/zroot/cbsd/fnos/dsk1.vhd ~cbsd/jails-data/fnos/dsk1.vhd

Or try to recreate the VM and if the problem persists (is there enough space on the pool? ) - please register an issue at https://github.com/cbsd/cbsd/issues
 
I got it work now, it's very interesting. The only difference is I leave the option of vm_cpu alone while set the cpu core to 1 before. The host cpu has two cores.

Also tried both "bhyve" and "refind" for hdd firmware and both works.

Thank you so much Ole, you let me feel so warm.
 
You must log in or register to reply here.
Back
Top