bhyve Creating VM Inside Jail Fails

[tivolo]

Hi all,

I am trying to jail a bhyve by following the step in the handbook (25.7.9). When i am inside a jail and i want to create a VM

bhyvectl --create --vm=bhyvevm0

I will get an error saying vm_open: bhyvevm0 could not be opened: No such file or directory

When i do this on the host, it succeeds without error. The strange thing is that i can see /dev/vmm/test_vm on the host, while inside the jail /dev/vmm is empty.

Here are some config files on the host:

devfs.rules

[devfsrules_jail_bhyve=100]
add include $devfsrules_hide_all
add include $devfsrules_unhide_login
add path 'urandom' unhide
add path 'random' unhide
add path 'crypto' unhide
add path 'shm' unhide
add path 'zero' unhide
add path 'null' unhide
add path 'mem' unhide
add path 'vmm' unhide
add path 'vmm/*' unhide
add path 'vmm.io' unhide
add path 'vmm.io/*' unhide
add path 'nmdmbhyve*' unhide
add path 'zvol' unhide
add path 'zvol/zroot' unhide
add path 'zvol/zroot/vms' unhide
add path 'zvol/zroot/vms/bhyvevm0' unhide
add path 'zvol/zroot/vms/bhyvevm1' unhide
add path 'tap10*' unhide

/etc/jail.conf.d/bhyve.conf

bhyve {
        $if = 0;
        exec.prestart = "/sbin/ifconfig epair${if} create up";
        exec.prestart += "/sbin/ifconfig epair${if}a up";
        exec.prestart += "/sbin/ifconfig epair${if}a name ${name}0";
        exec.prestart += "/sbin/ifconfig epair${if}b name jail${if}";
        exec.prestart += "/sbin/ifconfig ${name}0 inet 192.168.168.1/27";
        exec.prestart += "/sbin/sysctl net.inet.ip.forwarding=1";

        exec.clean;

        host.hostname = "jailed-bhyve";
        vnet;
        vnet.interface = "jail${if}";
        path = "/usr/local/jails/${name}";
        persist; #?
        securelevel = 3; #?
        devfs_ruleset = 100;
        mount.devfs;

        allow.vmm;
       
        exec.start += "/bin/sh /etc/rc";
        exec.stop = "/bin/sh /etc/rc.shutdown";

        exec.poststop += "/sbin/ifconfig ${name}0 destroy";
}

The base system on the jail is from http://ftp.freebsd.org/pub/FreeBSD/releases/amd64/15.1-RELEASE/base.txz

 

[gotnull]

I didn't try this myself, but looks like you have a missing devfs rule: /dev/vmmctl and possibly some more, there is a thread talking about this:
https://forums.freebsd.org/threads/...directory-in-jailed-bhyve.103574/#post-771749

I think I've read in the past another thread on that subject I mean "bhyve jailed" but I can find it anymore.

 

[tivolo]

/dev/vmmctl was exactly what was missing!

 

[tivolo]

I stay in this thread for the next issue that i have.

Bhyve is started now inside a jail and i use a zvol for virtio-blk:

bhyve -A -c 4 -D -H -m 2G \
        -s 0,hostbridge \
        -s 1,lpc \
        -s 2,virtio-net,tap100 \
        -s 3,virtio-blk,/dev/zvol/zroot/vms/bhyvevm0 \
    -s 4,ahci-cd,/vms/debian-13.7.0-amd64-netinst.iso \
        -s 31,fbuf,tcp=0.0.0.0:5900,w=1920,h=1080,wait \
        -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd,/usr/local/share/uefi-firmware/BHYVE_UEFI_VARS.fd \
        bhyvevm0

When I try to install debian, the partitioning fails due to input/output error:

This does not happen if i use a file instead of ZFS.