bhyve Jailed Bhyve: /dev/nmdm-acme.1B: No such file or directory. Should I make a problem report?

F

Ferocious1881

I have a jail with the configuration at /etc/jail.conf
Code: 
acme {
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.consolelog = "/var/log/jail_console_${name}.log";

allow.raw_sockets;
allow.vmm;
enforce_statfs = 2;
securelevel = 2;
exec.clean;
mount.devfs;
devfs_ruleset = 25;

path = "/jails/${name}";
host.hostname = "${name}";

$id = "5";
$ip = "192.168.0.${id}/24";
$gateway = "192.168.0.1";
$bridge = "bridge0";
$epair = "epair${id}";

vnet;
vnet.interface = "${epair}b";

exec.prestart  = "/sbin/ifconfig ${epair} create up";
exec.prestart += "/sbin/ifconfig ${epair}a up descr jail:${name}";
exec.prestart += "/sbin/ifconfig ${bridge} addm ${epair}a up";
exec.start    += "/sbin/ifconfig ${epair}b ${ip} up";
exec.start    += "/sbin/route add default ${gateway}";
exec.poststop = "/sbin/ifconfig ${bridge} deletem ${epair}a";
exec.poststop += "/sbin/ifconfig ${epair}a destroy";
}
I setup it with the following commands:
Code: 
bsdinstall jail /jails/acme
pkg -j acme install vm-bhyve bhyve-firmware edk2-bhyve grub2-bhyve
jexec acme sysrc vm_enable="YES"
jexec acme mkdir /vms
jexec acme sysrc vm_dir="/vms"
jexec acme vm init
jexec acme cp /usr/local/share/examples/vm-bhyve/* /vms/.templates/
jexec acme vm switch create public
jexec acme vm switch add public epair5b
jexec acme vm iso https://dl-cdn.alpinelinux.org/alpine/v3.20/releases/x86_64/alpine-virt-3.20.3-x86_64.iso
jexec acme vm create -t alpine acme
jexec acme sed -i .bak "s/vanilla/virt/g" /vms/acme/acme.conf
jexec acme vm install -f acme alpine-virt-3.20.3-x86_64.iso
Now I procced to install alpine linux
finally when I start the vm and try to enter the console I get the error:
Code: 
vm console acme
/dev/nmdm-acme.1B: No such file or directory
link down
 
Last edited:
I think I forgot to add the /etc/devfs.rules
Code: 
[devfs_rules_bhyve_jail=25]
add path vmm unhide
add path vmm/* unhide
add path tap* unhide
add path nmdm* unhide
Also /dev/nmdm-acme.1B certainly exists in the host system, but is not being shared correctly.
 
Noted the missing https://forums.freebsd.org/threads/...jail-operation-not-permited.62192/post-359064. Now my /etc/jail.conf is
Code: 
acme {
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.consolelog = "/var/log/jail_console_${name}.log";

allow.mount;
allow.mount.devfs;
allow.raw_sockets;
allow.vmm;
enforce_statfs = 2;
securelevel = 2;
exec.clean;
mount.devfs;
devfs_ruleset = 25;

path = "/jails/${name}";
host.hostname = "${name}";

$id = "5";
$ip = "192.168.0.${id}/24";
$gateway = "192.168.0.1";
$bridge = "bridge0";
$epair = "epair${id}";

vnet;
vnet.interface = "${epair}b";

exec.prestart  = "/sbin/ifconfig ${epair} create up";
exec.prestart += "/sbin/ifconfig ${epair}a up descr jail:${name}";
exec.prestart += "/sbin/ifconfig ${bridge} addm ${epair}a up";
exec.start    += "/sbin/ifconfig ${epair}b ${ip} up";
exec.start    += "/sbin/route add default ${gateway}";
exec.poststop = "/sbin/ifconfig ${bridge} deletem ${epair}a";
exec.poststop += "/sbin/ifconfig ${epair}a destroy";
}
Problem persits
 
More troubleshooting and I get that the bhyve instance /usr/local/sbin/grub-bhyve -c /dev/nmdm-acme.1A -m /vms/acme/device.map -M 512M -r host -d /vms/acme acme, starting the machine, think confirms that the problem is that devfs is not sharing correctly the nmdm devices after they are created.
 
Things got worse when running vm console acme as well as vm start acme, and even jexec acme /usr/local/sbin/grub-bhyve -c /dev/nmdm-acme.1A -m /vms/acme/device.map -M 512M -r host -d /vms/acme acme, as the device /dev/nmdm-acme.1A is not even created. Why is this just happening?
 
If I setup with
Code: 
jexec acme vm create acme
jexec acme sed -i .bak "s/bhyveload/uefi/g" /vms/acme/acme.conf
jexec acme vm install -f acme alpine-virt-3.20.3-x86_64.iso
rather than
Code: 
jexec acme vm create -t alpine acme
jexec acme sed -i .bak "s/vanilla/virt/g" /vms/acme/acme.conf
jexec acme vm install -f acme alpine-virt-3.20.3-x86_64.iso
I can enter the machine from the host with
bhyve -A -D -H -P -S -u -w -c 1 -m 256m -s 0,amd_hostbridge -s 3,virtio-blk,/jails/acme/vms/acme/disk0.img -s 20,virtio-rnd -s 31,lpc -l com1,stdio -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd acme
and adding -s 4,ahci-hd,/vms/.iso/alpine-virt-3.20.3-x86_64.iso allow to boot from the iso
 
You must log in or register to reply here.
Back
Top