Solved bhyve: vm_openf: No such file or directory in jailed bhyve

F

Ferocious1881

The jail follows the following config:
Code: 
acme {
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.consolelog = "/var/log/jail_console_${name}.log";

allow.mount;
allow.mount.devfs;
allow.raw_sockets;
allow.vmm;
enforce_statfs = 2;
securelevel = 2;
exec.clean;
mount.devfs;
devfs_ruleset = 25;

path = "/jails/${name}";
host.hostname = "${name}";
}
With the devfs rules:
Code: 
[devfs_rules_bhyve_jail=25]
add include $devfsrules_hide_all
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login
add path vmm unhide
add path vmm/* unhide
add path vmm.io unhide
add path vmm.io/* unhide
add path tap* unhide
add path nmdm* unhide
The jail was setup with
Code: 
bsdinstall jail /jails/acme
service jail start acme
pkg -j acme install bhyve-firmware
And then setup the vm as in the handbook (but without network) running inside the jail
Code: 
cd /root
truncate -s 16G guest.img
fetch https://download.freebsd.org/releases/ISO-IMAGES/15.1/FreeBSD-15.1-RELEASE-amd64-bootonly.iso
sh /usr/share/examples/bhyve/vmrun.sh -c 1 -m 1024M -d guest.img \
     -i -I FreeBSD-15.1-RELEASE-amd64-bootonly.iso guestname
I got a error of:
Code: 
Launching virtual machine "guestname" ...
bhyveload: vm_create: No such file or directory
Trying the alternative of running directly with bhyve as follows:
Code: 
bhyve -D -H -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd example
Gives me the error:
Code: 
bhyve: vm_openf: No such file or directory
 
Your devfs ruleset is missing /dev/vmmctl. Since FreeBSD 14 libvmmapi creates VMs through an ioctl on that node, so vm_create() fails with ENOENT when it can't open it. That's the "No such file or directory" you get from both bhyveload and bhyve.

Add "add path vmmctl unhide" to your ruleset and restart the jail, then it should work.
 
joneum@ said:
Your devfs ruleset is missing /dev/vmmctl. Since FreeBSD 14 libvmmapi creates VMs through an ioctl on that node, so vm_create() fails with ENOENT when it can't open it. That's the "No such file or directory" you get from both bhyveload and bhyve.

Add "add path vmmctl unhide" to your ruleset and restart the jail, then it should work.
Click to expand...
Well, now I am getting what I presume is some kind of kernel crash:
First as instructed did changed my devrules to something like:
Code: 
[devfs_rules_bhyve_jail=25]
add include $devfsrules_hide_all
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login
add path vmm unhide
add path vmm/* unhide
add path vmm.io unhide
add path vmm.io/* unhide
add path vmmctl unhide
add path tap* unhide
add path nmdm* unhide
Which did not work until I ran service devfs restart as said by devfs() and restart the jail.
Which solved the original error, but caused another error:
Code: 
bhyve: vm_openf: File exists
Which was solved restarting.

And finally when now when I do
Code: 
cd /root
truncate -s 16G guest.img
fetch https://download.freebsd.org/releases/ISO-IMAGES/15.1/FreeBSD-15.1-RELEASE-amd64-bootonly.iso
sh /usr/share/examples/bhyve/vmrun.sh -c 1 -m 1024M -d guest.img \
     -i -I FreeBSD-15.1-RELEASE-amd64-bootonly.iso guestname

Now there is a missing /dev/pci error,
Code: 
bhyve: pci_lpc_get_conf: Unable to open /dev/pci: No such file or directory
which is just ignored up to when I get this message:
Code: 
Welcome to FreeBSD!

Please choose the appropriate terminal type for your system.
Common console types are:
   ansi     Standard ANSI terminal
   vt100    VT100 or compatible terminal
   xterm    xterm terminal emulator (or compatible)
And shortly after the computer stops responding.

Additionally when running:
Code: 
bhyve -D -H -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd example
I got just nothing until do Control + C. I am not sure if such behavior is correct
 
Good progress. A few things:

The "File exists" error just meant the VM instance from the failed attempt was still around. No need to reboot for that, bhyvectl --destroy --vm=guestname cleans it up.

The /dev/pci warning is harmless, the lpc device only reads the host bridge IDs from it and falls back to defaults. If you want it gone, add add path pci unhide to your ruleset.

Your direct bhyve command showing nothing is expected. You only gave it a bootrom, no hostbridge, no disk and no com1 console, so there is simply nowhere for output to go. Stick with vmrun.sh for now, or add at least -s 0,hostbridge -s 31,lpc -l com1,stdio plus your disk and ISO.

The hang after the terminal type prompt is the part worth digging into. What exactly stops responding, just the VM console, or the whole host? Can you still ssh into the host from another machine when it happens? If the host really locks up, please post uname -a and anything that shows up on the host console.
 
joneum@ said:
The hang after the terminal type prompt is the part worth digging into. What exactly stops responding, just the VM console, or the whole host? Can you still ssh into the host from another machine when it happens? If the host really locks up, please post uname -a and anything that shows up on the host console.
Click to expand...
With hang, if meant the ssh session stops responding, so I close the ssh client, then try to reconnect the ssh no route to host error, finally go to check the computer it is not responding. Regardless looks like I confused this recurring problem of my setup to running the vm or it is transient and inconsistent, because it is not hanging now.
Anyway, thanks for the help.
 
I'm not sure I understand what you're describing, but i think i've had a similar problem.
I started a bhyve vm (not jailed) and my ssh session immediately hung. my server had lost access to the network. I went to the console and confirmed that I could not even ping my router. The connection came back on its own, but as soon as i typed a few keystrokes into ssh, it dropped again. it all resolved after shutting down the vm and i did not manage to reproduce the issue.
 
now i'm trying to jail bhyve and i'm having problems too. i added the necessary devices in /etc/devfs.rules like you did and restarted the devfs service
Code: 
[devfsrules_jail_bhyve=7]
add include $devfsrules_hide_all
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login
add path vmm unhide
add path vmm/* unhide
add path vmm.io unhide
add path vmm.io/* unhide
add path vmmctl unhide
add path nmdm* unhide

i also took the same jail config as you did

here is the output of ls /dev from the jail:
crypto fd/ null pts/ random stderr stdin stdout urandom vmm/ vmm.io/ vmmctl zero
but the vmm and vmm.io directories are empty, which is weird because they're not empty on the host

when i run bhyveload -d /disk0.img foo from inside the jail, i get:
bhyveload: openat: No such file or directory

/dev/vmm/foo is then create on the host instead of in the jail.
running bhyveload -d /disk0.img foo again will give the same output, but if i restart the jail and then run bhyveload, i get this:
bhyveload: vm_open: No such file or directory

if i try to create a vm named bar instead of foo, i get the previous error again.
 
Your devfs rules are fine, what's missing is the allow.vmm jail parameter. The vmm device nodes are jail-aware: VMs created on the host are intentionally invisible inside jails, which is why your vmm/ and vmm.io/ directories show up empty no matter what you unhide. Only VMs created from within a jail that has allow.vmm set will appear in that jail's devfs.

Without allow.vmm, the create ioctl through /dev/vmmctl still succeeds but the VM gets attached to the host, that's why /dev/vmm/foo appears on the host instead of in the jail, and bhyveload then can't open it (your openat error). After the jail restart the VM still exists from the first attempt, so the create fails and the fallback open hits the same invisible node, hence the vm_open error.

So: add allow.vmm; to your jail config (see jail(8)), restart the jail, and clean up the leftover VM on the host first with bhyvectl --destroy --vm=foo. If you plan to use PCI passthrough later you'd also need allow.vmm_ppt, but don't set that for untrusted jails.
 
thank you for the explaination, but i do use allow.vmm in my jail. i even used OP's jail config (adapted to my setup) because i couldn't figure out what was wrong with mine.
Code: 
tc@server ~
% head -n 19 /etc/jail.conf
acme {
    exec.start = "/bin/sh /etc/rc";
    exec.stop = "/bin/sh /etc/rc.shutdown";
    exec.consolelog = "/var/log/jail_console_${name}.log";

    allow.mount;
    allow.mount.devfs;
    allow.raw_sockets;
    allow.vmm;
    enforce_statfs = 2;
    securelevel = 2;
    exec.clean;
    mount.devfs;
    devfs_ruleset = 7;

    path = "/jails/containers/${name}";
    host.hostname = "${name}";
}

tc@server ~
% sudo jail -c acme
acme: created
tc@server ~
% sudo jexec acme
root@acme /
# bhyveload -d /disk0.img foo
bhyveload: openat: No such file or directory
(71)root@acme /
#
bhyve does work on the host.
P.S: i didn't think of showing it, but i did destroy the vm beforehand, as shown by the error message (openat, and not vm_open)
 
i think i found the source of the problem.
it was most likely some missing dependency of bhyve.I have a minimal jail template installed with pkgbase. I copied that to a new jail and installed bhyve on that jail with pkg.
And i guess pkg didn't pull all the necessary dependencies of bhyve, because after i created a new fresh install of a jail with bsdinstall, and did not uncheck "base-jail", i could create VMs inside that jail.
 
Good catch with the openat vs. vm_open distinction, and sorry, I sent you in the wrong direction: this openat has nothing to do with /dev/vmm at all. It comes from bhyveload's loader_open(), which opens the loader (userboot.so by default) relative to a directory fd on /boot, inside the jail:

fd = openat(bootfd, loader, O_RDONLY | O_RESOLVE_BENEATH);
if (fd == -1)
err(EX_OSERR, "openat");

So your allow.vmm/devfs side is fine now, the VM gets created, but your jail's filesystem is missing /boot/userboot.so. Check with ls -l /boot/userboot.so inside the jail; if the jail was built from a minimal/container image it won't be there. Copy it in from the host (it's part of base) or point bhyveload at a loader explicitly with -l /path/to/userboot.so.

One more thing: since the VM is created before the loader step fails, remember to bhyvectl --destroy it again before retrying, otherwise you'll see the vm_open variant of the error instead.
 
You must log in or register to reply here.
Back
Top