Upgrading 13.0-RELEASE-p11 to 13.1-RELEASE breaks bhyve tap interface inside jails

Y

Yze

Hi everyone,

another surprise occurred after upgrading 13.0R to 13.1-RELEASE. The linux VM stopped starting that is using 3 tap devices to talk to the host. My best guess is that access to tap interfaces changed but didn't yet find a way to give permission to the jail.

Jail used to start with:

Code: 
grub-bhyve -m device.map -r host -d /vmm/lnx1v -c /dev/nmdm3A -M 6GB lnx1v-jail
bhyve -c 2 -A -H -P -m 6GB -s 0,hostbridge -s 1,lpc -s 2,virtio-blk,/dev/md42 -s 6,virtio-blk,/dev/zvol/tank/vm/lnxdisk0 -s 3,virtio-net,tap0 -s 4,virtio-net,tap1 -s 5,virtio-net,tap2 -lcom1,/dev/nmdm3A lnx1v-jail

Could open socket
device emulation initialization error: Protocol not supported

Code: 
 jail.conf snipped:

bee {
    devfs_ruleset = 27;
    allow.vmm;
    allow.raw_sockets;
    persist;
    allow.socket_af;
    host.hostname = "bee";
    path = /tank/bee;
}

tried to add allow.raw_sockets as well allow.socket_af as it appears to be socket protocol related, but didn't allow bhyve to work with it.

sysctl that previously worked:

Code: 
net.link.tap.up_on_open=1
net.link.tap.user_open=1

and devfs.rules for bhyve;:

Code: 
[devfs_rules_bhyve_jail=27]
add include $devfsrules_hide_all
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login
add path vmm unhide
add path vmm/* unhide
add path vmm.io unhide
add path vmm.io/* unhide
add path tap* unhide
add path zvol unhide
add path zvol/tank unhide
add path zvol/tank/vm unhide
add path zvol/tank/vm/* unhide
add path nmdm* unhide
add path 'md*' unhide
add path 'md*' mode 0777
#add path mem unhide
#add path kmem unhide
#add path pci unhide
#add path io unhide

Wondering, if anyone in the field also runs bhyve inside jails using tap and got success in 13.1-RELEASE to start them (again). Any hints appreciated.
 
I have hot-fixed an issue, which could be related.

After the update I was unable to start my openvpn in a jail with an error message Cannot open TUN/TAP dev /dev/tun100: No such file or directory (errno=2). It turned out, that this was due to missing device notes under /dev/- no /dev/tun100 there, even though I have configured it in the hosts devfs.conf. I tried to further change my /etc/devfs.conf, but none of these adaptions showed the expected effect on jails configured with a devfs_ruleset in /etc/jail.conf, which used to work before the upgrade.

My current workaround is, that I copied my contents from /etc/devfs.conf to /etc/defaults/devfs.conf and after a restart of the machine it was working again. This is a hotfix rather then an ideal solution. I hope it helps someone to save that amount of time which is required to find out what has changed there :)
 
Last edited:
hmm the devfs seems to work so far with my rules - e.g. /dev/tap0 shows up. Running truss shows it "dies" when it tries to use fnctl syscall against that tap device and gets "unsupported protocol" back from kernel. Not much we can do I would assume. To me that's a kernel bug - I could now reproduce that in another lab box and have to see how I can report that. Running bhyve in jail seems is nobody doing or they have all systems which do not use networking...
 
Yze said:
Running bhyve in jail seems is nobody doing
Click to expand...
Just out of curiosity: What's the point? I mean, most of the hypervisor code is in-kernel anyways?

Doesn't mean it shouldn't work, of course. I just try to understand what could be the benefit...
 
I do agree, as it does not separate the actual VM execution into the jail instance. We used bhyve like this from day 1, so far we never had issues with that design.
However, we run all hosts itself with bare minimum on read-only storage, everything is spawned in jails using another storage. Base is ideal without any pkg installed at all - pure "installworld". For the lab - the workaround atm is to use chroot instead of jail (which obviously works) but it would not hurt to keep the design with jail.
 
You must log in or register to reply here.
Back
Top