Solved mount_nullfs: /jails/acme/usr/local/etc/step: Resource deadlock avoided

F

Ferocious1881

The jail configuration is
Code: 
# /root/acme-jail/jail.conf
acme {
exec.start = "/bin/sh /etc/rc";
exec.stop  = "/bin/sh /etc/rc.shutdown";
exec.consolelog = "/var/log/jail_console_${name}.log";

allow.raw_sockets;
allow.reserved_ports;
exec.clean;
mount.devfs;
devfs_ruleset = 5;

path = "/jails/${name}";
host.hostname = "${name}";

$id = "5";
$ip = "192.168.0.${id}/24";
$gateway = "192.168.0.1";
$bridge = "bridge0";
$epair = "epair${id}";

vnet;
vnet.interface = "${epair}b";

mount.fstab = "/root/acme-jail/fstab";

exec.prestart  = "/sbin/ifconfig ${epair} create up";
exec.prestart += "/sbin/ifconfig ${epair}a up descr jail:${name}";
exec.prestart += "/sbin/ifconfig ${bridge} addm ${epair}a up";
exec.start    += "/sbin/ifconfig ${epair}b ${ip} up";
exec.start    += "/sbin/route add default ${gateway}";
exec.poststop = "/sbin/ifconfig ${bridge} deletem ${epair}a";
exec.poststop += "/sbin/ifconfig ${epair}a destroy";
}
# /root/acme-jail/fstab
/jails/var-acme/        /jails/acme/usr/local/etc/step  nullfs  rw      0       5
/jails/var-acme/        /jails/acme/etc/step-ca nullfs  rw      0       5
Trying to run the Jail, goes like this:
Code: 
# jail -crm -f etc/acme.conf
mount_nullfs: /jails/acme/usr/local/etc/step: Resource deadlock avoided
jail: acme: /sbin/mount -t nullfs -o rw /jails/var-acme/ /jails/acme/usr/local/etc/step: failed
Have presetup the jail with
Code: 
#!/bin/sh

JAIL_ROOT="/jails/acme"
zfs clone zroot/jails/template@start "zroot/${JAIL_ROOT}"
zfs snapshot "zroot/${JAIL_ROOT}@start"
mkdir -p "${JAIL_ROOT}/usr/local/etc/step"
mkdir -p "${JAIL_ROOT}/etc/step-ca"
 
i found that mounts specified in the fstab
weren't umounted when shutting down the jail

Code: 
mount.fstab = "/root/acme-jail/fstab";

however using the mount option in the jails config
does automatically unmount all the mountpoints when shutting down the jail

Code: 
/etc/jail.conf.d/rocky.conf

Code: 
rocky {
    # hostname/path
    host.hostname = "${name}";
    path = "/usr/local/jails/linux/${name}";

    # permissions
    allow.raw_sockets;
    exec.clean;
    persist;
    sysvmsg=inherit;
    sysvsem=inherit;
    sysvshm=inherit;
    enforce_statfs=1;

    # permissions
    devfs_ruleset=7;

    # network
    ip4.addr="lo1|10.10.0.5/24";

    # mount
    mount += "devfs           $path/dev      devfs           rw                      0       0";
    mount += "tmpfs           $path/dev/shm  tmpfs           rw,size=1g,mode=1777    0       0";
    mount += "fdescfs         $path/dev/fd   fdescfs         rw,linrdlnk             0       0";
    mount += "linprocfs       $path/proc     linprocfs       rw                      0       0";
    mount += "linsysfs        $path/sys      linsysfs        rw                      0       0";
    mount += "/tmp            $path/tmp      nullfs          rw                      0       0";
    mount += "/home           $path/home     nullfs          rw                      0       0";

    # mount the video directory from the host to the jail after creating it
    mount += "/home/djwilcox/video $path/home/djwilcox/video  nullfs rw      0       0";
    # uncomment the line below for the xdg runtime directory for wayland after creating it
    mount += "/var/run/xdg/djwilcox $path/run/user/1001  nullfs rw            0       0";
}
 
NapoleonWils0n said:
i found that mounts specified in the fstab
weren't umounted when shutting down the jail

Code: 
mount.fstab = "/root/acme-jail/fstab";

however using the mount option in the jails config
does automatically unmount all the mountpoints when shutting down the jail

Code: 
/etc/jail.conf.d/rocky.conf

Code: 
rocky {
    # hostname/path
    host.hostname = "${name}";
    path = "/usr/local/jails/linux/${name}";

    # permissions
    allow.raw_sockets;
    exec.clean;
    persist;
    sysvmsg=inherit;
    sysvsem=inherit;
    sysvshm=inherit;
    enforce_statfs=1;

    # permissions
    devfs_ruleset=7;

    # network
    ip4.addr="lo1|10.10.0.5/24";

    # mount
    mount += "devfs           $path/dev      devfs           rw                      0       0";
    mount += "tmpfs           $path/dev/shm  tmpfs           rw,size=1g,mode=1777    0       0";
    mount += "fdescfs         $path/dev/fd   fdescfs         rw,linrdlnk             0       0";
    mount += "linprocfs       $path/proc     linprocfs       rw                      0       0";
    mount += "linsysfs        $path/sys      linsysfs        rw                      0       0";
    mount += "/tmp            $path/tmp      nullfs          rw                      0       0";
    mount += "/home           $path/home     nullfs          rw                      0       0";

    # mount the video directory from the host to the jail after creating it
    mount += "/home/djwilcox/video $path/home/djwilcox/video  nullfs rw      0       0";
    # uncomment the line below for the xdg runtime directory for wayland after creating it
    mount += "/var/run/xdg/djwilcox $path/run/user/1001  nullfs rw            0       0";
}
Click to expand...
It certainly solved my issue, but I don't think that non unmounted filesystems are the problem, as I checked with 'mount | grep acme', and only the ZFS entry for the root of the Jail, and no more, probably this merits a Problem Report. Thanks.
 
Question: can two jails mount the same host directory? I thought they could...but it's now working for me.

SUMMARY
15.0-RELEASE

Code: 
start jail_a, it mounts /foo at startup
start jail_b, it also tries to mount /foo, but fails with `Resource deadlock avoided`
stop jail_a
start jail_b, it mounts /foo at startup
start jail_a, it also tries to mount /foo, but fails with `Resource deadlock avoided`

So regardless of the jail, the first one to start is able to nullfs mount the host directory.

DETAILS
In my real world situation, the two jails are: backitup and dev_web. backitup has ro nullfs mounts, and dev_web has one rw nullfs mount. This error repros when the dev_web nullfs mount is ro, too.

/etc/jail.conf
Code: 
# vim: set syntax=sh:
$j="/jails";
path="${j}/containers/${name}";
host.hostname="${name}.jail";

exec.start="/bin/sh /etc/rc";
exec.stop="/bin/sh /etc/rc.shutdown";

exec.clean;
mount.devfs;
exec.timeout=90;
stop.timeout=90;

$ipaddr         =  "10.0.0.${id}";
$mask           =  "255.255.255.0";
$gw             =  "10.0.0.1";
vnet;
vnet.interface  =  "epair${id}b";

exec.prestart   =  "logger jail:prestart: trying to start jail ${name}...";

exec.prestart   += "ifconfig epair${id} create up";
exec.prestart   += "ifconfig epair${id}a up descr vnet-${name}";
exec.prestart   += "ifconfig bridge0 addm epair${id}a up";

exec.start      =  "/sbin/ifconfig lo0 127.0.0.1 up";
exec.start      += "/sbin/ifconfig epair${id}b ${ipaddr} netmask ${mask} up";
exec.start      += "/sbin/route add default ${gw}";
exec.start      += "/bin/sh /etc/rc";

exec.poststart  =  "logger jail:poststart: jail ${name} has started";

exec.prestop    =  "logger jail:prestop: shutting down jail ${name}";
exec.prestop    += "ifconfig epair${id}b -vnet ${name}";

exec.poststop   =  "logger jail:poststop: jail ${name} has shut down";
exec.poststop   += "ifconfig bridge0 deletem epair${id}a";
exec.poststop   += "ifconfig epair${id}a destroy";

exec.consolelog="/var/log/jail-${name}.log";
mount.fstab="${j}/fstab/${name}.fstab";

#interface=em0;

# include all the jail files
.include "/etc/jail.conf.d/*.conf";

persist;

/etc/jail.conf.d/
Code: 
root@fbsdhost4:/home/toddg # cat /etc/jail.conf.d/*
backitup {
    $id     = "10";
}
dev_web {
    $id     = "30";
}

/jail/fstab/backitup.fstab
Code: 
/opt/dev/file      /jails/containers/backitup/opt/dev/file                        nullfs ro 0 0
/opt/dev/db        /jails/containers/backitup/opt/dev/db                       nullfs ro 0 0

/jail/fstab/dev_web.fstab
Code: 
/opt/dev/file      /jails/containers/backitup/opt/dev/file                        nullfs rw 0 0

EXAMPLE REPRO

1. backitup is running
Code: 
root@fbsdhost4:/home/toddg # jls
   JID  IP Address      Hostname                      Path
     3                  backitup.jail                 /jails/containers/backitup

2. start the dev_web jail
Code: 
root@fbsdhost4:/home/toddg # service jail start dev_web
Starting jails: cannot start jail  "dev_web":
mount_nullfs: /jails/containers/backitup/opt/dev/file: Resource deadlock avoided
jail: dev_web: /sbin/mount -t nullfs -o ro /opt/dev/file /jails/containers/backitup/opt/dev/file: failed

the jail failed to start b/c it could not mount the resource. strangely, i _can_ mount this directory from the command line:

Code: 
root@fbsdhost4:/home/toddg # /sbin/mount -t nullfs -o rw /opt/dev/file /jails/containers/dev_web/opt/dev/file
# unmount this and continue testing
root@fbsdhost4:/home/toddg # /sbin/umount /jails/containers/dev_web/opt/dev/file


3. stop the backitup jail
Code: 
root@fbsdhost4:/home/toddg # service jail stop backitup
Stopping jails: backitup.

4. start the dev_web jail
Code: 
root@fbsdhost4:/home/toddg # service jail start dev_web
Starting jails: dev_web.

5. start the backitup jail
Code: 
root@fbsdhost4:/home/toddg # service jail start backitup
Starting jails: cannot start jail  "backitup":
mount_nullfs: /jails/containers/backitup/opt/dev/file: Resource deadlock avoided
jail: backitup: /sbin/mount -t nullfs -o ro /opt/dev/file /jails/containers/backitup/opt/dev/file: failed

This link suggests that there is a flag that goes in the /etc/jail.conf file to allow nullfs mounts from within the jail, but these are getting mounted from outside the jail.

Any suggestions are most welcome.
 
You're mounting to the same target directory.
The *destination* in the stab are the same, but should be in different containers.
 
gtz42 sorry, I don't understand.

shouldn't these both be possible?
jail_a mount host path /foo/bar/baz to /jails/jail_a/foo/bar/baz
jail_b mount host path /foo/bar/baz to /jails/jail_b/foo/bar/baz

the host path is the same for both jails, but the jails have different paths...
 
toddg said:
gtz42 sorry, I don't understand.

shouldn't these both be possible?
jail_a mount host path /foo/bar/baz to /jails/jail_a/foo/bar/baz
jail_b mount host path /foo/bar/baz to /jails/jail_b/foo/bar/baz

the host path is the same for both jails, but the jails have different paths...
Click to expand...
I haven't tried it inside of a jail, but it looks like it would: (I haven't noticed any real difference in terms of what's mountable via both methods, just whether or not the jail knows about it)
Code: 
# mkdir -p foo/bar/baz
# mkdir -p jail_a/foo/bar/baz
# mkdir -p jail_b/foo/bar/baz
# mount -t nullfs foo/bar/baz jail_a/foo/bar/baz
# mount -t nullfs foo/bar/baz jail_b/foo/bar/baz
# touch foo/bar/baz/1
# ls jail_*/foo/bar/baz
jail_a/foo/bar/baz:
1

jail_b/foo/bar/baz:
1
 
My experiments show me that while the above works on the host, once I mount directories into a jail, they are not mountable in other jails. Not sure if that's expected or not.
 
Well, all I can say, it's that there are a lot of people, including I, who mount the same dir for multiple jails and that works. It's the very principle of nullfs thin jails. So, there is something wrong in the way you do, but I don't know what.

I suggest to carefully examine your fstab files. An invisible character can sometimes cause serious headaches.
 
You must log in or register to reply here.
Back
Top