Solved mount_nullfs: /jails/acme/usr/local/etc/step: Resource deadlock avoided

F

Ferocious1881

The jail configuration is
Code: 
# /root/acme-jail/jail.conf
acme {
exec.start = "/bin/sh /etc/rc";
exec.stop  = "/bin/sh /etc/rc.shutdown";
exec.consolelog = "/var/log/jail_console_${name}.log";

allow.raw_sockets;
allow.reserved_ports;
exec.clean;
mount.devfs;
devfs_ruleset = 5;

path = "/jails/${name}";
host.hostname = "${name}";

$id = "5";
$ip = "192.168.0.${id}/24";
$gateway = "192.168.0.1";
$bridge = "bridge0";
$epair = "epair${id}";

vnet;
vnet.interface = "${epair}b";

mount.fstab = "/root/acme-jail/fstab";

exec.prestart  = "/sbin/ifconfig ${epair} create up";
exec.prestart += "/sbin/ifconfig ${epair}a up descr jail:${name}";
exec.prestart += "/sbin/ifconfig ${bridge} addm ${epair}a up";
exec.start    += "/sbin/ifconfig ${epair}b ${ip} up";
exec.start    += "/sbin/route add default ${gateway}";
exec.poststop = "/sbin/ifconfig ${bridge} deletem ${epair}a";
exec.poststop += "/sbin/ifconfig ${epair}a destroy";
}
# /root/acme-jail/fstab
/jails/var-acme/        /jails/acme/usr/local/etc/step  nullfs  rw      0       5
/jails/var-acme/        /jails/acme/etc/step-ca nullfs  rw      0       5
Trying to run the Jail, goes like this:
Code: 
# jail -crm -f etc/acme.conf
mount_nullfs: /jails/acme/usr/local/etc/step: Resource deadlock avoided
jail: acme: /sbin/mount -t nullfs -o rw /jails/var-acme/ /jails/acme/usr/local/etc/step: failed
Have presetup the jail with
Code: 
#!/bin/sh

JAIL_ROOT="/jails/acme"
zfs clone zroot/jails/template@start "zroot/${JAIL_ROOT}"
zfs snapshot "zroot/${JAIL_ROOT}@start"
mkdir -p "${JAIL_ROOT}/usr/local/etc/step"
mkdir -p "${JAIL_ROOT}/etc/step-ca"
 
i found that mounts specified in the fstab
weren't umounted when shutting down the jail

Code: 
mount.fstab = "/root/acme-jail/fstab";

however using the mount option in the jails config
does automatically unmount all the mountpoints when shutting down the jail

Code: 
/etc/jail.conf.d/rocky.conf

Code: 
rocky {
    # hostname/path
    host.hostname = "${name}";
    path = "/usr/local/jails/linux/${name}";

    # permissions
    allow.raw_sockets;
    exec.clean;
    persist;
    sysvmsg=inherit;
    sysvsem=inherit;
    sysvshm=inherit;
    enforce_statfs=1;

    # permissions
    devfs_ruleset=7;

    # network
    ip4.addr="lo1|10.10.0.5/24";

    # mount
    mount += "devfs           $path/dev      devfs           rw                      0       0";
    mount += "tmpfs           $path/dev/shm  tmpfs           rw,size=1g,mode=1777    0       0";
    mount += "fdescfs         $path/dev/fd   fdescfs         rw,linrdlnk             0       0";
    mount += "linprocfs       $path/proc     linprocfs       rw                      0       0";
    mount += "linsysfs        $path/sys      linsysfs        rw                      0       0";
    mount += "/tmp            $path/tmp      nullfs          rw                      0       0";
    mount += "/home           $path/home     nullfs          rw                      0       0";

    # mount the video directory from the host to the jail after creating it
    mount += "/home/djwilcox/video $path/home/djwilcox/video  nullfs rw      0       0";
    # uncomment the line below for the xdg runtime directory for wayland after creating it
    mount += "/var/run/xdg/djwilcox $path/run/user/1001  nullfs rw            0       0";
}
 
NapoleonWils0n said:
i found that mounts specified in the fstab
weren't umounted when shutting down the jail

Code: 
mount.fstab = "/root/acme-jail/fstab";

however using the mount option in the jails config
does automatically unmount all the mountpoints when shutting down the jail

Code: 
/etc/jail.conf.d/rocky.conf

Code: 
rocky {
    # hostname/path
    host.hostname = "${name}";
    path = "/usr/local/jails/linux/${name}";

    # permissions
    allow.raw_sockets;
    exec.clean;
    persist;
    sysvmsg=inherit;
    sysvsem=inherit;
    sysvshm=inherit;
    enforce_statfs=1;

    # permissions
    devfs_ruleset=7;

    # network
    ip4.addr="lo1|10.10.0.5/24";

    # mount
    mount += "devfs           $path/dev      devfs           rw                      0       0";
    mount += "tmpfs           $path/dev/shm  tmpfs           rw,size=1g,mode=1777    0       0";
    mount += "fdescfs         $path/dev/fd   fdescfs         rw,linrdlnk             0       0";
    mount += "linprocfs       $path/proc     linprocfs       rw                      0       0";
    mount += "linsysfs        $path/sys      linsysfs        rw                      0       0";
    mount += "/tmp            $path/tmp      nullfs          rw                      0       0";
    mount += "/home           $path/home     nullfs          rw                      0       0";

    # mount the video directory from the host to the jail after creating it
    mount += "/home/djwilcox/video $path/home/djwilcox/video  nullfs rw      0       0";
    # uncomment the line below for the xdg runtime directory for wayland after creating it
    mount += "/var/run/xdg/djwilcox $path/run/user/1001  nullfs rw            0       0";
}
Click to expand...
It certainly solved my issue, but I don't think that non unmounted filesystems are the problem, as I checked with 'mount | grep acme', and only the ZFS entry for the root of the Jail, and no more, probably this merits a Problem Report. Thanks.
 
Question: can two jails mount the same host directory? I thought they could...but it's now working for me.

SUMMARY
15.0-RELEASE

Code: 
start jail_a, it mounts /foo at startup
start jail_b, it also tries to mount /foo, but fails with `Resource deadlock avoided`
stop jail_a
start jail_b, it mounts /foo at startup
start jail_a, it also tries to mount /foo, but fails with `Resource deadlock avoided`

So regardless of the jail, the first one to start is able to nullfs mount the host directory.

DETAILS
In my real world situation, the two jails are: backitup and dev_web. backitup has ro nullfs mounts, and dev_web has one rw nullfs mount. This error repros when the dev_web nullfs mount is ro, too.

/etc/jail.conf
Code: 
# vim: set syntax=sh:
$j="/jails";
path="${j}/containers/${name}";
host.hostname="${name}.jail";

exec.start="/bin/sh /etc/rc";
exec.stop="/bin/sh /etc/rc.shutdown";

exec.clean;
mount.devfs;
exec.timeout=90;
stop.timeout=90;

$ipaddr         =  "10.0.0.${id}";
$mask           =  "255.255.255.0";
$gw             =  "10.0.0.1";
vnet;
vnet.interface  =  "epair${id}b";

exec.prestart   =  "logger jail:prestart: trying to start jail ${name}...";

exec.prestart   += "ifconfig epair${id} create up";
exec.prestart   += "ifconfig epair${id}a up descr vnet-${name}";
exec.prestart   += "ifconfig bridge0 addm epair${id}a up";

exec.start      =  "/sbin/ifconfig lo0 127.0.0.1 up";
exec.start      += "/sbin/ifconfig epair${id}b ${ipaddr} netmask ${mask} up";
exec.start      += "/sbin/route add default ${gw}";
exec.start      += "/bin/sh /etc/rc";

exec.poststart  =  "logger jail:poststart: jail ${name} has started";

exec.prestop    =  "logger jail:prestop: shutting down jail ${name}";
exec.prestop    += "ifconfig epair${id}b -vnet ${name}";

exec.poststop   =  "logger jail:poststop: jail ${name} has shut down";
exec.poststop   += "ifconfig bridge0 deletem epair${id}a";
exec.poststop   += "ifconfig epair${id}a destroy";

exec.consolelog="/var/log/jail-${name}.log";
mount.fstab="${j}/fstab/${name}.fstab";

#interface=em0;

# include all the jail files
.include "/etc/jail.conf.d/*.conf";

persist;

/etc/jail.conf.d/
Code: 
root@fbsdhost4:/home/toddg # cat /etc/jail.conf.d/*
backitup {
    $id     = "10";
}
dev_web {
    $id     = "30";
}

/jail/fstab/backitup.fstab
Code: 
/opt/dev/file      /jails/containers/backitup/opt/dev/file                        nullfs ro 0 0
/opt/dev/db        /jails/containers/backitup/opt/dev/db                       nullfs ro 0 0

/jail/fstab/dev_web.fstab
Code: 
/opt/dev/file      /jails/containers/backitup/opt/dev/file                        nullfs rw 0 0

EXAMPLE REPRO

1. backitup is running
Code: 
root@fbsdhost4:/home/toddg # jls
   JID  IP Address      Hostname                      Path
     3                  backitup.jail                 /jails/containers/backitup

2. start the dev_web jail
Code: 
root@fbsdhost4:/home/toddg # service jail start dev_web
Starting jails: cannot start jail  "dev_web":
mount_nullfs: /jails/containers/backitup/opt/dev/file: Resource deadlock avoided
jail: dev_web: /sbin/mount -t nullfs -o ro /opt/dev/file /jails/containers/backitup/opt/dev/file: failed

the jail failed to start b/c it could not mount the resource. strangely, i _can_ mount this directory from the command line:

Code: 
root@fbsdhost4:/home/toddg # /sbin/mount -t nullfs -o rw /opt/dev/file /jails/containers/dev_web/opt/dev/file
# unmount this and continue testing
root@fbsdhost4:/home/toddg # /sbin/umount /jails/containers/dev_web/opt/dev/file


3. stop the backitup jail
Code: 
root@fbsdhost4:/home/toddg # service jail stop backitup
Stopping jails: backitup.

4. start the dev_web jail
Code: 
root@fbsdhost4:/home/toddg # service jail start dev_web
Starting jails: dev_web.

5. start the backitup jail
Code: 
root@fbsdhost4:/home/toddg # service jail start backitup
Starting jails: cannot start jail  "backitup":
mount_nullfs: /jails/containers/backitup/opt/dev/file: Resource deadlock avoided
jail: backitup: /sbin/mount -t nullfs -o ro /opt/dev/file /jails/containers/backitup/opt/dev/file: failed

This link suggests that there is a flag that goes in the /etc/jail.conf file to allow nullfs mounts from within the jail, but these are getting mounted from outside the jail.

Any suggestions are most welcome.
 
gtz42 sorry, I don't understand.

shouldn't these both be possible?
jail_a mount host path /foo/bar/baz to /jails/jail_a/foo/bar/baz
jail_b mount host path /foo/bar/baz to /jails/jail_b/foo/bar/baz

the host path is the same for both jails, but the jails have different paths...
 
toddg said:
gtz42 sorry, I don't understand.

shouldn't these both be possible?
jail_a mount host path /foo/bar/baz to /jails/jail_a/foo/bar/baz
jail_b mount host path /foo/bar/baz to /jails/jail_b/foo/bar/baz

the host path is the same for both jails, but the jails have different paths...
Click to expand...
I haven't tried it inside of a jail, but it looks like it would: (I haven't noticed any real difference in terms of what's mountable via both methods, just whether or not the jail knows about it)
Code: 
# mkdir -p foo/bar/baz
# mkdir -p jail_a/foo/bar/baz
# mkdir -p jail_b/foo/bar/baz
# mount -t nullfs foo/bar/baz jail_a/foo/bar/baz
# mount -t nullfs foo/bar/baz jail_b/foo/bar/baz
# touch foo/bar/baz/1
# ls jail_*/foo/bar/baz
jail_a/foo/bar/baz:
1

jail_b/foo/bar/baz:
1
 
You must log in or register to reply here.
Back
Top