Jail lost all packages

pming

Member

Reaction score: 5
Messages: 30

Hello everyone

I recently set up Nextcloud in a jail using apache24, mariadb101 and php70 on FreeBSD 10.3.
After rebooting my server the jail will not come up correctly. It won't mount datasets and all the packages I installed and their configuration seems gone. pkg info only returns pkg.

I used the following scripts to automount datasets when the jail starts:

http://vedad.kajtaz.net/perso/ezjail/rc.boot.zfs
http://vedad.kajtaz.net/perso/ezjail/rc.mount.zfs
http://vedad.kajtaz.net/perso/ezjail/rc.shutdown.zfs

Here's my /etc/jail.conf:
Code:
# /etc/jail.conf

exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.clean;
host.hostname = "$name";
mount.devfs;
path = "/zroot/jails/$name";

nextcloud {
    persist = true;
    exec.poststart = "/bin/sh /usr/local/bin/jail_scripts/nextcloud/jail_datasets.sh";
    exec.start = "/bin/sh /usr/local/etc/jails/rc.boot.zfs zroot/jails/nextcloud/zroot zroot/jails/nextcloud/db tank/nextcloud/data";
    exec.stop = "/bin/sh /usr/local/etc/jails/rc.shutdown.zfs";
    exec.poststop = "/bin/sh /usr/local/bin/jail_scripts/nextcloud/unjail_datasets.sh";
    allow.mount = true;
    allow.mount.zfs = true;
    enforce_statfs = 1;
    interface = "ixl0.50";
    ip4.addr = "192.168.50.41";
}
The poststart and poststop scripts will /sbin/zfs jail nextcloud <dataset> or unjail them respectively.

Stopping the jail takes forever. If I restart it and execute mount on the host, I see the line

Code:
devfs on /zroot/jails/nextcloud/dev (devfs, local, multilabel)
as many times as I restarted the jail. Same thing inside the jail. The data itself (Nextcloud data and database) still lives in the corresponding datasets.

Any suggestions are welcome.


Best Regards
Philipp
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 7,672
Messages: 30,574

Your start script never executes /etc/rc so the jail is never actually booted. Also note that poststart is executed after start, you probably want to start it before.

The order is:
Code:
prestart
start
poststart
--
prestop
stop
poststop
 
OP
OP
pming

pming

Member

Reaction score: 5
Messages: 30

I am pretty sure /etc/rc should get executed as soon as the necessary datasets are available to the jail, because rc.boot.zfs calls rc.mount.zfs, which in turn starts /etc/rc.

Here's the scripts so we can look them up easier:

rc.boot.zfs
Code:
#!/bin/sh
# Startup/shutdown scripts for ZFS-enabled jails

daemon -f /usr/local/etc/jails/rc.mount.zfs "$@"
exit 0
rc.mount.zfs
Code:
#!/bin/sh
# Startup/shutdown scripts for ZFS-enabled jails

# What we do here is:
# 1. Wait for jailed zfs to pop-up - which unfortunately doesn't happen at real boot time due to /etc/rc.d/jail weirdness
# 2. If mount is allowed, also mount the datasets
#    For the mount to be allowed, all of the following jail parameters need to be set:
#    * enforce_statfs=1 (or enforce_statfs=0)
#    * allow.mount.zfs=1
#    However, in order to manage the dataset without ability to mount/unmount (handy on backup jails where we MUST NOT mount the received datasets), the following is enough/required:
#    * enforce_statfs=1 (or enforce_statfs=0)


test_zfs_datasets()
{
   for i in "$@"; do
        if [ "`/sbin/zfs list -H -o name $i 2>/dev/null`" != "$i" ]; then
            return 1
        fi
    done

    return 0
}

while [ 1 ]; do
    sleep 1.5
    test_zfs_datasets "$@"
    if [ "$?" -eq 0 ]; then
        if [ `/sbin/sysctl -n security.jail.mount_zfs_allowed` -eq 1 -a `/sbin/sysctl -n security.jail.enforce_statfs` -lt 2 ]; then
            /sbin/zfs unmount -a 2>/dev/null
            /sbin/zfs mount -a 2>/dev/null
        fi

        /bin/sh /etc/rc
        exit 0
    fi
done
rc.shutdown.zfs
Code:
#!/bin/sh
# Startup/shutdown scripts for ZFS-enabled jails

/bin/sh /etc/rc.shutdown
_SHUTDOWN_RET=$?

/sbin/zfs unmount -a 2>/dev/null

exit $_SHUTDOWN_RET
I stumbled upon them on this post.

As the man page for jail says,
Code:
exec.poststart
        Command(s)    to run in the system environment after a jail is cre-
         ated, and after any exec.start commands have completed.
I think it's possible the poststart script does not get executed, because the exec.start script does not complete successfully (because the datasets need to be available for that).

I tested with prestart too, but no datasets will get mounted either. Using prestart, the datasets don't even get attached and zfs list inside the jail gives me:
Code:
no datasets available
I don't know why this should stop working after a reboot. Before that I could restart the jail without any problems. Also configuration files and rc scripts in /usr/local/etc for the packages are gone. Where else would a jail even save such files? Using the configuration in my first post, mount in the jail shows the root is mounted:

Code:
zroot/jails/nextcloud on / (zfs, local, noatime, nfsv4acls)
 
Top