jails Jail mounted base nullfs sometimes not un-mounted after jail stop

Roy Chan · Feb 2, 2019

Hello all,

I follow the handbook (https://www.freebsd.org/doc/handbook/jails-application.html) and set up a nginx jail. I use a standard base partition mounted as read-only nullfs, and a skel partition mounted as read-write nullfs. My fstab is shown as follows:

Code:

/data/jail/template/base /data/jail/jails/nginx nullfs ro 0 0
/data/jail/skels/nginx /data/jail/jails/nginx/skel nullfs rw 0 0

I specified this fstab file in my jail.conf:

Code:

exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown && sleep 1";
exec.clean;
mount.devfs;

nginx {
    host.hostname = nginx;
    ip4.addr = 192.168.10.1;
    path = "/data/jail/jails/nginx";
    mount.fstab = "/data/jail/fstabs/nginx.fstab";
}

Within the jail, I use port to installed www/nginx with the default option. Everything works fine, but when I trigger service jail stop nginx, sometimes the base partition could not be un-mounted.

I used jail -v to check the output. Sometimes the jail could be stopped with everything clean, but sometimes the output was like:

Code:

nginx: run command: /sbin/umount /data/jail/jails/nginx/dev
nginx: run command: /sbin/umount -t nullfs /data/jail/jails/nginx/skel
nginx: run command: /sbin/umount -t nullfs /data/jail/jails/nginx
umount: unmount of /data/jail/jails/nginx failed: Device busy
jail: nginx: /sbin/umount -t nullfs /data/jail/jails/nginx: failed

After this, jls doesn't show this jail. However, the base partition is still mounted. I have no idea why. I've checked the opened files but no files were open. However, when I un-mount the file system from the host system, it was processed smoothly without any difficulties.

This problem sometimes happens but I haven't caught its pattern. I'm not sure if this is an Nginx issue or a jail issue, or just a fault in my configuration. I've checked multiple similar discussions but did not find a solution:

Solved - [Solved] Jail shutdown does not unmount

When I stop/shutdown jails, the mounts specified in /etc/fstab.jailname do not get unmounted. The explanation by Jamie (maintainer of jails) for the problem is: If you want to keep track of such things for proper handling in later removal, you should put those options in /etc/jail.conf (or some...

forums.freebsd.org

Cannot cleanly stop jails after using graphical applications

Starting and stopping the jail works fine when I stick to console apps. I can also successfully launch and use graphical jailed apps on the host display using e.g., jailme 1 firefox, but I cannot cleanly stop the jail after closing jailed graphical apps from my host X display. Here is my...

forums.freebsd.org

Jail root nullfs won't unmount Device Busy

I recently upgraded my server from FreeBSD 9.0 to FreeBSD 10.0. After the upgrade I converted all of my old jails from using /etc/rc.conf to using /etc/jail.conf. However when I attempt to stop some jails they refuse to unmount their root filesystem which is a read-only nullfs mount that points...

forums.freebsd.org

Jail fails to unmount a directory

It would be great if you could leave some possible solutions or general ideas about this annoying problem. Many thanks!

garrett · Dec 19, 2024

I have a similar setup as you describe above and unmount the jail path by using the jail() option exec.release to run a script on the host system which forcefully unmounts the directory: umount -f

I couldn't get it to work if I used exec.poststop.

Additional: I should also mention this worked for stopping an individual jail with
service jail stop jailB
However, "jailB" also depended on "jailA' to run and trying to stop all of them with
service jail stop
didn't work. Therefore, I removed the depend option on"jailB" and added to "jailA":
exec.prestop+="service jail stop JailB";
and this worked.

I also add the emptydir option to each jail's mount.fstab jail root path mount line and this seems to help.

NapoleonWils0n · Dec 19, 2024

hi mate

i had the same issue
the solution is to not use the fstab at all

because as you found out the mount points dont get unmounted when you stop the jail

what you do is add the the mount points to the jails config
that way the jail knows what the mount points are

Code:

classic {
    # hostname/path
    host.hostname = "${name}";
    path = "/usr/local/jails/containers/${name}";

    # permissions
    allow.raw_sockets;
    exec.clean;
    persist;
    sysvmsg=inherit;
    sysvsem=inherit;
    sysvshm=inherit;
    enforce_statfs=1;

    # permissions
    devfs_ruleset=7;

    # network
    ip4.addr="lo1|10.10.0.5/24";

    # mount
    mount += "devfs           $path/dev      devfs           rw                      0       0";
    mount += "tmpfs           $path/dev/shm  tmpfs           rw,size=1g,mode=1777    0       0";
    mount += "/tmp            $path/tmp      nullfs          rw                      0       0";
    mount += "/home           $path/home     nullfs          rw                      0       0";
}

GitHub - NapoleonWils0n/jailfox: firefox in a freebsd jail

firefox in a freebsd jail. Contribute to NapoleonWils0n/jailfox development by creating an account on GitHub.

github.com

garrett · Dec 19, 2024

NapoleonWils0n said:
hi mate

i had the same issue
the solution is to not use the fstab at all

because as you found out the mount points dont get unmounted when you stop the jail

what you do is add the the mount points to the jails config
that way the jail knows what the mount points are

Code:

classic { # hostname/path host.hostname = "${name}"; path = "/usr/local/jails/containers/${name}"; # permissions allow.raw_sockets; exec.clean; persist; sysvmsg=inherit; sysvsem=inherit; sysvshm=inherit; enforce_statfs=1; # permissions devfs_ruleset=7; # network ip4.addr="lo1|10.10.0.5/24"; # mount mount += "devfs $path/dev devfs rw 0 0"; mount += "tmpfs $path/dev/shm tmpfs rw,size=1g,mode=1777 0 0"; mount += "/tmp $path/tmp nullfs rw 0 0"; mount += "/home $path/home nullfs rw 0 0"; }

GitHub - NapoleonWils0n/jailfox: firefox in a freebsd jail

firefox in a freebsd jail. Contribute to NapoleonWils0n/jailfox development by creating an account on GitHub.

github.com

Ahh! I see it in man now!! Yeah, that's great I will try that. Thanks!

garrett · Dec 19, 2024

OK, tried it: the dependent jail root path is still mouted afterwards. But the mount option approach handles unmounting in the other jail noticably quicker.

In the jail with the remaining mountpoint, there is a prestop command that is unmounting a fuse filesystem within the jail. I suspect it is related to this, even though the fuse filesystem is unmounting without any problem.

Thanks again for the pointer!

NapoleonWils0n · Dec 19, 2024

no problem

was a tip from SirDice from memory

garrett · Dec 19, 2024

I should add too... when I try to stop both jails the new mount option errors out on the dependent jail
umount: unmount of /usr/local/jails/develop failed: Device busy
and then the other jail is skipped and not stopped.

Lots of combinations to try, for sure. I will still use the mount option on the first jail.

NapoleonWils0n · Dec 19, 2024

Lots of combinations to try, for sure. I will still use the mount option on the first jail.

thats all part of the fun
its like lego trying all the different permutations to see which blocks fit together

i have a jail called classic which i stop using the following command
and everything gets unmounted

Code:

doas service jail onestop classic

however if i just have one jail running
and use the following commad without the name of the jail

then sometimes the mount points dont get unmounted

Code:

doas service jail onestop