jails thinjail nullfs mountpoint does not get unmounted on stop

P

pgzh1

Hi,
I have a small issue with thinjails set up according to the FreeBSD handbook ( https://docs.freebsd.org/en/books/handbook/jails/ ).

For some reason the root nullfs mountpoint of the jail does not get unmounted after stopping it with
Code: 
service jail onestop thinjail
Therefore the jail cannot be started again:
Code: 
# service jail onestart thinjail
Starting jails: cannot start jail  "thinjail":
mount_nullfs: /jail/nullfs-base/thinjail: Resource deadlock avoided
jail: thinjail: /sbin/mount -t nullfs -o ro /jail/templates/freebsd-base /jail/nullfs-base/thinjail: failed
.

My setup is the following:
Code: 
zroot/jail                                     884M   183G   158M  /jail
zroot/jail/containers                          352M   183G   112K  /jail/containers
zroot/jail/containers/thinjail                 188K   183G  5.41M  /jail/containers/thinjail
zroot/jail/templates                           374M   183G    96K  /jail/templates
zroot/jail/templates/freebsd-base              369M   183G   369M  /jail/templates/freebsd-base
zroot/jail/templates/freebsd-overlay          5.42M   183G  5.33M  /jail/templates/freebsd-overlay

Code: 
# ls -lh /jail/containers/thinjail/
total 35 KB
drwxr-xr-x  30 root wheel  108B Apr  6 14:57 etc
drwxr-xr-x   2 root wheel    2B Mar 26 20:36 home
drwxr-x---   2 root wheel    8B Apr  6 14:57 root
drwxr-xr-x  24 root wheel   24B Apr  6 14:57 tmp
drwxr-xr-x   3 root wheel    3B Mar 26 21:16 usr
drwxr-xr-x  24 root wheel   24B Nov 28 04:42 var

# ls -lh /jail/templates/freebsd-base/
total 78 KB
-r--r--r--   1 root wheel  5.9K Nov 28 04:51 COPYRIGHT
drwxr-xr-x   2 root wheel   49B Mar 26 20:30 bin
drwxr-xr-x  15 root wheel   71B Mar 26 20:30 boot
dr-xr-xr-x   2 root wheel    2B Nov 28 04:42 dev
lrwxr-xr-x   1 root wheel   11B Apr  4 17:08 etc -> overlay/etc
lrwxr-xr-x   1 root wheel   12B Apr  4 17:08 home -> overlay/home
drwxr-xr-x   4 root wheel   80B Mar 26 20:30 lib
drwxr-xr-x   3 root wheel    8B Nov 28 04:43 libexec
drwxr-xr-x   2 root wheel    2B Nov 28 04:42 media
drwxr-xr-x   2 root wheel    2B Nov 28 04:42 mnt
drwxr-xr-x   2 root wheel    2B Nov 28 04:42 net
drwxr-xr-x   2 root wheel    2B Mar 26 20:42 overlay
dr-xr-xr-x   2 root wheel    2B Nov 28 04:42 proc
drwxr-xr-x   2 root wheel  152B Mar 26 20:30 rescue
lrwxr-xr-x   1 root wheel   12B Apr  4 17:08 root -> overlay/root
drwxr-xr-x   2 root wheel  150B Mar 26 20:30 sbin
lrwxr-xr-x   1 root wheel   11B Nov 28 04:42 sys -> usr/src/sys
drwxr-xr-x   2 root wheel    2B Apr  6 13:33 tmp
drwxr-xr-x  13 root wheel   14B Apr  4 17:09 usr
lrwxr-xr-x   1 root wheel   11B Apr  4 17:08 var -> overlay/tmp

Code: 
# cat /etc/jail.conf.d/thinjail.conf
thinjail {
  # PERMISSIONS
  devfs_ruleset="5";
  enforce_statfs="1"; # needed to allow file system mounts below jail root
  allow.raw_sockets;
  allow.reserved_ports;
  mount.devfs;
 
  vnet;
  vnet.interface = "iavf0";
 
  # STARTUP/LOGGING
  exec.start = "/bin/sh /etc/rc";
  exec.stop = "/bin/sh /etc/rc.shutdown";
  exec.poststop = "";
  exec.consolelog = "/var/log/jail_console_${name}.log";
  exec.clean;
 
  # HOSTNAME/PATH
  host.hostname = "${name}";
  path = "/jail/nullfs-base/${name}";
 
  # MOUNT
  mount.fstab = "/jail/${name}.fstab";
}

Code: 
# cat /jail/thinjail.fstab
/jail/templates/freebsd-base    /jail/nullfs-base/thinjail                      nullfs  ro                      0 0
/jail/containers/thinjail       /jail/nullfs-base/thinjail/overlay              nullfs  rw                      0 0
tmpfs                           /jail/nullfs-base/thinjail/tmp                  tmpfs   rw,mode=777,size=512M   0 0

Code: 
# cat /var/log/jail_console_thinjail.log
ELF ldconfig path: /lib /usr/lib /usr/lib/compat
32-bit compatibility ldconfig path: /usr/lib32
Starting Network: lo0 iavf0.
lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet 127.0.0.1 netmask 0xff000000
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0xc
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
iavf0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=4e507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
        ether 3c:fd:fe:d5:98:f2
        inet 10.0.7.12 netmask 0xffffff00 broadcast 10.0.7.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
route: message indicates error: File exists
add host 127.0.0.1: gateway lo0 fib 0: route already in table
add net default: gateway 10.0.7.1
route: message indicates error: File exists
add host ::1: gateway lo0 fib 0: route already in table
add net fe80::: gateway ::1
add net ff02::: gateway ::1
add net ::ffff:0.0.0.0: gateway ::1
add net ::0.0.0.0: gateway ::1
Updating motd:.
Creating and/or trimming log files.
Clearing /tmp (X related).
Updating /var/run/os-release done.
Starting syslogd.
Starting cron.
 
Mon Apr  6 14:57:34 CEST 2026
Stopping cron.
Waiting for PIDS: 4933.
.
Terminated

This is a jail I just created from the template I made according to the handbook.

I know I could just use exec.prestop to unmount the nullfs base, but something in the process of stopping the jails seems to be wrong here.
Actually I have another jail in which I try to run nfsd which also does not stop as it should - that nfsd jail goes to the "dying" state when I try to stop it and does not release the vnet device, and I have to reboot the host to resolve this situation.
Since this basic jail already exhibits an issue when beeing stopped, I would like to resolve that first and see if it also resolves the issue with the other jail.

As always, any help would be very much appreciated!

Edit: I'm running FreeBSD 15.0-RELEASE-p5 .
 
pgzh1 said:
I know I could just use exec.prestop to unmount the nullfs base
Click to expand...
Since you use mount.fstab, it should be automatically unmounted. At least, it's what I get.
I don't use jail service, but my own scripts (thin jails, too). jail service doesn't work with my jails, so I can't say why it causes this problem.

Reading the rc script, it uses the good command, yet: jail -r -f /path/to/jail.conf JailName
Try with the above command, just to see.
 
Thanks for the tip.
It doesn't solve my problem, but at least I get some feedback:
Code: 
# jail -r -f /etc/jail.conf.d/thinjail.conf thinjail
thinjail: removed
umount: unmount of /jail/nullfs-base/thinjail failed: Device busy
jail: thinjail: /sbin/umount -t nullfs /jail/nullfs-base/thinjail: failed

So something is keeping the root mount of the jail busy, but not for long. I can unmount it manually after the jail is removed without any problem.
I suppose when a jail is stopped, /etc/rc.shutdown should take care of stopping all running processes, shouldn't it?
Or is my mistake to assume the jail root can actually be unmounted by jail itself?
Even though almost all examples I found do it like this, but maybe I should manually mount the jails root before even invoking jail and let it persist on the host?

Emrion said:
I don't use jail service, but my own scripts (thin jails, too). jail service doesn't work with my jails, so I can't say why it causes this problem.
Click to expand...
May I ask why jail service does not work for you or why you have your own scripts?
 
pgzh1 said:
May I ask why jail service does not work for you or why you have your own scripts?
Click to expand...
I think it's related to my jails conf files because they aren't in /etc/jail.conf.d/.
Note that they don't have this overlay thing. It's strange to me. Could it be the overlay still mounted when it wants to unmount /jail/nullfs-base/thinjail?
 
You must log in or register to reply here.
Back
Top