I recently upgraded my server from FreeBSD 9.0 to FreeBSD 10.0. After the upgrade I converted all of my old jails from using /etc/rc.conf to using /etc/jail.conf. However when I attempt to stop some jails they refuse to unmount their root filesystem which is a read-only nullfs() mount that points to a shared FreeBSD 10.0 world. All jails start and run correctly the only issue is at shutdown, and only occurs for some jails.
Output from running
Everything appears to be unmounting in the proper order, but the root mount just refuses.
One jail that refuses to umount its root was upgraded from a FreeBSD 9.0 built world to a FreeBSD 10.0 built world and has gone through the complete upgrade process of using mergemaster() and upgrading all ports. Another jail was newly created using only a FreeBSD 10.0 built world without requiring any upgrades to ports or files since it was up-to-date already. Yet both jails fail to unmount their root mounts.
I'm at an extreme loss as to why this is happening and have used google and searched the forums but can only find people with the similar issues but all tend to be related to files being open and displayed when using fstat(), however in my case fstat() shows nothing even if told to show memory mapped files.
Edit: also here's the output of
Output from running
jail -r -v www1
Code:
www1: run command in jail as root: /bin/sh /etc/rc.shutdown
www1: removed
www1: run command as root: /sbin/umount /mnt/disk1/j/10.0.3.1-www1/dev
www1: run command as root: /sbin/umount -t nullfs /mnt/disk1/j/10.0.3.1-www1/tmp
www1: run command as root: /sbin/umount -t nullfs /mnt/disk1/j/10.0.3.1-www1/var
www1: run command as root: /sbin/umount -t nullfs /mnt/disk1/j/10.0.3.1-www1/root
www1: run command as root: /sbin/umount -t nullfs /mnt/disk1/j/10.0.3.1-www1/usr/home
www1: run command as root: /sbin/umount -t nullfs /mnt/disk1/j/10.0.3.1-www1/usr/ports/portbuild
www1: run command as root: /sbin/umount -t nullfs /mnt/disk1/j/10.0.3.1-www1/usr/ports/distfiles
www1: run command as root: /sbin/umount -t nullfs /mnt/disk1/j/10.0.3.1-www1/usr/X11R6
www1: run command as root: /sbin/umount -t nullfs /mnt/disk1/j/10.0.3.1-www1/usr/local
www1: run command as root: /sbin/umount -t nullfs /mnt/disk1/j/10.0.3.1-www1/etc
www1: run command as root: /sbin/umount -t nullfs /mnt/disk1/j/10.0.3.1-www1
umount: unmount of /mnt/disk1/j/10.0.3.1-www1 failed: Device busy
jail: www1: /sbin/umount -t nullfs /mnt/disk1/j/10.0.3.1-www1: failed
www1: run command as root: /sbin/ifconfig ale0 inet 10.0.3.1 netmask 255.255.255.255 -alias
Everything appears to be unmounting in the proper order, but the root mount just refuses.
umount -f /mnt/disk1/j/10.0.3.1-www1
will remove the mount, and waiting about 2 minutes will allow me to do a nonforced umount. fstat() also shows nothing being accessed in the root directory.One jail that refuses to umount its root was upgraded from a FreeBSD 9.0 built world to a FreeBSD 10.0 built world and has gone through the complete upgrade process of using mergemaster() and upgrading all ports. Another jail was newly created using only a FreeBSD 10.0 built world without requiring any upgrades to ports or files since it was up-to-date already. Yet both jails fail to unmount their root mounts.
I'm at an extreme loss as to why this is happening and have used google and searched the forums but can only find people with the similar issues but all tend to be related to files being open and displayed when using fstat(), however in my case fstat() shows nothing even if told to show memory mapped files.
Edit: also here's the output of
uname -a
just to be clear to show the version, kernel, and architecture if that's relevant
Code:
FreeBSD host1.localdomain.local 10.0-RELEASE FreeBSD 10.0-RELEASE #0 r260789: Thu Jan 16 22:34:59 UTC 2014 root@snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64