Solved "cannot unmount" messages appear during the shutdown after upgrade from 12.2-p12 to 12.3-p1

[Kaminar]

I upgraded from FreeBSD 12.2-RELEASE-p12 to 12.3-RELEASE-p1. On the new FreeBSD 12.3-RELEASE-p1, "unmount" messages appear during the shutdown sequence:

Writing entropy file: .
Writing early boot entropy file: .
cannot unmount '/var/log': Device busy
cannot unmount '/tmp': Device busy
.
Terminated

I use default ZFS layout without anything special.

I have never seen these messages during shutdown in any previous version of FreeBSD. Is there something wrong with the upgrade?

 

[SirDice]

Something's keeping files open on those filesystem which prevents them from being unmounted. Not common, so you're going to have to do some investigating.

 

[Cath O'Deray]

If you shutdown now, then browse the file system in single user mode, do the directory contents offer any clue?

If you start the operating system in safe mode, does the issue occur at shutdown time?

 

[Kaminar]

I have been using 12.3 for only a few days. But yesterday I noticed one clean shutdown without these issues. I'll try to investigate it more and I'll let you know.

 

[SirDice]

Check for services (daemons) that are running and perhaps refuse to stop when you shutdown. That could keep files open on those filesystems.

 

[Kaminar]

If you shutdown now, then browse the file system in single user mode, do the directory contents offer any clue?

I didn't see anything strange.

After shutdown now:

...
Stopping devd.
Writing entropy file: .
Writing early boot entropy file: .
cannot unmount '/var/log': Device busy
Terminated
.
Feb  4 17:40:07 ... syslogd: exiting on signal 15
Enter root password, or ^D to go multi-user
...

If you start the operating system in safe mode, does the issue occur at shutdown time?

Yes, it also occurs in safe mode.

I got rid of everything from /boot/loader.conf and /etc/rc.conf, what it is really not needed. Only syslogd had files open in /var/log filesystem.

# fstat -f /var/log
USER     CMD          PID   FD MOUNT      INUM MODE         SZ|DV R/W
root     syslogd      670   14 /var/log     10 -rw-r--r--  382445  w
root     syslogd      670   15 /var/log     12 -rw-------      62  w
root     syslogd      670   16 /var/log      6 -rw-------   21353  w
root     syslogd      670   17 /var/log      9 -rw-r-----      62  w
root     syslogd      670   18 /var/log      7 -rw-------      62  w
root     syslogd      670   19 /var/log      8 -rw-------     283  w
root     syslogd      670   21 /var/log     13 -rw-------      62  w
root     syslogd      670   22 /var/log     14 -rw-r--r--      62  w
root     syslogd      670   23 /var/log     15 -rw-r-----      62  w

# fstat -f /tmp
USER     CMD          PID   FD MOUNT      INUM MODE         SZ|DV R/W

If I kill syslogd, the issue will not occur during the shutdown. I think the problem is syslogd.

 

[SirDice]

I think the problem is syslogd.

Looks that way. It should stop before any filesystems are unmounted, certainly before /var/log can be unmounted. Now we need to figure out why it fails to stop gracefully. With the system booted normally can you easily stop/start the service? service syslogd stop, service syslogd start. Also check the number of syslogd(8) processes, there should only be one.

It's either an order issue (/var/log is unmounted before syslogd(8) is stopped), or syslogd(8) doesn't want to stop gracefully and eventually gets killed when the rug gets pulled out from under it.

Oh, double check the files /etc/syslog.conf and under /etc/syslog.d/, maybe there are some merge issues left over. I can't remember if there were any changes there between 12.2 and 12.3 but it's good to rule out configuration issues.

 

[Kaminar]

This problem is caused not only by syslogd. I found that everything that has opened the files in /var/log must be stopped before the shutdown.

I created a fresh installation of FreeBSD 12.3 in VirtualBox. The same problem occurs.

I found a difference in the /etc/rc.d/zfs file from 12.2-RELEASE and 12.3-RELEASE. In the FreeBSD 12.3-RELEASE-p1 is added line:

# KEYWORD: shutdown

If I deleted this line from file from 12.3 version, the issue disappeared.

Out of curiosity, I tried to add this line to a fresh installation of FreeBSD 13.0 in VirtualBox. The same problem occurs after adding this line to /etc/rc.d/zfs.

 

[Kaminar]

The problem does not occur in FreeBSD 13.1-RELEASE.