NFS - reloading mountd disrupts in progress IO

This is on 12.0-RELEASE-p10+. Using TCP for all mounts. When I run service mountd reload (e.g. after editing /etc/exports) any in progress IO to the server gets completely disrupted. This really surprised me considering the exports() man page doesn't mention this at all which makes it seem like a non-service impacting thing to do:

Code:
     The mountd(8) utility can be made to re-read the exports file by sending
     it a hangup signal as follows:

           /etc/rc.d/mountd reload

     After sending the SIGHUP, check the syslogd(8) output to see whether
     mountd(8) logged any parsing errors in the exports file.

This is what happens when reading from a Linux client and running service mountd reload on the server:

Code:
# dd if=/mnt/testing/testfile-READ.mp4 of=/dev/null iflag=direct
dd: error reading '/mnt/testing/testfile-READ.mp4': Permission denied
10950+0 records in
10950+0 records out
5606400 bytes (5.6 MB, 5.3 MiB) copied, 1.40982 s, 4.0 MB/s

The process is immediately aborted with the above error. cp does the same thing. I'm using dd here to bypass the cache.

Here's what happens on write:

Code:
# cp testfile.mp4 /mnt/testing/testfile-WRITE.mp4
cp: failed to close '/mnt/testing/testfile-WRITE.mp4': Permission denied

Anyone else running an extremely busy NFS server? Do you wait until a maintenance window to modify exports and reload mountd?

I don't think we ever saw this when we were running Linux, but then again, it doesn't happen that often. It could have been doing the same thing and we never noticed.
 
Last edited by a moderator:
Hi,

I have confirmed this behaviour using FreeBSD 12.0-RELEASE NFS server (ZFS file system), and a Debian Linux 9.11 NFS client.
Code:
[orac.806] $ uname -a
Linux orac 4.9.0-11-amd64 #1 SMP Debian 4.9.189-3+deb9u1 (2019-09-20) x86_64 GNU/Linux

[orac.807] $ dd if=/work/junk of=/dev/null iflag=direct   
dd: error reading '/work/junk': Permission denied
5213+0 records in
5213+0 records out
2669056 bytes (2.7 MB, 2.5 MiB) copied, 1.22656 s, 2.2 MB/s

[orac.822] $ cp /work/junk /var/tmp
cp: error reading '/work/junk': Input/output error

[orac.825] $ mount | grep -i nfs | grep work
sherman10b:/tank/work on /work type nfs (rw,noatime,nodiratime,vers=3,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.0.1.1,mountvers=3,mountport=708,mountproto=tcp,local_lock=none,addr=10.0.1.1,_netdev)
This disturbed me too.
Changing the NFS mount from "hard" to "soft" does not change the outcomes.
 
This is probably something specific to NFSv3 (or FreeBSD 12.0 but I doubt that).

I do not see this behaviour with NFSv4.1 on FreeBSD 11.3-RELEASE-p5 nor 12.1-RELEASE-p1 (from a CentOS 7 client).

So probably something to do with the separate mount protocol in NFSv3 (and NFSv2) that isn't used in NFSv4.
 
You need to use the -S option with mountd. It really ought to be the default. In fact, it is set by default in /etc/defaults/rc.conf, but it's easy to overwrite that if you set mountd_flags in /etc/rc.conf.
 
You need to use the -S option with mountd. It really ought to be the default. In fact, it is set by default in /etc/defaults/rc.conf, but it's easy to overwrite that if you set mountd_flags in /etc/rc.conf.

Thank you for taking the time to post this. I can't believe I missed this. I will roll this out as soon as I get a chance.
 
Back
Top