ZFS weird NFS behavior, permission denied while writiing in the midst

[belon_cfy]

Hi

I'm having the following NFS problem, the mount point is writable but it will prompt permission denied in the midst of reading/writing, but no issue on writing dd locally .

The system is FreeBSD 10.2-RELEASE-p24

root@storage:/mnt # mount -t nfs localhost:/vol/test /mnt/test/
root@storage:/mnt # cd test/
root@storage:/mnt/test # dd if=/dev/zero of=test bs=1M
dd: test: Permission denied
9138+0 records in
9137+0 records out
9580838912 bytes transferred in 7.471400 secs (1282335156 bytes/sec)
root@storage:/mnt/test # ls -al
total 2
drwxr-xr-x  2 root  wheel           3 Dec 13 11:33 .
drwxr-xr-x  4 root  wheel           4 Dec 13 10:58 ..
-rw-r--r--  1 root  wheel  9580838912 Dec 13 11:42 test

my mount point has configured as below:

NAME      PROPERTY  VALUE                   SOURCE
vol/test  sharenfs  -maproot=root -alldirs  local

Below is my NFS setting

# NFS
nfsv4_server_enable="YES"
rpcbind_enable="YES"
nfs_server_enable="YES"
nfs_server_flags="-u -t -n 16"
mountd_flags="-r"
rpc_lockd_enable="YES"
rpc_statd_enable="YES"

 

[SirDice]

Is the disk full? A quota perhaps? It stops at around 9.5 GB, so you may be hitting a limit on the server. Which will then simply refuse to write resulting in a permission denied.

 

[belon_cfy]

Is the disk full? A quota perhaps? It stops at around 9.5 GB, so you may be hitting a limit on the server. Which will then simply refuse to write resulting in a permission denied.

Still have 3.5TB left which is 70% full, no quota on that volume. 9.5GB just the data been written during this test, sometime it can write 20GB, and sometime less than 5GB .

DD read also having the same issue, it just dead randomly.

 

[Deleted member 45312]

I am having the same issue with my home server when heavily loaded, like building packages with ports-mgmt/poudriere.
It looks like NFS timeout, but I don't know how to increase timeout on NFS share mount. I tried to increase retrans without success (perhaps a bit better).
All is working flawlessly when not too loaded.

 

[belon_cfy]

The problem occur when I'm running zfs sync over ssh to this server, but through another network interface.

Increasing nfs timeout doesn't help.

No idea why it happens on NFS only , possible the ssh using the port that nfs portmap are using?

 

[belon_cfy]

I had found the root cause, it was due to the sharenfs was enabled on the destination zfs filesystem.

I suspect the zfs will re-generate the /etc/zfs/export and restart the nfs service everytime when the zfs receive has been completed. During that time the mounted host will become unreadable due to nfs permission setting was disappear. Although restarting take less that a sec however the permission setting on all the running nfs mount point will be affected.

The symptom is similar to when you off and on the nfs with the following command while reading or writing the file
zfs set sharenfs="off" vol/test ; zfs set sharenfs="-maproot=root -alldirs" vol/test

To avoid this issue, simply off the nfs setting on those filesystems pending for receiving snapshop update from zfs send.

Is there anyway to avoid regen the nfs export setting or restart after zfs receive when the setting are same as previous?

 

[kpa]

I wouldn't use sharenfs on FreeBSD because it doesn't do anything you couldn't do without it unlike on Solaris where it actually interracts with a kernel level service to set up or tear down the sharing. The standard exports(5) file does the job just as well.