How to NFS share whole zfs pool instead of individual dataset?

jazzi · Mar 27, 2024

Hi,

Under my zfs pool named "data" I have several datasets like:

data/book

data/music

On the server side, I run the following command:

Code:

zfs set sharenfs="network=192.168.31.0/24" data

On the client side, it can list the directory but can not see the content of book or music.

Once I changed to share dataset separately like

Code:

zfs set sharenfs="network=192.168.31.0/24" data/book

, then the client can see the content.

My question is, is there a way to share the WHOLE pool?

T-Daemon · Mar 27, 2024

jazzi said:
On the client side, it can list the directory but can not see the content of book or music.

This is related to NFSv3. I haven't figured out why this is happening.

jazzi said:
My question is, is there a way to share the WHOLE pool?

Use NFSv4 instead.

mer · Mar 27, 2024

I think for NFSv3 the sharenfs is per dataset. Perhaps it can be set on the pool and inherited by the datasets,

jazzi · Mar 27, 2024

Thank you T-Daemon and mer
Already put the following line into /etc/rc.conf, is it possible to get it work without reboot?

nfsv4_server_enable="YES"

zirias@ · Mar 27, 2024

from nfsv4(4):

Unlike NFS Version 2 and 3, Version 4 allows a client mount to span across multiple server file systems, although not all clients are capable of doing this.

So, you can always only export one filesystem at a time, the difference with NFSv4 is that clients can do mounts spanning across multiple server filesystems.

Yes, use NFSv4. There's no real reason for v3 any more anyways. With v4, many things get a lot simpler (all functions in a single protocol on a single TCP port, no need for RPC portmapper any more), and you just need to have nfsuserd running because it uses actual names instead of uids.

T-Daemon · Mar 27, 2024

jazzi said:
is it possible to get it work without reboot?

Run service nfsd restart.

zirias@ · Mar 27, 2024

But remember you will need to enable nfsuserd as well, and you will need to add a NFSv4 root to your /etc/exports (not sure about ZFS properties, I still prefer to configure all exports outside of ZFS features ...)

Also note you can add nfsv4_server_only="YES", then you don't need to run rpcbind any more.

jazzi · Mar 28, 2024

$ cat /etc/rc.conf

# Enable NFS service
mountd_enable="YES"
mountd_flags="-r -n"
nfs_server_enable="YES"
nfs_server_flags="-u -t -n 4"
nfsuserd_enable="YES" # needed for NFSv4
nfsuserd_flags="-verbose"
nfsv4_server_enable="YES"
nfsv4_server_only="YES"
rpcbind_enable="YES" # not needed for NFSv4
rpc_lockd_enable="YES" # for MacOS client
rpc_statd_enable="YES" # for MacOS client

Above is my /etc/rc.conf, restarted all services, also double checked all services are running by command `service name status`.
Still no luck, try to connect with following command from MacOS Ventura:

Code:

sudo mount -t nfs -o vers=4 192.168.31.240:/data    /nfs

mount_nfs: can't mount /data from 192.168.31.240 onto /nfs: RPC prog. not avail
mount: /nfs failed with 74

jazzi · Mar 28, 2024

Changed the syntax of command zfs set sharenfs and above "RPC prog not avail" error is gone.

Code:

zfs set sharenfs='-network=192.168.31.0/24' data

I added a - before network.

Even stopped service rpcbind, rpc_lockd and rpc_statd and no errors any more.

But still the client can not list the content even now NFSv4 is used. Hereby is the nfs status on client after mounted:

$ nfsstat -m

/nfs from 192.168.31.240:/data
-- Original mount options:
General mount flags: 0x0
NFS parameters: vers=4
File system locations:
/data @ 192.168.31.240 (192.168.31.240)
-- Current mount parameters:
General mount flags: 0x4000000 multilabel
NFS parameters: vers=4.0,tcp,port=2049,hard,nointr,noresvport,callback,negnamecache,nonamedattr,noacl,noaclonly,locks,quota,rsize=32768,wsize=32768,readahead=16,dsize=32768,rdirplus,nodumbtimer,timeo=10,maxgroups=16,acregmin=5,acregmax=60,acdirmin=5,acdirmax=60,nomutejukebox,noephemeral,nonfc,sec=sys

File system locations:
/data @ 192.168.31.240 (192.168.31.240)
Status flags: 0x0

jazzi · Mar 28, 2024

Maybe I found the problem.

After run the following command in server:

Code:

zfs set sharenfs='-mapall=1000,-network=192.168.31.0/24' data

I mounted it on MacOS client side with following command:

Code:

sudo mount -t nfs -o vers=4 192.168.31.240:/data    /nfs

Then on the MacOS client terminal, I tried to list the content:

Code:

$ ls -l  /nfs
total 262
drwxrwxr-x   6 1000  1000  22  2 29 09:16 book
drwxrwxr-x   2 1000  1000   6  3 14 21:37 download
drwxrwxr-x  33 1000  1000  35  3  5 13:48 movie

Then, tried to list the content of directory book which is not shown in MacOS Finder

Code:

$ ls -l /nfs/book
total 0
ls: /Users/jazzi/nfs/book: Operation not permitted

So, it might be permission problem, try again with sudo

Code:

$ sudo ls -l /nfs/book
total 567328
drwxr-xr-x  2 1000  1000          4 11 18 19:46 Python
drwxrwxr-x  2 1000  1000          7  2 19 16:17 Slackware

Seem found the answer, but wait a minute, I tried the following command on Server side with option -mapall=root, but didn't work, common user on client side still can not list the content.

Code:

zfs set sharenfs='-mapall=root,-network=192.168.31.0/24' data

By the way, both client and server has the same user name and uid=1000. Hereby is the permission status on server side:

Code:

$ ls -l   /
drwxr-xr-x   8 root wheel    8 Mar  2 10:50 data

$ ls -l. /data
drwxrwxr-x   6 jazzi jazzi 22 Feb 29 09:16 book
drwxrwxr-x   2 jazzi jazzi  6 Mar 14 21:37 download

I am confused now.

jazzi · Mar 28, 2024

The _more_ weird part is what I just tried after posted above post, that is once ever used sudo to list the directory, the content will be accessible for normal user after that.

Code:

sudo ls -l /nfs/book

Then it will be no problem to use it without sudo.

The _more_more_ weird part is next, it will become inaccessible again after about two minutes.

And I get some info before it becomes inaccessible:

Code:

% ls -l ~/nfs/download

total 240328706
-rw-r--r--  1 root  1000  61524148224  3 14 21:34 freebsd-usb-ddrescue.img
-rw-r--r--  1 root  1000  61524148224  3 14 20:32 freebsd-usb.img
-rw-r--r--  1 root  1000          368  3 14 21:34 usb.logfile
-rw-r--r--  1 root  1000          433  3 14 21:29 usb.logfile.bak

Code:

ls -l /nfs/
      
total 262
drwxrwxr-x   6 1000  1000  22  2 29 09:16 book
drwxrwxr-x   2 1000  1000   6  3 14 21:37 download

What evil is inside?

zirias@ · Mar 28, 2024

Two things:

* "RPC prog not avail" is something you should NEVER see with NFSv4, it doesn't need any of this RPC mapper services. Also lockd and statd are integrated into nfsd with NFSv4.

* Regarding the permission error, I can only assume that the client transparently mounts child filesystems when using a mount spanning multiple filesystems and for some reason, only a privileged user can do this on your client.

jazzi · Mar 30, 2024

Surprisingly it works perfect on an old MacBook Air Big Sur as client, hereby is the status:

Code:

Big Sur@% nfsstat -m

/nfs from 192.168.31.240:/data
  -- Original mount options:
     General mount flags: 0x0
     NFS parameters: vers=4
     File system locations:
       /data @ 192.168.31.240 (192.168.31.240)
  -- Current mount parameters:
     General mount flags: 0x4000000 multilabel
     NFS parameters: vers=4.0,tcp,port=2049,hard,nointr,noresvport,callback,negnamecache,nonamedattr,noacl,noaclonly,locks,quota,rsize=32768,wsize=32768,readahead=16,dsize=32768,rdirplus,nodumbtimr,timeo=10,maxgroups=16,acregmin=5,acregmax=60,acdirmin=5,acdirmax=60,nomutejukebox,noephemeral,nonfc,sec=sys

     File system locations:
       /data @ 192.168.31.240 (192.168.31.240)
     Status flags: 0x0

What's wrong with the new MacOS Sonoma 14.4.1?