Solved NFSv3/NFSv4, mounting, permissions, NFSv4 man-page, and more

[mefizto]

Greetings all,

after discovering that the vaunted FreeBSD handbook is obsolete regarding NFS, I did some reading, and I though that I was able to figure out, how to set NFS sharing. To start, I enabled read and write access on the entire XXX.XXX.XXX.XXX/24 sub-net for a pool and a filesystem: zfs set sharenfs='rw=XXX.XXX.XXX.XXX' pool/filesystem
Then I mounted the filesystem: mount -t nfs XXX.XXX.XXX.XXX:/pool/filesystem /mnt
So far so good. However, ls -als /mnt results in:

ls: Directory_1: Permission denied
ls: File_1: Permission denied

I understand that this a user id (UID), group id (GUI) mapping problem. For example, I have user(s) with the same username but a different UID(s), GID(s) on different machines.

How is this mismatch solved in practice so that specific users can be given permissions to specific filesystem?

Kindest regards,

M

 

[gpw928]

NFS v3: fix your UIDs and GIDs so they are the same
NFS v4: you can map id's (v4 operates on names).

 

[Lamia]

You will include the permissions in the zfs set commands as shown here - https://klarasystems.com/articles/nfs-shares-with-zfs/.

/etc/exports works for non-zfs shares. I got the permissions into the zfs command for them to work.

 

[mefizto]

Hi gpw928,

thank you for the reply. Regarding the suggestion to "fix your UIDs and GIDs so they are the same", I had thought about that. However, I also thought that there must be a different way because although this can be done on a small scale, this is completely unfeasible for a company with thousands employees. And what about sharing a filesystem created by user_1 with user_2?

I will do the search for NFS v4 mapping, but if you have a good reference, could you point me to it?

Hi Lamia,

thank you for the reply. I did find and read the article. However, once I mount the NFS filesystem on a host machine, the user(s) on the host machine cannot access the filesystem. Am I missing a point in the article?

Kindest regards,

M

 

[SirDice]

However, I also thought that there must be a different way because although this can be done on a small scale, this is completely unfeasible for a company with thousands employees.

They typically use a centralized user management with LDAP or Active Directory. That ensures the UID/GID of a user is the same everywhere.

 

[mefizto]

Hi SirDice,

thank you for the reply. I have a recollection of reading somewhere that NFS v.4 does not use LDAP anymore.

On a different note, I tried mounting the filesystem with -o nfsv4 option as recommended by gpw928, and all the files and directories are mounted with user nobody and group nogroup, with all permissions set to (d)rwx regardless of the permissions on the original directories and/or files.

Also the ACL permissions are ignored, which is very confusing because my understanding was that the NFS ACLs were designed to be compatible with Windows ACLs.

What a confusing subject without a clear explanation - or, maybe better to say - I have not found one yet. As such, any reference would be appreciated.

Kindest regards,

M

 

[gpw928]

thank you for the reply. Regarding the suggestion to "fix your UIDs and GIDs so they are the same", I had thought about that. However, I also thought that there must be a different way because although this can be done on a small scale, this is completely unfeasible for a company with thousands employees.

There is no single solution with NFSv3. Point solutions depend on the scope of the problem. Sometimes you can issue new, unique, UIDs and/or GIDs. Other times you can add users to a new group, and rely solely on group permissions (see below).

And what about sharing a filesystem created by user_1 with user_2?

This is pretty much the same problem as with non-NFS file systems (without ACLs). The most common solution is to add all the users in a common group (i.e. same numeric GID), and use group permissions.

I will do the search for NFS v4 mapping, but if you have a good reference, could you point me to it?

Apart from nfsv4(4), I don't have anything you can't find with Google: freebsd nfsv4 howto. However, Rick Macklem's Plan B relating to parallel NFS seems to have some good insights regarding NFSv4 setup.

I tried mounting the filesystem with -o nfsv4 option as recommended by gpw928, and all the files and directories are mounted with user nobody and group nogroup, with all permissions set to (d)rwx regardless of the permissions on the original directories and/or files.

Running NFSv4 requires a fair bit of effort in planning, designing, and implementation. It really can't be hacked up quickly. For example, you must specifically facilitate the UID/GID mappings.The options are to run nfsuserd(8), or use sysctl settings found with sysctl -a | grep nfs | grep uid.

 

[Jose]

What a confusing subject without a clear explanation - or, maybe better to say - I have not found one yet. As such, any reference would be appreciated.

Unfortunately there is none that I'm aware of. Thread here:

Kerberized NFSv4 -> NFS over TLS on 13.0

Has anyone got a good reference for how to set up krb5p security? I've got NFSv4 running fine without security, but that kerberos setup has been a major pain in the neck. There doesn't seem to be much logging going on, no matter how much -d or -h's I use. It appears, Wireshark is my only utility...

forums.freebsd.org

Link from that thread:

https://people.freebsd.org/~rmacklem/nfs-krb5-setup.txt

 

[mefizto]

Hi gpw928,

thank you for the reply.

There is no single solution with NFSv3. Point solutions depend on the scope of the problem. Sometimes you can issue new, unique, UIDs and/or GIDs. Other times you can add users to a new group, and rely solely on group permissions (see below).

I think that I have several problems. First, some references suggest that the mapping resulting in nobody:nobody and ignoring of the ACL permission is due to mis-configured idmap(8). When I looked at the server's permissions, I have discovered some strange discrepancies, some directories/files had ownership user:user, but others had user:1004. Where the 1004 came from is a mystery, there is no trace of it in the /etc/password.

So I looked the id user and the result is:

uid=(user) gid=1001 groups=1001,0(wheel)

As I interpret it user does not belong to the primary group 1001. I looked at /etc/password, but it shows a correct uid (1001) and gid (1001) for the user. I am tying to fix it, but as I understand the pw(8), there does not seem to be an option to add the user to the 1001 primary group. I am getting response that the gid = 1001 does not exit. But, perhaps I am making some mistake, the pw(8) is rather dense reading. So, at this moment I am trying to solve this.

This is pretty much the same problem as with non-NFS file systems (without ACLs). The most common solution is to add all the users in a common group (i.e. same numeric GID), and use group permissions.

Indeed, but as I noted above, that solution will eventually became unmanageable. I have filesystems that I want to share with another user, filesystems that I want to share with the world, and if I make a group configuration mistake, some of the information might leak. Perhaps it might be necessary to set up some application managing LDAP.

Apart from nfsv4(4), I don't have anything you can't find with Google: freebsd nfsv4 howto. However, Rick Macklem's Plan B relating to parallel NFS seems to have some good insights regarding NFSv4 setup.

Thank you. The main problem that I have is that most of the guides are written for non-ZFS filesystems and they just list the commands to be issued, without any explanation what they do. But, I keep looking.

Running NFSv4 requires a fair bit of effort in planning, designing, and implementation. It really can't be hacked up quickly. For example, you must specifically facilitate the UID/GID mappings.The options are to run nfsuserd(8), or use sysctl settings found with

Ha, ha, you are absolutely correct. I had set up the file server in an ad hoc manner, but using SMB/CIFS everything seemd to work, partially thanks to the ACLs. Unfortunately, FreeBSD mount_smb(8) does not work for security reasons, and there does not seem to be an interest in fixing it. So, if I ever solve my user:primary_group issue, and make the NFS work on the current file server, to understan all the issues, I will re-organize the layout.

Hi Jose,

Unfortunately there is none that I'm aware of. Thread here:

Ha, ha, you are correct, see above. Thank you for the link, every little bit helps. I know a little bit more than when I started.

Kindest regards,

M

 

[Lamia]

In one of the instances I setup, the data on the nfs server is rsynced to a client (which is equally another server and exposed to the Internet).
There was little to wrong about w.r.t. security as one mentioned above that NFS cannot be easily hacked.

I can remember that I had issues with ACL for other users like you do. Maybe, I did not give it more attention. That is why I mentioned that /etc/exports works best for access control in non-zfs instances. I reckon root access works when set in the 'zfs set sharenfs......-maproot=root.....'.

Samba, on the other hand, works fine with no issues about ACL.

 

[mefizto]

Hi Lamia,

as some sources suggest, my issue with the ACL permission is mis-configuration. Since I was educated by gpw928 about the perceived problem with a user, I will continue with the NFS setup.

Kindest regards,

M

 

[Lamia]

In one of the clients here, I have:

nfs_client_enable="YES"
nfs_client_flags="-n 4"
nfsuserd_flags="-verbose"
rpc_lockd_enable="YES"
rpc_statd_enable="YES"
amd_enable="YES"
autofs_enable="YES"
#NFS needs the below else errors will be emitted on console; follow up here - forums.freebsd.org/threads/kerberized-nfsv4-nfs-over-tls-on-13-0.83484
tlsservd_enable="YES"
tlsclntd_enable="YES"

And for the server, it is:

rpcbind_enable="YES"
nfs_server_enable="YES"
nfs_server_flags="-u -t -n 4"
#nfs_reserved_port_only="YES"
mountd_enable="YES"
mountd_flags="-n -r -p 650"

You can do without some of the sysrc values in the client, such as amd and autofs.

 

[Phishfry]

3 Cheers for rickm@.

Somebody needs to make a pNFS wiki. That work needs exploitation.

 

[mefizto]

Hi Lamia,

thank you for the configuration.

Regarding the client, is is much simpler, based on the hanbook: https://docs.freebsd.org/en/books/handbook/network-servers/.

Kindest regards,

M

 

[mefizto]

Greetings all,

I have changed the uid/gid for two accounts and with the setting zfs set sharenfs='rw' pool/filesystem the UNIX permissions are correctly mapped, the ACLs are not mapped to the client side, but I will worry about it later.

When I try to limit the clients that can mount the share zfs set sharenfs='rw=XXX.XXX.XXX.111 pool/filesystem, I cannot mount the share receiving an error

RPCPROG_MNT: RPC: Timed out

Kindest regards,

M

 

[Lamia]

I cannot mount the share receiving an error

RPCPROG_MNT: RPC: Timed out

Kindest regards

It was one of the errors that I received. Chances are high that it is network (ports blocked for rpc services) related. There are a few ports that you need open - mountd, rpc, etc. I guess that is what is missing in my sent confs.
See https://forums.freebsd.org/threads/nfs-required-ports.17984/
and https://access.redhat.com/documenta...ministration_guide/s2-nfs-nfs-firewall-config.

 

[mefizto]

Hi Lamia,

thank you for the reply. I am not sure that this is the case.

First, if the entire sub-net IP space is allowed, there is no problem with the mount. So, why would restricting the sub-net space to a single IP address caused the problem?

Second, as none of the computers I am experimenting on are not facing Internet, I turned off firewall(s) for testing purposes, but this did not solve the problem.

I wonder if I am not encountering some incompatibility. For example I tried to use mount -o nfsv4acls per mount(8), but received back that the option is unknown. Can verbose login of the mount negotiation be logged as e.g., for ssh, there does not to be any such option in the mount(8).

Kindest regards,

M

 

[Lamia]

You will fix it; see https://forums.freebsd.org/search/274404/?q=RPCPROG_MNT:+RPC:+Timed+out&o=date.

Try insert the host IP and hostname of your workstation to the /etc/hosts file of the server.

You should lookup the internet too for other suggestions. This URL - https://www.truenas.com/community/threads/nfs-mount-failed-rpcprog_mnt-rpc-timed-out.5816/ - points back at this forum. Other platforms would have some other suggestions if the above fails to fix it.

 

[T-Daemon]

I wonder if I am not encountering some incompatibility. For example I tried to use mount -o nfsv4acls per mount(8), but received back that the option is unknown.

The command you are executing uses the NFSv3 protocol. To get nfsv4acls one must mount NFSv4 exported shares with the "nfsv4" option.

nfsv4(4)

DESCRIPTION
     The NFS client and server provides support for the NFSv4 specification;
     ...
     It provides several optional features not present in NFS Version 3:

           - NFS Version 4 ACLs
...
CLIENT MOUNTS
     To do an NFSv4 mount, specify the ``nfsv4'' option on the mount_nfs(8)
     command line.

And receiving back the nfsv4acls mount option as unknown (in case of the NFSv3 protocol, or -o nfsv4,nfsv4acls options) looks inappropriate as a error message and could be misleading. The error message prints also "Invalid argument". I suspect nfsv4acls are enabled by default on the FreeBSD ZFS server and being mounted by default with nfsv4acls on the client (when using -o nfsv4) and because of that it's rejected as mount option. If my suspicion is correct the error message should be revised to indicate the real problem:

zfsprops(8)

 acltype=off|nfsv4|posix
          ...
          nfsv4     default on FreeBSD, indicates that NFSv4-style ZFS ACLs
                 should be used.  These ACLs can be managed with the
                 getfacl(1) and setfacl(1) commands on FreeBSD. ...

Default Root-on-ZFS installation on a NFS server:

# zfs get acltype -r zroot
NAME                     PROPERTY  VALUE     SOURCE
zroot               acltype   nfsv4     default
zroot/ROOT          acltype   nfsv4     default
zroot/ROOT/default  acltype   nfsv4     default
zroot/sharenfs      acltype   nfsv4     default
zroot/tmp           acltype   nfsv4     default
zroot/usr           acltype   nfsv4     default
zroot/usr/home      acltype   nfsv4     default
zroot/usr/ports     acltype   nfsv4     default
zroot/usr/src       acltype   nfsv4     default
zroot/var           acltype   nfsv4     default
zroot/var/audit     acltype   nfsv4     default
zroot/var/crash     acltype   nfsv4     default
zroot/var/log       acltype   nfsv4     default
zroot/var/mail      acltype   nfsv4     default
zroot/var/tmp       acltype   nfsv4     default

After mount -o nfsv4 ... an NFSv4 exported share (/sharenfs/share1, with V4:/sharenfs root) on a FreeBSD client:

# mount
192.168.0.15:/share1 on /mnt (nfs, nfsv4acls)

 

[mefizto]

Hi Lamia,

I really appreciate your willingness to help.

Try insert the host IP and hostname of your workstation to the /etc/hosts file of the server.

Ha, ha, I have already searched and found the proposed solution. Although I did not understand it because (i) when I specified the entire sub-net, I can mount the shared data-set on two of my testing computers, and (ii) I am designating the server by IP address, so there should be no issue with name resolution, I still did it. Did not help at all.

Thank you for the other link.

Hi T-Daemon

thank you for the reply.

The command you are executing uses the NFSv3 protocol. To get nfsv4acls one must mount NFSv4 exported shares with the "nfsv4" option.

Ah, I will try that. I was under the impression that 13.1-RELEASE defaults to the NFSv4 protocol. That understanding was confirmed when I was trying to find a solution, cf. https://forums.freebsd.org/threads/cant-mount-pi4s-neither-samba-nor-nfs-shares.82669/#post-539668, post 7.

Regarding your reference to zfsprops(8), that is one problem I have, the settings are all over the place. As I wrote, I found the -o nfsv4acls in mount(8).

Default Root-on-ZFS installation on a NFS server:

I do not understand, what you are trying to communicate. I do not have a dedicated NFS server, I have a generic installation, on which one of the data-set has been exported/shared via SMB with a Windows machine. But I want to share the data-set also with a FreeBSD machine, hence I set the sharenfs property to it.

Could you please explain what was the import of you posting the NFS server structure?

Kindest regards,

M

 

[mefizto]

Hi T-Daemon,

just wanted to let you know that your advice to invoke the -o nfsv4 fixed the pesky

RPCPROG_MNT: RPC: Timed out

error, and replaced it with

mount_nfs: nmount: /mnt: Permission denied

I will try to figure it out; just to reiterate my steps, in case someone spots whether I am making any other moronic error:

The server has an IP address XXX.XXX.XXX.111. To allow mounting only on a client with IP address XXX.XXX.XXX.110, I set: zfs set sharenfs='rw=XXX.XXX.XXX.110' pool/filesystem on the server, and verify that the property was set: zfs get sharenfs pool/filesystem

Then on the client I issue (as a root): mount -t nfs -o nfsv4 XXX.XXX.XXX.111:/pool/filesystem /mnt, and promptly receive the aforementioned error.

Kindest regards,

M

 

[Lamia]

mount_nfs: nmount: /mnt: Permission denied

I would go back to check if it has something to do with maproot in the 'zfs set sharenfs...-maproot' command. And then that you are using nfsv4, it should be like this:

# setfacl -m u:thorsten:rwxp::allow bla

See https://forums.freebsd.org/threads/nfsv4-setfacl-does-not-work.16169/

 

[mefizto]

Hi Lamia,

thank you again for your points.

I do not understand the first one, with the -maproot command. I might have missed such a setting in any of the posts/tutorials I looked at, and they claimed that the setup works, but I will now learn what it does. Your second point, I believe is that I might be using POSIX instead of NFS ACLs. I do not believe that being the case because they are set on the filesystem that I am using to share via SMB/CIFS with the Windows machine and were; thus. likely set by the Windows machine. And the NFS and Windows ACLs are supposed to be compatible.

I am, though, rather discouraged because I have (i) changed the UIDs/GIDs of all three accounts to be the same, (ii) changed the domain on one of the machines to agree with the other two, (iii) used the -o nfsv4 option as recommended by T-Daemon, (iv) set zfs set sharenfs=rw pool/filesystem to allow mounting at both clients, and was again rewarded with permission of type nobody:nobody.

On a top of that, when I add the option -o nfsv4acls, it again reports illegal option.

Maybe I am just too stupid to make it work.

Kindest regards,

M

 

[Lamia]

mefizto:

You will include the permissions in the zfs set commands as shown here - https://klarasystems.com/articles/nfs-shares-with-zfs/.

Quoting a line in the page, "If you see entries with no maproot setting in your own configuration, you should evaluate if you need to add restrictions to that filesystem’s sharenfs property."

An example with maproot, in the klarasystems link, is:
zfs set sharenfs=’on,-maproot=root,192.168.15.0/24’ mypool/usr/ports

Interestingly, I use nfsv4 too (going by zfs get acltype) and cannot remember inserting the argument '-o nfsv4' in mount or so.

On NFS server is:

 zfs get sharenfs | grep on
.....
......
......
zdata/poudriere/jails/131StaAmd64-default-workstation-ref                  sharenfs  -maproot=0,192.168.1.12,-alldirs,ro      inherited from zdata/poudriere
.....
......

Take note of the additional permissions "-alldirs,ro". And I mount very easily on a client with:
/sbin/mount 192.168.1.19:/zdata/poudriere/ /zremote/poudriereshared

 

[T-Daemon]

The command you are executing uses the NFSv3 protocol. To get nfsv4acls one must mount NFSv4 exported shares with the "nfsv4" option.

Ah, I will try that. I was under the impression that 13.1-RELEASE defaults to the NFSv4 protocol. That understanding was confirmed when I was trying to find a solution, cf. https://forums.freebsd.org/threads/cant-mount-pi4s-neither-samba-nor-nfs-shares.82669/#post-539668, post 7.

Not sure where SirDice has the information that NFSv4 is default, but according to nfsv4(4) and on all my systems some configuration must be set to enable the NFSv4 protocol. Those configurations are not set on a default system installation. Please see the answer in my next posting for those configurations.

To check which NFS protocol is used execute on the client side after mounting the NFS share nfsstat -m.

Regarding your reference to zfsprops(8), that is one problem I have, the settings are all over the place. As I wrote, I found the -o nfsv4acls in mount(8).

That option is not a valid mount_nfs(8) option but to mount file systems which have not nfsv4acls enabled by default, for example UFS2.

Assuming /dev/ada1p1 is a UFS2 file system:

# tunefs -p /dev/ada1p1
tunefs: POSIX.1e ACLs: (-a)                      disabled
tunefs: NFSv4 ACLs: (-N)                         disabled

# mount /dev/ada1p1 /mnt
# mount | grep /mnt
/dev/ada1p1 on /mnt (ufs, local, soft-updates, journales soft-updates)

# umount /mnt

# mount /dev/ada1p1 -o nfsv4acls /mnt
/dev/ada1p1 on /mnt (ufs, local, soft-updates, journales soft-updates, nfsv4acls)

Or a /etc/fstab entry:

# Device        Mountpoint    FStype    Options        Dump    Pass#
/dev/ada1p1     /UFS2       ufs     rw,nfsv4acls  1       1

Of course one can enable nfsv4acls on the file system directly (tunefs(8) -N enable).

Default Root-on-ZFS installation on a NFS server:

I do not understand, what you are trying to communicate. I do not have a dedicated NFS server, I have a generic installation, on which one of the data-set has been exported/shared via SMB with a Windows machine. But I want to share the data-set also with a FreeBSD machine, hence I set the sharenfs property to it.

That "sharenfs" property and NFS options set in /etc/rc.conf makes it a NFS server.