Solved NFS server and user mapping

Hi all,

I am having a strange issue with a FreeBSD nfs server and CentOS nfs client.

The FreeBSD NFS server contains all data that are mounted by CentOS web servers.

NFS server Configuration:

Code:
nfs_server_enable="YES"
nfs_server_flags="-u -t -n 4"
nfsv4_server_enable="YES"
nfsuserd_enable="YES"
nfsuserd_flags="-domain mydomain.inside"
rpcbind_enable="YES"
mountd_enable="YES"
mountd_flags="-r"
/etc/exports:
Code:
/storage -alldirs -maproot=root -network 192.168.10.0/24
V4: /
NFS client Configuration /etc/idmapd.conf:

Code:
[General]
#Verbosity = 0
# The following should be set to the local NFSv4 domain name
# The default is the host's DNS domain name.
Domain = mydomain.inside

Now, on the client we mount everything under the /home directory. On a first look everything appears to be ok:

Code:
root@members1 home]# ls -alh
total 38K
drwxr-xr-x   5 nobody nobody    5 Apr 27  2014 .
dr-xr-xr-x. 23 root   root   4.0K Apr  3 07:22 ..
drwx------   2 admin  admin     7 Oct 20 07:38 admin
drwx------   2 admin  admin     2 Feb 19  2013 lost+found
drwxr-xr-x  27 admin  admin    32 Mar  2 11:26 sites

But for some reason the local user admin has no write access to those directories.

Also, directories that are owned by the local webserver user apache are not writable from the webserver.

If I create a file as the local user admin then the file has owner nobody and group admin when it should be admin:admin

So, for some reason, the group ownerships are honored while the user are not.

Could anyone shed some light on here?
 
Is the client definitely mounting the share using NFSV4?
I really need to do some testing with v4 as I find everything about it incredibly confusing at the moment, but the man page suggests that users are sent over the wire by name not UID. The server has a nfsuserd process which maps the username to ID, and it appears to use the local user database for this, which makes me think you need all the users on the client to exist on the server?

If this is the case (and a bit of a step back for me as I don't really want hundreds of user accounts on my NFS servers), then not having an admin user on the NFS server may end up with the server mapping your user to nobody.

It does look like you have everything configured reasonably and I can't see any other obviously reason why you should have this problem if you already have an admin user on the server.
 
Is the client mounting the share using NFSV4?
I really need to do some testing with v4 as I find everything about it incredibly confusing at the moment, but the man page suggests that users are sent over the wire by name not UID. The server has a nfsuserd process which maps the username to ID, and it appears to use the local user database for this, which makes me think you need all the users on the client to exist on the server?

Yes, NFSV4 is being used:
Code:
192.168.10.32:/storage/members_pw/ on /home type nfs (rw,vers=4,addr=192.168.10.32,clientaddr=192.168.10.6)
All users exist both on the NFS server and on the NFS clients. I also had the impression that users are sent by username and not by UID.

If this is the case (and a bit of a step back for me as I don't really want hundreds of user accounts on my NFS servers), then not having an admin user on the NFS server may end up with the server mapping your user to nobody.

That's not really an option here. Besides the admin user, we have some other users that need to exist. Also, the webserver user, apache, needs to be able to write on some directories. I don't want to make those directories world writable.
 
I'm confused by your last paragraph. I'm not suggesting removing accounts from the server or making things world writable. In my case I use v3, and rely on the fact that users can create files on my server, owned correctly by their uid, even though they don't have a matching account on the server. The server just shows the raw ID when viewing the files (as it obviously can't map the ID to a name), but everything works as expected. It appears I won't be able to do this with v4.

If you do have all the users on both client and server then I can't really see any reason you should be having the issue you are seeing. As I said I don't know much about v4 but your config all seems reasonable and it's obviously mounting without a problem and showing the correct UID/GID info on files/dirs that already exist.
 
I'm confused by your last paragraph. I'm not suggesting removing accounts from the server or making things world writable. In my case I use v3, and rely on the fact that users can create files on my server, owned correctly by their uid, even though they don't have a matching account on the server. The server just shows the raw ID when viewing the files (as it obviously can't map the ID to a name), but everything works as expected. It appears I won't be able to do this with v4.

The problem I had to face is this. The content of the webserver is over 20TB and growing. I decided to rsync the data to the storage, create the appropriate users and use NFSv4 to export the directories to the existing webserver(s). Now, if I use NFSv3 I just see the raw IDs and there is no way to assign the correct permissions again. I am not sure if that explains my issue...

If you do have all the users on both client and server then I can't really see any reason you should be having the issue you are seeing. As I said I don't know much about v4 but your config all seems reasonable and it's obviously mounting without a problem and showing the correct UID/GID info on files/dirs that already exist.

I know, that's why I am asking for help here!
 
Now it gets even more weird.... On the Linux NFS client:

Code:
[root@members2 home]# ll
total 34
drwx------  2 admin admin  7 Oct 20 07:38 admin
drwx------  2 admin admin  2 Feb 19  2013 lost+found
drwxr-xr-x 27 admin admin 32 Mar  2 11:26 sites
Code:
[root@members2 home]# ls -ln
total 34
drwx------  2 500 500  7 Oct 20 07:38 admin
drwx------  2 500 500  2 Feb 19  2013 lost+found
drwxr-xr-x 27 500 500 32 Mar  2 11:26 sites
[root@members2 home]# cat /etc/passwd | grep admin
Code:
admin:x:500:500::/home/admin:/bin/bash
[root@members2 home]# su - admin
Code:
su: warning: cannot change directory to /home/admin: Permission denied
-bash: /home/admin/.bash_profile: Permission denied
cd /home/sites/upload/temp
touch test
ll
Code:
-rw-rw-r-- 1 nobody admin 0 Apr  3 11:55 test
ls -ln
Code:
-rw-rw-r-- 1 99 500 0 Apr  3 11:55 test
 
There's a -verbose flag for nfsuserd so maybe that will log something useful on the server when you touch a file and it tries to map the username?
 
There's a -verbose flag for nfsuserd so maybe that will log something useful on the server when you touch a file and it tries to map the username?
Do you know if restarting the daemon might cause service interruption?
 
No idea. I assume the service will restart pretty quick but I don't know if requests that come in during that time will wait or fail.
 
No idea. I assume the service will restart pretty quick but I don't know if requests that come in during that time will wait or fail.
Then, I will wait till Monday because weekends are the busiest times for those websites.

EDIT I run it in verbose mode and nothing was logged. Any other ideas?
 
The -verbose flag should result in something like this in /var/log/messages when a client accesses a mount point.
Code:
Apr  4 01:15:21 <daemon.err> myhost nfsuserd:[68095]: Added uid=1000 name=jason
Apr  4 01:15:21 <daemon.err> myhost nfsuserd:[68066]: Added gid=1000 name=unovitch

Additionally, an item I've found helpful dealing with Linux clients is to enable the string to UID mapping. This only exists on 10.1 and up but allows things to work normally when the UID is sent across the wire rather than the name.
sysctl vfs.nfsd.enable_stringtouid=1
 
Apparently I needed to restart nfsuserd daemon for verbosity to work:

Code:
Apr  4 06:06:25 storage nfsuserd:[20995]: Added uid=0 name=root
Apr  4 06:06:25 storage nfsuserd:[20995]: Added gid=32767 name=nogroup
Apr  4 06:06:27 storage nfsuserd:[20998]: Added uid=48 name=nobody
Apr  4 06:06:30 storage nfsuserd:[20996]: Added gid=32767 name=nogroup
Apr  4 06:06:36 storage nfsuserd:[20997]: Added gid=32767 name=nogroup
Apr  4 06:06:39 storage nfsuserd:[20996]: Added uid=48 name=nobody
Apr  4 06:06:42 storage nfsuserd:[20998]: Added gid=32767 name=nogroup
Apr  4 06:06:44 storage nfsuserd:[20995]: Added uid=48 name=nobody
Apr  4 06:06:48 storage nfsuserd:[20997]: Added gid=32767 name=nogroup
Apr  4 06:06:52 storage nfsuserd:[20996]: Added uid=48 name=nobody
Apr  4 06:06:54 storage nfsuserd:[20997]: Added gid=32767 name=nogroup
Apr  4 06:07:00 storage nfsuserd:[20998]: Added gid=32767 name=nogroup
Apr  4 06:07:06 storage nfsuserd:[20995]: Added gid=32767 name=nogroup
Apr  4 06:07:12 storage nfsuserd:[20997]: Added gid=32767 name=nogroup
Apr  4 06:07:13 storage nfsuserd:[20996]: Added uid=48 name=nobody
Apr  4 06:07:18 storage nfsuserd:[20995]: Added gid=32767 name=nogroup
Apr  4 06:07:20 storage nfsuserd:[20996]: Added uid=48 name=nobody
Apr  4 06:07:24 storage nfsuserd:[20995]: Added gid=32767 name=nogroup
Apr  4 06:07:25 storage nfsuserd:[20997]: Added uid=0 name=root
Apr  4 06:07:25 storage nfsuserd:[20995]: Added uid=1002 name=admin
Apr  4 06:07:25 storage nfsuserd:[20998]: Added gid=1002 name=admin
Apr  4 06:07:26 storage nfsuserd:[20996]: Added uid=1004 name=ross
Apr  4 06:07:26 storage nfsuserd:[20995]: Added gid=1004 name=ross
Apr  4 06:07:30 storage nfsuserd:[20997]: Added gid=32767 name=nogroup
Apr  4 06:07:35 storage nfsuserd:[20998]: Added uid=1003 name=apache
Apr  4 06:07:35 storage nfsuserd:[20996]: Added gid=1003 name=apache
Apr  4 06:07:36 storage nfsuserd:[20995]: Added gid=32767 name=nogroup
Apr  4 06:07:36 storage nfsuserd:[20995]: Added gid=0 name=wheel
Apr  4 06:07:38 storage nfsuserd:[20998]: Added uid=48 name=nobody
Apr  4 06:07:42 storage nfsuserd:[20996]: Added gid=32767 name=nogroup
Apr  4 06:07:48 storage nfsuserd:[20997]: Added gid=32767 name=nogroup
Apr  4 06:07:54 storage nfsuserd:[20998]: Added gid=32767 name=nogroup
Apr  4 06:07:57 storage nfsuserd:[20995]: Added gid=65534 name=nobody
Apr  4 06:08:00 storage nfsuserd:[20996]: Added gid=32767 name=nogroup
Apr  4 06:08:01 storage nfsuserd:[20997]: Added uid=48 name=nobody
Apr  4 06:08:06 storage nfsuserd:[20995]: Added gid=32767 name=nogroup
Apr  4 06:08:12 storage nfsuserd:[20998]: Added gid=32767 name=nogroup
Apr  4 06:08:13 storage nfsuserd:[20996]: Added uid=48 name=nobody
Apr  4 06:08:18 storage nfsuserd:[20995]: Added gid=32767 name=nogroup
Apr  4 06:08:23 storage nfsuserd:[20997]: Added uid=48 name=nobody
Apr  4 06:08:24 storage nfsuserd:[20995]: Added gid=32767 name=nogroup
Apr  4 06:08:26 storage nfsuserd:[20998]: Added uid=1002 name=admin
Apr  4 06:08:26 storage nfsuserd:[20995]: Added gid=1002 name=admin
Apr  4 06:08:26 storage nfsuserd:[20996]: Added uid=0 name=root
Apr  4 06:08:26 storage nfsuserd:[20997]: Added uid=1004 name=ross
Apr  4 06:08:26 storage nfsuserd:[20995]: Added gid=1004 name=ross
Apr  4 06:08:30 storage nfsuserd:[20998]: Added gid=32767 name=nogroup
Apr  4 06:08:32 storage nfsuserd:[20997]: Added uid=48 name=nobody
Apr  4 06:08:33 storage nfsuserd:[20996]: Added uid=500 name=nobody
Apr  4 06:08:36 storage nfsuserd:[20995]: Added gid=32767 name=nogroup
Apr  4 06:08:37 storage nfsuserd:[20997]: Added uid=48 name=nobody
Apr  4 06:08:37 storage nfsuserd:[20998]: Added gid=1003 name=apache

This uid=48 name=nobody is the user admin in the Linux box. The same user uid=1002 name=admin exists in the FreeBSD server.

I also tried sysctl vfs.nfsd.enable_stringtouid=1 but it didn't make any difference.
 
I am not sure if this is the way that it should work. But if I change the UID & GID on the FreeBSD server to match the UID & GID of the client it appears that the problem goes away.

Of course, doing that for all users will require a lot of scripting. Find old UID & GID and replace it with the new one.
 
After seeing junovitch's response it makes quite a lot of sense. It appears some/all? Linux NFSv4 clients like sending uid instead of user@domain.

If Linux sends a UID, then the server will have to try and map that ID to a username, so as you have found, the UIDs will need to match at both ends.
 
Crap yes! That was it. Fortunately, all web servers had the same UIDs & GIDs for the users involved. Therefore, changing the UIDs & GIDs in the FreeBSD server appears to have solved the problem. Of course I had to replace all files as well, 20TB is a lot!

This method did the trick for each user: (old ID new ID)

find / -user 1002 -exec chown -h 500 {} \;
find / -group 1002 -exec chgrp -h 500 {} \;
 
Excellent, the sysctl vfs.nfsd.enable_stringtouid=1 I mentioned only helps in that case when the Linux client send the UID and they actually match on both sides. Unfortunately, that wasn't this case here so the fallback was the nobody user.
 
This smells like a bug though and it is difficult to troubleshoot. What we know so far is that the Linux client maps the correct user and group even if the ID's don't match. At least that's what it shows initially. However, since the ID's don't much it refuses to honor any sort of ownership. Having the same ID's solves the problem but it beats the purpose of using NFSv4.
 
It's hard to know exactly what is happening without the ability to test anything, but is it possible that the FreeBSD server is sending the username in read requests, which Linux is mapping to its local admin user, so you see everything as expected. However, when Linux is writing, it's sending its UID for admin instead of the username. FreeBSD can't map that to a user as the UID doesn't match any of its local users, so it using nobody, and giving you the permission issues and new file ownership of nobody that you are seeing.

I did have a quick look for an option to make Linux send username@domain instead, but couldn't really find anything. All I found what a thread mentioning it seems to have started sending UIDs instead of usernames around kernel 3.2. That thread did have a tuneable to make Linux use usernames but it seems it may be for the NFS server not client (unless the same setting affects both).
 
Thanks, I have spend 2 days trying to solve this. I am quite happy that at least now it works. I have also read tons of articles regarding NFSv4 and UIDs. I think the most relevant is this. Although I could not make much use of it.
 
It still sounds very haphazard and error prone to me. Either way you have a possibility of name collisions or uid collisions and there's no way to get everything automatically configured, you have to do manual work on both the server and the clients to ensure everything is in order. As much as I detest MS Windows they at least get the user identities right with domain users that are never ambiguous under any circumstances.
 
It still sounds very haphazard and error prone to me. Either way you have a possibility of name collisions or uid collisions and there's no way to get everything automatically configured, you have to do manual work on both the server and the clients to ensure everything is in order. As much as I detest MS Windows they at least get the user identities right with domain users that are never ambiguous under any circumstances.

It does and you need to be very careful when adding new users to the other boxes. They need to have the same UID apparently. But in all honesty, I never had this problem before in this environment. All systems are using CentOS and NFSv4 is used for many purposes. I just stumbled upon this now that I brought the FreeBSD storage in the picture.
 
I would be curious what actually goes over the wire. I'd bet there would be something interesting to see if you saved the capture with tcpdump(1). I took a quick look with an Ubuntu client and FreeBSD server and see my name in many of the FreeBSD -> Ubuntu packets but not in any of the Ubuntu -> FreeBSD packets. When I make a file on the NFS mount, Wireshark just shows that packet from the Ubuntu machine as having the owner field be a blob of data rather than a name. Pretty odd.
 
It sounds like a good idea. The Linux client apparently gets the correct name over the wire. That explains the fact that it shows the permissions to be correct. Now, when a user tries to access a file it gets very interesting because:
  1. The Linux client does not transmit the correct name or
  2. The FreeBSD server does not interpret correct the information that it receives.
So, access is denied and any file that gets created if a directory is world writable gets the ownership of nobody. However, the fact that when UIDs and GIDs much on both client and server everything works further complicates things.
 
After some research that I did, it looks like for mappings to work in NFSv4 you need kerberos.

References: nfsv4() and Ubuntu Community Wiki

NFSv4 uses names for users and groups instead of numbers. On the wire,
they take the form:

<user>@<dns.domain>

where ``<dns.domain>'' is not the same as the DNS domain used for host
name lookups, but is usually set to the same string. Most systems set
this ``<dns.domain>'' to the domain name part of the machine's
hostname(1) by default. However, this can normally be overridden by a
command line option or configuration file for the daemon used to do the
name<->number mapping. Under FreeBSD, the mapping daemon is called
nfsuserd(8) and has a command line option that overrides the domain com-
ponent of the machine's hostname. For use of NFSv4, either client or
server, this daemon must be running. If this ``<dns.domain>'' is not set
correctly or the daemon is not running, ``ls -l'' will typically report a
lot of ``nobody'' and ``nogroup'' ownerships.

Although uid/gid numbers are no longer used in the NFSv4 protocol, they
will still be in the RPC authentication fields when using AUTH_SYS
(sec=sys), which is the default.
As such, in this case both the
user/group name and number spaces must be consistent between the client
and server.

However, if you run NFSv4 with RPCSEC_GSS (sec=krb5, krb5i, krb5p), only
names and KerberosV tickets will go on the wire.

Following this guide will result in UID/GID on the export being generic despite having same UID on client and server. (According to my experience it works at least with "precise", if uid/gid are equal on both sides. hwehner) Mounting same share on NFSv3 works correctly with regards to UID/GID. Does this need Kerberos to work fully? According to http://arstechnica.com/civis/viewtopic.php?f=16&t=1128994 and http://opensolaris.org/jive/thread.jspa?threadID=68381you need to use Kerberos for the mapping to have any effect.
 
Back
Top