NFS doesn't register with rpcbind

Hi all,

I have longstanding issue with our NFS server, going probably from 10.x release times. Also problematic counterpart (client) has undergone multiple upgrades, but problem persists.

Quick intro - NFS server on FreeBSD 12.2 (running there probably for about 3-4 years, hence mentioning 10.x release), couple of clients, mostly Linux. The problematic one is xen on two servers in pool. Now running latest xcp-ng 8.2 (opensource fork of the Citrix product after their licensing changes, based on CentOS 7), also with history of upgrades over the years.

The problem is I can mount and use NFS shares from any client, even those xcp-ng servers, but I can't add what xen calls NFS storage repository, which is basically only fancy name for NFS mount with info stored in xen internal configuration database. There are three ways to do it (known to me), xe command line utility, web tool from authors of the mentioned fork named Xen Orchestra and Windows application, which is remnant of the Citrix era called XenCenter. All three use (AFAIK) some Python scripting on the server (NFS client in this case) to do it.

It would be easy to blame this Python part of the problem, but I noticed, that on the FreeBSD side nfsd does not register service with the rpcbind, so I think this may be (at least part of) the problem. I read somewhere, that NFSv4 can work without RPC and in fact, it does for me, at least partially, but I am not sure what specification says and which part has to be blamed here.

So my questions are:
1. Why my nfsd doesn't register with rpcbind?
2. Is this registration somewhat optional at least for NFSv4?
3. How can I get some useful debug info? Using -d options in our servers startup configuration where it is available doesn't produce much. Should I watch exchanged packets with tcpdump/Wireshark or is there some better way?

In the following snippets, 172.22.1.4 is the NFS server and 172.22.1.7 and 172.22.1.27 are the troublesome clients.

From the server
Code:
uname -a
FreeBSD storage-smc.ujezd.net 12.2-RELEASE-p1 FreeBSD 12.2-RELEASE-p1 GENERIC  amd64

/etc/rc.conf
Code:
hostname="storage-smc.ujezd.net"
cloned_interfaces="vlan1000 vlan1500 vlan2000"
ifconfig_vlan1000="inet 172.22.1.4 netmask 255.255.0.0 vlan 1000 vlandev igb1"
ifconfig_vlan2000="inet 10.128.99.4 netmask 255.255.255.0 vlan 2000 vlandev igb1"
ifconfig_vlan1500="inet 192.168.222.1 netmask 255.255.255.0 vlan 1500 vlandev igb1"
ifconfig_igb1="up"
nfsuserd_flags="-verbose"
defaultrouter="172.22.0.1"
sshd_enable="YES"
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="AUTO"
zfs_enable="YES"
nfsv4_server_enable="YES"
nfsuserd_enable="YES"
rpcbind_enable="YES"
nfs_server_enable="YES"
#nfs_server_flags="-h 172.22.1.4"
mountd_enable="YES"
mountd_flags="-r"
kld_list="aesni ipmi"
smartd_enable="YES"
ntpdate_hosts="10.128.99.95"
ntpdate_enable="YES"
ntpd_sync_on_start="YES"
ntpd_enable="YES"
netdata_enable="YES"
microcode_update_enable="YES"
rpc_lockd_enable="YES"           # Run NFS rpc.lockd needed for client/server.
rpc_statd_enable="YES"           # Run NFS rpc.statd needed for client/server.

/etc/sysctl.conf
Code:
vfs.zfs.arc_max="48000000000"
vfs.nfsd.server_min_nfsvers=4
vfs.nfs.enable_uidtostring=1
vfs.nfsd.tcpcachetimeo: 300
vfs.nfsd.tcphighwater: 100000

/etc/exports
Code:
/zdata/email -maproot=root 10.128.99.79
/zdata/nf -maproot=root 172.22.255.249
/zdata/odkladgalery -maproot=root 172.22.1.10
/zdata/ios -maproot=root 172.22.1.14
/zdata/xen-iso-library -maproot=root 172.22.1.27
/zdata/xen-iso-library -maproot=root 172.22.1.7
/zdata/xen-nfs-storage -maproot=root 172.22.1.27
/zdata/xen-nfs-storage -maproot=root 172.22.1.7
/zdata/servers/xenserver -maproot=root 172.22.1.27
/zdata/servers/xenserver -maproot=root 172.22.1.7
/zdata/servers/xenserver -maproot=root 172.22.1.32
/zdata/virt -maproot=root 172.22.1.27
/zdata/virt -maproot=root 172.22.1.7
/zdata/virt -maproot=root 172.22.1.13
/zdata/virt -maproot=root 172.22.1.2
/zdata/virt -maproot=root -sec=sys 172.22.1.11
V4: / -sec=sys -network 172.22.0.0 -mask 255.255.0.0
V4: / -sec=sys -network 10.128.99.0 -mask 255.255.255.0

Code:
rpcinfo -s
   program version(s) netid(s)                         service     owner
    100000  2,3,4     local,udp6,tcp6,udp,tcp          rpcbind     superuser
    100024  1         tcp,udp,tcp6,udp6                status      superuser
    100021  4,3,1,0   tcp,udp,tcp6,udp6                nlockmgr    superuser
    100005  3,1       tcp,udp,tcp6,udp6                mountd      superuser

And one of the Linux clients - rpcinfo doesn't show nfsd, but mounts can be probed and mounted without problem.

Code:
showmount -e 172.22.1.4
Export list for 172.22.1.4:
/zdata/virt              172.22.1.11,172.22.1.2,172.22.1.13,172.22.1.7,172.22.1.27
/zdata/xen-nfs-storage   172.22.1.7,172.22.1.27
/zdata/xen-iso-library   172.22.1.7,172.22.1.27
/zdata/odkladgalery      172.22.1.10
/zdata/email             10.128.99.79
/zdata/servers/xenserver 172.22.1.32,172.22.1.7,172.22.1.27
/zdata/ios               172.22.1.14
/zdata/nf                172.22.255.249

rpcinfo -s 172.22.1.4
   program version(s) netid(s)                         service     owner
    100000  2,3,4     local,udp6,tcp6,udp,tcp          portmapper  superuser
    100024  1         tcp,udp,tcp6,udp6                status      superuser
    100021  4,3,1,0   tcp,udp,tcp6,udp6                nlockmgr    superuser
    100005  3,1       tcp,udp,tcp6,udp6                mountd      superuser

mount.nfs4 172.22.1.4:/zdata/xen-iso-storage /iso-storage/
mount.nfs4 172.22.1.4:/zdata/xen-iso-library /iso-storage/

mount
172.22.1.4:/zdata/xen-iso-library on /iso-storage type nfs4 (rw,relatime,vers=4.1,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=172.22.1.7,local_lock=none,addr=172.22.1.4)
172.22.1.4:/zdata/xen-nfs-storage on /nfs-storage type nfs4 (rw,relatime,vers=4.1,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=172.22.1.7,local_lock=none,addr=172.22.1.4)

Also attached is the client Python code which I think may be problem here.
 

Attachments

  • nfs.py.txt
    8.8 KB · Views: 121
I recalled I had this issue with rpc bind until i added this line... you need a domain name for your server otherwise it can't bind to it.

Code:
nfsuserd_flags="-domain localhost.com"
 
The nfsuserd daemon shall take domain name from hostname if not otherwise specified and server has fully qualified domain name, but I tried it anyway and no success. But thanks for input.

I also found between many self-denying Linux how-tos and wikis, that server part has to be name, not IP address (ArchLinux I think), so I tried it also using /etc/hosts on the client, no luck.

Digging again deeper, I see in log on the client, that missing nfsd in rpcinfo -s call is probably main culprit here, so my questions 1 and 2 stand. From where came "missing serverpath" error I don't know. Also setting rpcdebug -m [nfs|rpc] -s all (set all debug flags for those two modules) on client yeld nothing at all.
Code:
Dec 27 22:31:36 xen-2u SM: [25530] _testHost: Testing host/port: storage-smc.ujezd.net,2049
Dec 27 22:31:36 xen-2u SM: [25530] scanning2 (target=storage-smc.ujezd.net)
Dec 27 22:31:36 xen-2u SM: [25530] scanning
Dec 27 22:31:36 xen-2u SM: [25530] ['/usr/sbin/showmount', '--no-headers', '-e', 'storage-smc.ujezd.net']
Dec 27 22:31:36 xen-2u SM: [25530]   pread SUCCESS
Dec 27 22:31:36 xen-2u SM: [25530] Raising exception [101, The request is missing the serverpath parameter]
Dec 27 22:31:36 xen-2u SM: [25530] lock: released /var/lock/sm/sr
Dec 27 22:31:36 xen-2u SM: [25530] ***** generic exception: sr_probe: EXCEPTION <class 'SR.SROSError'>, The request is missing the serverpath parameter
Dec 27 22:31:36 xen-2u SM: [25530]   File "/opt/xensource/sm/SRCommand.py", line 110, in run
Dec 27 22:31:36 xen-2u SM: [25530]     return self._run_locked(sr)
Dec 27 22:31:36 xen-2u SM: [25530]   File "/opt/xensource/sm/SRCommand.py", line 159, in _run_locked
Dec 27 22:31:36 xen-2u SM: [25530]     rv = self._run(sr, target)
Dec 27 22:31:36 xen-2u SM: [25530]   File "/opt/xensource/sm/SRCommand.py", line 332, in _run
Dec 27 22:31:36 xen-2u SM: [25530]     txt = sr.probe()
Dec 27 22:31:36 xen-2u SM: [25530]   File "/opt/xensource/sm/NFSSR", line 164, in probe
Dec 27 22:31:36 xen-2u SM: [25530]     self.validate_remotepath(True)
Dec 27 22:31:36 xen-2u SM: [25530]   File "/opt/xensource/sm/NFSSR", line 109, in validate_remotepath
Dec 27 22:31:36 xen-2u SM: [25530]     raise xs_errors.XenError('ConfigServerPathMissing')
Dec 27 22:31:36 xen-2u SM: [25530]
Dec 27 22:31:36 xen-2u SM: [25530] ***** NFS VHD: EXCEPTION <class 'SR.SROSError'>, The request is missing the serverpath parameter
Dec 27 22:31:36 xen-2u SM: [25530]   File "/opt/xensource/sm/SRCommand.py", line 378, in run
Dec 27 22:31:36 xen-2u SM: [25530]     ret = cmd.run(sr)
Dec 27 22:31:36 xen-2u SM: [25530]   File "/opt/xensource/sm/SRCommand.py", line 110, in run
Dec 27 22:31:36 xen-2u SM: [25530]     return self._run_locked(sr)
Dec 27 22:31:36 xen-2u SM: [25530]   File "/opt/xensource/sm/SRCommand.py", line 159, in _run_locked
Dec 27 22:31:36 xen-2u SM: [25530]     rv = self._run(sr, target)
Dec 27 22:31:36 xen-2u SM: [25530]   File "/opt/xensource/sm/SRCommand.py", line 332, in _run
Dec 27 22:31:36 xen-2u SM: [25530]     txt = sr.probe()
Dec 27 22:31:36 xen-2u SM: [25530]   File "/opt/xensource/sm/NFSSR", line 164, in probe
Dec 27 22:31:36 xen-2u SM: [25530]     self.validate_remotepath(True)
Dec 27 22:31:36 xen-2u SM: [25530]   File "/opt/xensource/sm/NFSSR", line 109, in validate_remotepath
Dec 27 22:31:36 xen-2u SM: [25530]     raise xs_errors.XenError('ConfigServerPathMissing')
Dec 27 22:31:36 xen-2u SM: [25530]
Dec 27 22:32:02 xen-2u SM: [25713] sr_create {'sr_uuid': '34c61bf5-32d6-a0b7-47e7-65209f69ddb9', 'subtask_of': 'DummyRef:|2360ef68-4b8d-4db8-8943-ec4370f77fa7|SR.create', 'args': ['0'], 'host_ref': 'OpaqueRef:2669e2f3-300e-4748-916f-5811c337e830', 'session_ref': 'OpaqueRef:8f327b0d-9799-420e-8164-a81353aa99ee', 'device_config': {'location': 'storage-smc.ujezd.net:/zdata/xen-iso-library', 'type': 'nfs_iso', 'SRmaster': 'true', 'nfsversion': '4'}, 'command': 'sr_create', 'sr_ref': 'OpaqueRef:55ecd645-3bf5-4a06-8aae-79443b019047'}
Dec 27 22:32:02 xen-2u SM: [25713] _testHost: Testing host/port: storage-smc.ujezd.net,2049
Dec 27 22:32:02 xen-2u SM: [25713] ['/usr/sbin/rpcinfo', '-s', 'storage-smc.ujezd.net']
Dec 27 22:32:02 xen-2u SM: [25713]   pread SUCCESS
Dec 27 22:32:02 xen-2u SM: [25713] NFS service not ready on server storage-smc.ujezd.net
 
Back
Top