Hi all,
I have longstanding issue with our NFS server, going probably from 10.x release times. Also problematic counterpart (client) has undergone multiple upgrades, but problem persists.
Quick intro - NFS server on FreeBSD 12.2 (running there probably for about 3-4 years, hence mentioning 10.x release), couple of clients, mostly Linux. The problematic one is xen on two servers in pool. Now running latest xcp-ng 8.2 (opensource fork of the Citrix product after their licensing changes, based on CentOS 7), also with history of upgrades over the years.
The problem is I can mount and use NFS shares from any client, even those xcp-ng servers, but I can't add what xen calls NFS storage repository, which is basically only fancy name for NFS mount with info stored in xen internal configuration database. There are three ways to do it (known to me), xe command line utility, web tool from authors of the mentioned fork named Xen Orchestra and Windows application, which is remnant of the Citrix era called XenCenter. All three use (AFAIK) some Python scripting on the server (NFS client in this case) to do it.
It would be easy to blame this Python part of the problem, but I noticed, that on the FreeBSD side nfsd does not register service with the rpcbind, so I think this may be (at least part of) the problem. I read somewhere, that NFSv4 can work without RPC and in fact, it does for me, at least partially, but I am not sure what specification says and which part has to be blamed here.
So my questions are:
1. Why my nfsd doesn't register with rpcbind?
2. Is this registration somewhat optional at least for NFSv4?
3. How can I get some useful debug info? Using -d options in our servers startup configuration where it is available doesn't produce much. Should I watch exchanged packets with tcpdump/Wireshark or is there some better way?
In the following snippets, 172.22.1.4 is the NFS server and 172.22.1.7 and 172.22.1.27 are the troublesome clients.
From the server
/etc/rc.conf
/etc/sysctl.conf
/etc/exports
And one of the Linux clients - rpcinfo doesn't show nfsd, but mounts can be probed and mounted without problem.
Also attached is the client Python code which I think may be problem here.
I have longstanding issue with our NFS server, going probably from 10.x release times. Also problematic counterpart (client) has undergone multiple upgrades, but problem persists.
Quick intro - NFS server on FreeBSD 12.2 (running there probably for about 3-4 years, hence mentioning 10.x release), couple of clients, mostly Linux. The problematic one is xen on two servers in pool. Now running latest xcp-ng 8.2 (opensource fork of the Citrix product after their licensing changes, based on CentOS 7), also with history of upgrades over the years.
The problem is I can mount and use NFS shares from any client, even those xcp-ng servers, but I can't add what xen calls NFS storage repository, which is basically only fancy name for NFS mount with info stored in xen internal configuration database. There are three ways to do it (known to me), xe command line utility, web tool from authors of the mentioned fork named Xen Orchestra and Windows application, which is remnant of the Citrix era called XenCenter. All three use (AFAIK) some Python scripting on the server (NFS client in this case) to do it.
It would be easy to blame this Python part of the problem, but I noticed, that on the FreeBSD side nfsd does not register service with the rpcbind, so I think this may be (at least part of) the problem. I read somewhere, that NFSv4 can work without RPC and in fact, it does for me, at least partially, but I am not sure what specification says and which part has to be blamed here.
So my questions are:
1. Why my nfsd doesn't register with rpcbind?
2. Is this registration somewhat optional at least for NFSv4?
3. How can I get some useful debug info? Using -d options in our servers startup configuration where it is available doesn't produce much. Should I watch exchanged packets with tcpdump/Wireshark or is there some better way?
In the following snippets, 172.22.1.4 is the NFS server and 172.22.1.7 and 172.22.1.27 are the troublesome clients.
From the server
Code:
uname -a
FreeBSD storage-smc.ujezd.net 12.2-RELEASE-p1 FreeBSD 12.2-RELEASE-p1 GENERIC amd64
/etc/rc.conf
Code:
hostname="storage-smc.ujezd.net"
cloned_interfaces="vlan1000 vlan1500 vlan2000"
ifconfig_vlan1000="inet 172.22.1.4 netmask 255.255.0.0 vlan 1000 vlandev igb1"
ifconfig_vlan2000="inet 10.128.99.4 netmask 255.255.255.0 vlan 2000 vlandev igb1"
ifconfig_vlan1500="inet 192.168.222.1 netmask 255.255.255.0 vlan 1500 vlandev igb1"
ifconfig_igb1="up"
nfsuserd_flags="-verbose"
defaultrouter="172.22.0.1"
sshd_enable="YES"
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="AUTO"
zfs_enable="YES"
nfsv4_server_enable="YES"
nfsuserd_enable="YES"
rpcbind_enable="YES"
nfs_server_enable="YES"
#nfs_server_flags="-h 172.22.1.4"
mountd_enable="YES"
mountd_flags="-r"
kld_list="aesni ipmi"
smartd_enable="YES"
ntpdate_hosts="10.128.99.95"
ntpdate_enable="YES"
ntpd_sync_on_start="YES"
ntpd_enable="YES"
netdata_enable="YES"
microcode_update_enable="YES"
rpc_lockd_enable="YES" # Run NFS rpc.lockd needed for client/server.
rpc_statd_enable="YES" # Run NFS rpc.statd needed for client/server.
/etc/sysctl.conf
Code:
vfs.zfs.arc_max="48000000000"
vfs.nfsd.server_min_nfsvers=4
vfs.nfs.enable_uidtostring=1
vfs.nfsd.tcpcachetimeo: 300
vfs.nfsd.tcphighwater: 100000
/etc/exports
Code:
/zdata/email -maproot=root 10.128.99.79
/zdata/nf -maproot=root 172.22.255.249
/zdata/odkladgalery -maproot=root 172.22.1.10
/zdata/ios -maproot=root 172.22.1.14
/zdata/xen-iso-library -maproot=root 172.22.1.27
/zdata/xen-iso-library -maproot=root 172.22.1.7
/zdata/xen-nfs-storage -maproot=root 172.22.1.27
/zdata/xen-nfs-storage -maproot=root 172.22.1.7
/zdata/servers/xenserver -maproot=root 172.22.1.27
/zdata/servers/xenserver -maproot=root 172.22.1.7
/zdata/servers/xenserver -maproot=root 172.22.1.32
/zdata/virt -maproot=root 172.22.1.27
/zdata/virt -maproot=root 172.22.1.7
/zdata/virt -maproot=root 172.22.1.13
/zdata/virt -maproot=root 172.22.1.2
/zdata/virt -maproot=root -sec=sys 172.22.1.11
V4: / -sec=sys -network 172.22.0.0 -mask 255.255.0.0
V4: / -sec=sys -network 10.128.99.0 -mask 255.255.255.0
Code:
rpcinfo -s
program version(s) netid(s) service owner
100000 2,3,4 local,udp6,tcp6,udp,tcp rpcbind superuser
100024 1 tcp,udp,tcp6,udp6 status superuser
100021 4,3,1,0 tcp,udp,tcp6,udp6 nlockmgr superuser
100005 3,1 tcp,udp,tcp6,udp6 mountd superuser
And one of the Linux clients - rpcinfo doesn't show nfsd, but mounts can be probed and mounted without problem.
Code:
showmount -e 172.22.1.4
Export list for 172.22.1.4:
/zdata/virt 172.22.1.11,172.22.1.2,172.22.1.13,172.22.1.7,172.22.1.27
/zdata/xen-nfs-storage 172.22.1.7,172.22.1.27
/zdata/xen-iso-library 172.22.1.7,172.22.1.27
/zdata/odkladgalery 172.22.1.10
/zdata/email 10.128.99.79
/zdata/servers/xenserver 172.22.1.32,172.22.1.7,172.22.1.27
/zdata/ios 172.22.1.14
/zdata/nf 172.22.255.249
rpcinfo -s 172.22.1.4
program version(s) netid(s) service owner
100000 2,3,4 local,udp6,tcp6,udp,tcp portmapper superuser
100024 1 tcp,udp,tcp6,udp6 status superuser
100021 4,3,1,0 tcp,udp,tcp6,udp6 nlockmgr superuser
100005 3,1 tcp,udp,tcp6,udp6 mountd superuser
mount.nfs4 172.22.1.4:/zdata/xen-iso-storage /iso-storage/
mount.nfs4 172.22.1.4:/zdata/xen-iso-library /iso-storage/
mount
172.22.1.4:/zdata/xen-iso-library on /iso-storage type nfs4 (rw,relatime,vers=4.1,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=172.22.1.7,local_lock=none,addr=172.22.1.4)
172.22.1.4:/zdata/xen-nfs-storage on /nfs-storage type nfs4 (rw,relatime,vers=4.1,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=172.22.1.7,local_lock=none,addr=172.22.1.4)
Also attached is the client Python code which I think may be problem here.