Help understand the pNFS server settings on FreeBSD 14.1 and with FreeBSD 14.1 and Linux Ubuntu 20/22/24 clients.
I have one MDS server - mds0 and three DS servers with names - ds1, ds2, ds3.
Configuration of DS servers:
Configuration of MDS server:
Configuration of FreeBSD 14.1 client:
Configuration of Linux Ubuntu 20/22/24 and CentOS 9 clients:
With the above configuration, I have serious problems:
1) When writing a file from a Linux client to a pNFS server, the recording hangs stably. Moreover, the "cp" copying process cannot be "killed" even through "kill -9". When examining traffic between the Linux client and MDS+DS, "TCP ZeroWindow" errors are detected. A Linux client hang is accompanied by constant requests from the Linux client "LAYOUTGET" to the MDS server. And MDS responses "LAYOUTGET Status: NFS4ERR_BAD_STATEID".
The "TCP ZeroWindow" error can probably be defeated by using async for the MDS server. But I could not find the cause of the NFS4ERR_BAD_STATEID error.
2) When writing a file from the FreeBSD 14.1 client to the pNFS server, the recording works confidently, but the recording itself does not go directly to the DS server, but through the MDS server. That is, the MDS server is used as a proxy. In this case, there is no effect from the use of pNFS.
Who has encountered this behavior of FreeBSD as a pNFS server? Please help!
I have one MDS server - mds0 and three DS servers with names - ds1, ds2, ds3.
Configuration of DS servers:
Code:
/etc/rc.conf:
nfs_server_maxio="1048576"
rpcbind_enable="YES"
mountd_enable="YES"
nfs_server_enable="YES"
nfsv4_server_enable="YES"
nfs_server_flags="-u -t -n 32"
nfsuserd_enable="YES"
mountd_flags="-S"
nfsuserd_flags="-manage-gids"
/etc/sysctl.conf
#kern.ipc.maxsockbuf=4737024
#vfs.nfsd.srvmaxio=1048576
vfs.nfsd.maxthreads=128
/etc/exports
V4: /DSstore -sec=sys -network 10.100.31.0 -mask 255.255.255.0
/DSstore -sec=sys -maproot=root mds0
/DSstore -sec=sys -network 10.100.31.0 -mask 255.255.255.0
# ls /DSstore
.snap ds0 ds1 ds10 ds11 ds12 ds13 ds14 ds15 ds16 ds17 ds18 ds19 ds2 ds3 ds4 ds5 ds6 ds7 ds8 ds9Configuration of MDS server:
Code:
/etc/rc.conf:
# NFS
nfs_server_maxio="1048576"
rpcbind_enable="YES"
mountd_enable="YES"
nfs_server_enable="YES"
nfsv4_server_enable="YES"
nfs_server_flags="-u -t -n 32 -p ds1:/data1,ds2:/data2,ds3:/data3"
nfsuserd_enable="YES"
mountd_flags="-S"
nfsuserd_flags="-manage-gids"
# NFS client
nfs_client_enable="YES"
/etc/sysctl.conf
#kern.ipc.maxsockbuf=4737024
#vfs.nfsd.srvmaxio=1048576
vfs.nfsd.maxthreads=128
/etc/exports:
V4: /mds -sec=sys -network 10.100.31.0 -mask 255.255.255.0
/mds -sec=sys -maproot=root -network 10.100.31.0 -mask 255.255.255.0
#nfsstat -m
ds1:/ on /data1
nfsv4,minorversion=2,tcp,resvport,nconnect=1,soft,cto,sec=sys,acdirmin=3,acdirmax=60,acregmin=5,acregmax=60,nametimeo=60,negnametimeo=60,rsize=1048576,wsize=1048576,readdirsize=1048576,readahead=1,wcommitsize=4194304,timeout=120,retrans=2
ds2:/ on /data2
nfsv4,minorversion=2,tcp,resvport,nconnect=1,soft,cto,sec=sys,acdirmin=3,acdirmax=60,acregmin=5,acregmax=60,nametimeo=60,negnametimeo=60,rsize=1048576,wsize=1048576,readdirsize=1048576,readahead=1,wcommitsize=4194304,timeout=120,retrans=2
ds3:/ on /data3
nfsv4,minorversion=2,tcp,resvport,nconnect=1,soft,cto,sec=sys,acdirmin=3,acdirmax=60,acregmin=5,acregmax=60,nametimeo=60,negnametimeo=60,rsize=1048576,wsize=1048576,readdirsize=1048576,readahead=1,wcommitsize=4194304,timeout=120,retrans=2Configuration of FreeBSD 14.1 client:
Code:
/etc/rc.conf:
# NFS client
nfs_client_enable="YES"
# for pnfs
nfscbd_enable="YES"
# mount -t nfs -o nfsv4,minorversion=2,pnfs mds0:/ /mnt
# nfsstat -m
mds0:/ on /mnt
nfsv4,minorversion=2,pnfs,tcp,resvport,nconnect=1,hard,cto,sec=sys,acdirmin=3,acdirmax=60,acregmin=5,acregmax=60,nametimeo=60,negnametimeo=60,rsize=131072,wsize=131072,readdirsize=131072,readahead=1,wcommitsize=4194304,timeout=120,retrans=2147483647Configuration of Linux Ubuntu 20/22/24 and CentOS 9 clients:
Code:
# mount -t nfs4 -o rw,vers=4.2 mds0:/ /mnt/123
# nfsstat -m
/mnt/123 from mds0:/
Flags: rw,relatime,vers=4.2,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=10.100.31.50,local_lock=none,addr=10.100.31.36With the above configuration, I have serious problems:
1) When writing a file from a Linux client to a pNFS server, the recording hangs stably. Moreover, the "cp" copying process cannot be "killed" even through "kill -9". When examining traffic between the Linux client and MDS+DS, "TCP ZeroWindow" errors are detected. A Linux client hang is accompanied by constant requests from the Linux client "LAYOUTGET" to the MDS server. And MDS responses "LAYOUTGET Status: NFS4ERR_BAD_STATEID".
The "TCP ZeroWindow" error can probably be defeated by using async for the MDS server. But I could not find the cause of the NFS4ERR_BAD_STATEID error.
2) When writing a file from the FreeBSD 14.1 client to the pNFS server, the recording works confidently, but the recording itself does not go directly to the DS server, but through the MDS server. That is, the MDS server is used as a proxy. In this case, there is no effect from the use of pNFS.
Who has encountered this behavior of FreeBSD as a pNFS server? Please help!