NFSv4 file server problems

Hi,

Part of my job is supporting Linux users at a university, and we have recently migrated one of our file servers from Solaris to FreeBSD.

We've had some problems and hence I'm posting here.

HW:

CPU: Intel(R) Xeon(R) CPU E5-2667 v4 @ 3.20GHz (3200.07-MHz K8-class CPU)
real memory = 412304277504 (393204 MB)

The server is running zfs, samba & nfsv4.

FreeBSD 11.2.

/etc/rc.conf:
Code:
zfs_enable="YES"

# NFS
nfs_server_enable="YES"
nfs_server_flags="-u -t -n 4"    # Flags to nfsd (if enabled).
nfsv4_server_enable="YES"        # Enable support for NFSv4
nfsuserd_enable="YES"            # NFSv4 user/group name mapping daemon
nfsuserd_flags="-domain ourdomain.se -usermax 5000 20"
gssd_enable="YES"                # Run the gssd daemon (or NO).
gssd_flags=""

#Linux compat
linux_enable="YES"

#Autofs
autofs_enable="YES"
autounmountd_flags="-r 200 -t 200 -v"
automountd_flags="-v"

# Samba (in /usr/local)
samba_server_enable="YES"
winbindd_enable="YES"
nmbd_enable="NO"

# nscd
nscd_enable="YES"

#TSM
dsmcad_enable="YES"

# PF Firewall (from OpenBSD 4.5) see /etc/pf.conf
pf_enable="YES"

# inetd (needed for rquouta)
inetd_enable="YES"

/etc/nscd.conf:

Code:
#
# Default caching daemon configuration file
# $FreeBSD: releng/11.2/etc/nscd.conf 172742 2007-10-18 08:26:20Z bushman $
#

#
enable-cache passwd yes
enable-cache group yes
enable-cache hosts yes
enable-cache services yes
enable-cache protocols yes
enable-cache rpc yes
enable-cache networks yes

#
suggested-size passwd 19997
suggested-size group 19997
keep-hot-count passwd 163840
keep-hot-count group 163840

/etc/nsswitch.conf:

Code:
#
# nsswitch.conf(5) - name service switch configuration file
# $FreeBSD: releng/11.2/etc/nsswitch.conf 301711 2016-06-09 01:28:44Z markj $
#
group: files
hosts: files dns
netgroup: compat
networks: files
passwd: cache files
shells: files
services: files
protocols: files
rpc: files

NOTE: We don't run the nfscbd daemon.

Code:
# wc -l /etc/group
   34371 /etc/group
# wc -l /etc/passwd
  124167 /etc/passwd

320 Linux clients, mostly Redhat Enterprise 7 but also approx. 100 RHEL 6.
Also an unknown number of windows samba clients.
Main applications on Linux: GNOME, Matlab, Firefox.
The server delivers NFSv4 + Samba homedirectories, group file directories and
various nfs shares with software.

Samba is mainly for group file directories, not homedirectories.

Mount flags on RHEL 7 clients:
Flags: rw,nosuid,nodev,relatime,vers=4.1,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=krb5p,clientaddr=x.x.x.x,local_lock=none,addr=x.x.x.x

PROBLEMS:

NOTE: I'm not the admin for this server, just a curious second line support person,
so the description below is going to be a bit vague as I have not been involved
in much of the actual troubleshooting.

The server has been running for approx. six months. The server performed well
during the first months, probably because the weren't many active NFS clients.
Apart from a few isolated incidents samba has caused no problems.

The server runs in a school environment and the problems started to appear when many students at once started
to login to linux computers in classrooms. In the beginning it caused the server to crash & re-boot, it was not able
to start the nfs-service again without some persuasion. The load was not high on the server but nscd & gssd was using a lot
of cpu.

The situation improved when we:

-decreased the number of groups in /etc/group from approx. 155000 to 38000.
-only used "files" for the group entry in nsswitch.conf instead of "cache files"
-installed a "nanny" script that re-starts gssd when it stops working

Now the server is able to start the nfs service again after a crash, in fact hasn't crashed
so much lately, perhaps because students/teachers have given up on using the Linux clients
for labs...;-(


QUESTIONS:

I wonder if there is any configuration flags to nfsd, nfsuserd or nscd that could be used
or set differently?

How many NFSv4 workstation clients should a single server be able to handle?

Threads: how many threads should the nfs daemon use? I've seen conflicting opinions about this...
Currently we have it set to 4, when we used higher numbers like 50 the server would run well but
crash as soon as many nfs clients became active (= students login in to comps.)

Would it be better to use ldap instead of passwd/group files (and skip nscd)?

I would be grateful for any input!
 
Some additional info about our config:

/etc/sysctl.conf:

Code:
# $FreeBSD: releng/11.2/etc/sysctl.conf 112200 2003-03-13 18:43:50Z mux $
#
#  This file is read when going to multi-user and its contents piped thru
#  ``sysctl'' to adjust kernel values.  ``man 5 sysctl.conf'' for details.
#

# Uncomment this to prevent users from seeing information about processes that
# are being run under another UID.
#security.bsd.see_other_uids=0
net.inet.icmp.icmplim_output=0

vfs.nfsd.tcphighwater=50000
vfs.nfsd.tcpcachetimeo=1000

/boot/loader.conf:

Code:
aesni_load="YES"
ispfw_load="YES"
isp_load="YES"
geom_multipath_load="YES"
nvme_load="YES"
nvd_load="YES"
kern.geom.label.disk_ident.enable="0"
kern.geom.label.gptid.enable="0"
vfs.zfs.min_auto_ashift=12
zfs_load="YES"
kern.msgbufsize=262144
 
O.K., if no one else starts wild-guessing what's going on on that machine , then I'll start :)
....
nfs_server_enable="YES"
nfs_server_flags="-u -t -n 4"
...
I wonder if there is any configuration flags to nfsd, nfsuserd or nscd that could be used
or set differently?
"set-differently-suggestion" ( normally only for NFS V3 ( not V4) ) :
Code:
rpcbind_enable="YES" or remove : nfs_server_flags="-u -t -n 4" (if only V4 is required)

I have a "suspicion" that RHEL is trying to access nfsv3 ... then you would also need to set mountd_enable = "YES".


Flags: .....vers=4.1,
perhaps
Code:
  vers=4

a look into your / etc /exports and crashdump(if available) could be interesting.. and an information about your zfs-environment...

... from Solaris to FreeBSD ...
For now, if I had such an unstable machine in a production environment, I would go backwards ;-)
 
mountd & rpcbind seems to be running even though they are not started from /etc/rc.conf:

Code:
root@sol:~ # grep rpcbind /etc/rc.conf
root@sol:~ # grep mountd  /etc/rc.conf
autounmountd_flags="-r 200 -t 200 -v"
automountd_flags="-v"
root@sol:~ # ps -auxw | grep mountd
root           873    0.0  0.0   289040   3812  -  Ss   11Sep18    129:18.15 /usr/sbin/mountd -r -S /etc/exports /etc/zfs/exports
root         59901    0.0  0.0     6424   2072  -  Is   Fri14        0:08.01 /usr/sbin/automountd -v
root         60079    0.0  0.0     8472   1400  -  Is   Fri14        0:37.66 /usr/sbin/autounmountd -r 200 -t 200 -v
root         86622    0.0  0.0     6660   2524  7  S+   09:12        0:00.00 grep mountd
root@sol:~ # ps -auxw | grep rpcbind
root           858    0.0  0.0     6368   2076  -  Ss   11Sep18      1:30.59 /usr/sbin/rpcbind
root         87218    0.0  0.0     6660   2524  7  S+   09:12        0:00.00 grep rpcbind
root@sol:~ #

I will post crashdumps & /etc/exports info later.
 
/etc/exports:

Code:
V4: /ufs -sec=krb5i:krb5p
/ufs/usersz/users21    -alldirs fox.xyz.ourdomain.se elk.xyz.ourdomain.se cat.xyz.ourdomain.se fly.xyz.ourdomain.se dog.xyz.ourdomain.se
/ufs/usersz/users22    -alldirs fox.xyz.ourdomain.se elk.xyz.ourdomain.se cat.xyz.ourdomain.se fly.xyz.ourdomain.se dog.xyz.ourdomain.se
/ufs/usersz/users23    -alldirs fox.xyz.ourdomain.se elk.xyz.ourdomain.se cat.xyz.ourdomain.se fly.xyz.ourdomain.se dog.xyz.ourdomain.se
/ufs/usersz/users24    -alldirs fox.xyz.ourdomain.se elk.xyz.ourdomain.se cat.xyz.ourdomain.se fly.xyz.ourdomain.se dog.xyz.ourdomain.se
/ufs/usersz/users25    -alldirs fox.xyz.ourdomain.se elk.xyz.ourdomain.se cat.xyz.ourdomain.se fly.xyz.ourdomain.se dog.xyz.ourdomain.se
/ufs/usersz/users26    -alldirs fox.xyz.ourdomain.se elk.xyz.ourdomain.se cat.xyz.ourdomain.se fly.xyz.ourdomain.se dog.xyz.ourdomain.se
/ufs/usersz/users27    -alldirs fox.xyz.ourdomain.se elk.xyz.ourdomain.se cat.xyz.ourdomain.se fly.xyz.ourdomain.se dog.xyz.ourdomain.se
/ufs/usersz/users28    -alldirs fox.xyz.ourdomain.se elk.xyz.ourdomain.se cat.xyz.ourdomain.se fly.xyz.ourdomain.se dog.xyz.ourdomain.se
/ufs/usersz/users29    -alldirs fox.xyz.ourdomain.se elk.xyz.ourdomain.se cat.xyz.ourdomain.se fly.xyz.ourdomain.se dog.xyz.ourdomain.se
/ufs/usersz/users30    -alldirs fox.xyz.ourdomain.se elk.xyz.ourdomain.se cat.xyz.ourdomain.se fly.xyz.ourdomain.se dog.xyz.ourdomain.se
/ufs/usersz/users31    -alldirs fox.xyz.ourdomain.se elk.xyz.ourdomain.se cat.xyz.ourdomain.se fly.xyz.ourdomain.se dog.xyz.ourdomain.se
/ufs/usersz/users32    -alldirs fox.xyz.ourdomain.se elk.xyz.ourdomain.se cat.xyz.ourdomain.se fly.xyz.ourdomain.se dog.xyz.ourdomain.se
/ufs/usersz/users33    -alldirs fox.xyz.ourdomain.se elk.xyz.ourdomain.se cat.xyz.ourdomain.se fly.xyz.ourdomain.se dog.xyz.ourdomain.se
/ufs/usersz/users34    -alldirs fox.xyz.ourdomain.se elk.xyz.ourdomain.se cat.xyz.ourdomain.se fly.xyz.ourdomain.se dog.xyz.ourdomain.se
/ufs/usersz/users35    -alldirs fox.xyz.ourdomain.se elk.xyz.ourdomain.se cat.xyz.ourdomain.se fly.xyz.ourdomain.se dog.xyz.ourdomain.se
/ufs/usersz/users36    -alldirs fox.xyz.ourdomain.se elk.xyz.ourdomain.se cat.xyz.ourdomain.se fly.xyz.ourdomain.se dog.xyz.ourdomain.se
/ufs/usersz/users37    -alldirs fox.xyz.ourdomain.se elk.xyz.ourdomain.se cat.xyz.ourdomain.se fly.xyz.ourdomain.se dog.xyz.ourdomain.se
/ufs/usersz/users38    -alldirs fox.xyz.ourdomain.se elk.xyz.ourdomain.se cat.xyz.ourdomain.se fly.xyz.ourdomain.se dog.xyz.ourdomain.se
/ufs/usersz/users39    -alldirs fox.xyz.ourdomain.se elk.xyz.ourdomain.se cat.xyz.ourdomain.se fly.xyz.ourdomain.se dog.xyz.ourdomain.se
/ufs/groups/groups01   -alldirs fox.xyz.ourdomain.se elk.xyz.ourdomain.se cat.xyz.ourdomain.se fly.xyz.ourdomain.se dog.xyz.ourdomain.se
/ufs/groups/groups02   -alldirs fox.xyz.ourdomain.se elk.xyz.ourdomain.se cat.xyz.ourdomain.se fly.xyz.ourdomain.se dog.xyz.ourdomain.se
/ufs/groups/groups03   -alldirs fox.xyz.ourdomain.se elk.xyz.ourdomain.se cat.xyz.ourdomain.se fly.xyz.ourdomain.se dog.xyz.ourdomain.se
/ufs/groups/groups04   -alldirs fox.xyz.ourdomain.se elk.xyz.ourdomain.se cat.xyz.ourdomain.se fly.xyz.ourdomain.se dog.xyz.ourdomain.se
/ufs/groups/groups05   -alldirs fox.xyz.ourdomain.se elk.xyz.ourdomain.se cat.xyz.ourdomain.se fly.xyz.ourdomain.se dog.xyz.ourdomain.se
/ufs/groups/groups06   -alldirs fox.xyz.ourdomain.se elk.xyz.ourdomain.se cat.xyz.ourdomain.se fly.xyz.ourdomain.se dog.xyz.ourdomain.se
/ufs/groups/groups07   -alldirs fox.xyz.ourdomain.se elk.xyz.ourdomain.se cat.xyz.ourdomain.se fly.xyz.ourdomain.se dog.xyz.ourdomain.se
/ufs/groups/groups08   -alldirs fox.xyz.ourdomain.se elk.xyz.ourdomain.se cat.xyz.ourdomain.se fly.xyz.ourdomain.se dog.xyz.ourdomain.se
/ufs/groups/groups09   -alldirs fox.xyz.ourdomain.se elk.xyz.ourdomain.se cat.xyz.ourdomain.se fly.xyz.ourdomain.se dog.xyz.ourdomain.se
/ufs/groups/groups10   -alldirs fox.xyz.ourdomain.se elk.xyz.ourdomain.se cat.xyz.ourdomain.se fly.xyz.ourdomain.se dog.xyz.ourdomain.se
/ufs/groups/groups11   -alldirs fox.xyz.ourdomain.se elk.xyz.ourdomain.se cat.xyz.ourdomain.se fly.xyz.ourdomain.se dog.xyz.ourdomain.se
/ufs/groups/groups12   -alldirs fox.xyz.ourdomain.se elk.xyz.ourdomain.se cat.xyz.ourdomain.se fly.xyz.ourdomain.se dog.xyz.ourdomain.se
/ufs/groups/groups13   -alldirs fox.xyz.ourdomain.se elk.xyz.ourdomain.se cat.xyz.ourdomain.se fly.xyz.ourdomain.se dog.xyz.ourdomain.se
/ufs/groups/groups14   -alldirs fox.xyz.ourdomain.se elk.xyz.ourdomain.se cat.xyz.ourdomain.se fly.xyz.ourdomain.se dog.xyz.ourdomain.se
/ufs/groups/groups15   -alldirs fox.xyz.ourdomain.se elk.xyz.ourdomain.se cat.xyz.ourdomain.se fly.xyz.ourdomain.se dog.xyz.ourdomain.se
/ufs/groups/groups16   -alldirs fox.xyz.ourdomain.se elk.xyz.ourdomain.se cat.xyz.ourdomain.se fly.xyz.ourdomain.se dog.xyz.ourdomain.se
/ufs/groups/groups17   -alldirs fox.xyz.ourdomain.se elk.xyz.ourdomain.se cat.xyz.ourdomain.se fly.xyz.ourdomain.se dog.xyz.ourdomain.se
/ufs/groups/groups18   -alldirs fox.xyz.ourdomain.se elk.xyz.ourdomain.se cat.xyz.ourdomain.se fly.xyz.ourdomain.se dog.xyz.ourdomain.se
/ufs/groups/groups19   -alldirs fox.xyz.ourdomain.se elk.xyz.ourdomain.se cat.xyz.ourdomain.se fly.xyz.ourdomain.se dog.xyz.ourdomain.se
/ufs/groups/manual01   -alldirs fox.xyz.ourdomain.se elk.xyz.ourdomain.se cat.xyz.ourdomain.se fly.xyz.ourdomain.se dog.xyz.ourdomain.se
/ufs/groups/manual02   -alldirs fox.xyz.ourdomain.se elk.xyz.ourdomain.se cat.xyz.ourdomain.se fly.xyz.ourdomain.se dog.xyz.ourdomain.se
/ufs/groups/manual03   -alldirs fox.xyz.ourdomain.se elk.xyz.ourdomain.se cat.xyz.ourdomain.se fly.xyz.ourdomain.se dog.xyz.ourdomain.se
/ufs/groups/manual03   -sec=krb5p vsc.xyz.ourdomain.se vsc2.xyz.ourdomain.se
#/ufs/groups/manual03   -sec=krb5i elk.xyz.ourdomain.se cat.xyz.ourdomain.se fly.xyz.ourdomain.se dog.xyz.ourdomain.se
#/ufs/groups/manual03   -alldirs fox.xyz.ourdomain.se 
/ufs/groups/manual03   -sec=krb5p -ro -network x.x.0.0/16
/ufs/groups/manual04   -alldirs fox.xyz.ourdomain.se elk.xyz.ourdomain.se cat.xyz.ourdomain.se fly.xyz.ourdomain.se dog.xyz.ourdomain.se
/ufs/groups/manual05   -alldirs fox.xyz.ourdomain.se elk.xyz.ourdomain.se cat.xyz.ourdomain.se fly.xyz.ourdomain.se dog.xyz.ourdomain.se
/ufs/groups/manual06   -alldirs fox.xyz.ourdomain.se elk.xyz.ourdomain.se cat.xyz.ourdomain.se fly.xyz.ourdomain.se dog.xyz.ourdomain.se
/ufs/groups/manual07   -alldirs fox.xyz.ourdomain.se elk.xyz.ourdomain.se cat.xyz.ourdomain.se fly.xyz.ourdomain.se dog.xyz.ourdomain.se
/ufs/groups/manual08   -alldirs fox.xyz.ourdomain.se elk.xyz.ourdomain.se cat.xyz.ourdomain.se fly.xyz.ourdomain.se dog.xyz.ourdomain.se
/ufs/groups/manual09   -alldirs fox.xyz.ourdomain.se elk.xyz.ourdomain.se cat.xyz.ourdomain.se fly.xyz.ourdomain.se dog.xyz.ourdomain.se
/ufs/groups/manual10   -alldirs fox.xyz.ourdomain.se elk.xyz.ourdomain.se cat.xyz.ourdomain.se fly.xyz.ourdomain.se dog.xyz.ourdomain.se
/ufs/groups/manual11   -alldirs fox.xyz.ourdomain.se elk.xyz.ourdomain.se cat.xyz.ourdomain.se fly.xyz.ourdomain.se dog.xyz.ourdomain.se
/ufs/groups/manual12   -alldirs fox.xyz.ourdomain.se elk.xyz.ourdomain.se cat.xyz.ourdomain.se fly.xyz.ourdomain.se dog.xyz.ourdomain.se
/ufs/groups/manual13   -alldirs fox.xyz.ourdomain.se elk.xyz.ourdomain.se cat.xyz.ourdomain.se fly.xyz.ourdomain.se dog.xyz.ourdomain.se
/ufs/groups/manual14   -alldirs fox.xyz.ourdomain.se elk.xyz.ourdomain.se cat.xyz.ourdomain.se fly.xyz.ourdomain.se dog.xyz.ourdomain.se
/ufs/groups/manual15   -alldirs fox.xyz.ourdomain.se elk.xyz.ourdomain.se cat.xyz.ourdomain.se fly.xyz.ourdomain.se dog.xyz.ourdomain.se
/ufs/groups/noback01   -alldirs fox.xyz.ourdomain.se elk.xyz.ourdomain.se cat.xyz.ourdomain.se fly.xyz.ourdomain.se dog.xyz.ourdomain.se
/ufs/groups/noback02   -alldirs fox.xyz.ourdomain.se elk.xyz.ourdomain.se cat.xyz.ourdomain.se fly.xyz.ourdomain.se dog.xyz.ourdomain.se
/ufs/groups/noback03   -alldirs fox.xyz.ourdomain.se elk.xyz.ourdomain.se cat.xyz.ourdomain.se fly.xyz.ourdomain.se dog.xyz.ourdomain.se
/ufs/groups/noback04   -alldirs fox.xyz.ourdomain.se elk.xyz.ourdomain.se cat.xyz.ourdomain.se fly.xyz.ourdomain.se dog.xyz.ourdomain.se
/ufs/groups/noback05   -alldirs fox.xyz.ourdomain.se elk.xyz.ourdomain.se cat.xyz.ourdomain.se fly.xyz.ourdomain.se dog.xyz.ourdomain.se
/ufs/groups/noback06   -alldirs fox.xyz.ourdomain.se elk.xyz.ourdomain.se cat.xyz.ourdomain.se fly.xyz.ourdomain.se dog.xyz.ourdomain.se
/ufs/groups/noback07   -alldirs fox.xyz.ourdomain.se elk.xyz.ourdomain.se cat.xyz.ourdomain.se fly.xyz.ourdomain.se dog.xyz.ourdomain.se
/ufs/groups/noback08   -alldirs fox.xyz.ourdomain.se elk.xyz.ourdomain.se cat.xyz.ourdomain.se fly.xyz.ourdomain.se dog.xyz.ourdomain.se
/ufs/groups/noback09   -alldirs fox.xyz.ourdomain.se elk.xyz.ourdomain.se cat.xyz.ourdomain.se fly.xyz.ourdomain.se dog.xyz.ourdomain.se
/ufs/groups/noback10   -alldirs fox.xyz.ourdomain.se elk.xyz.ourdomain.se cat.xyz.ourdomain.se fly.xyz.ourdomain.se dog.xyz.ourdomain.se
/ufs/groups/noback11   -alldirs fox.xyz.ourdomain.se elk.xyz.ourdomain.se cat.xyz.ourdomain.se fly.xyz.ourdomain.se dog.xyz.ourdomain.se
/ufs/groups/noback12   -alldirs fox.xyz.ourdomain.se elk.xyz.ourdomain.se cat.xyz.ourdomain.se fly.xyz.ourdomain.se dog.xyz.ourdomain.se
/ufs/groups/noback13   -alldirs fox.xyz.ourdomain.se elk.xyz.ourdomain.se cat.xyz.ourdomain.se fly.xyz.ourdomain.se dog.xyz.ourdomain.se
/ufs/groups/noback14   -alldirs fox.xyz.ourdomain.se elk.xyz.ourdomain.se cat.xyz.ourdomain.se fly.xyz.ourdomain.se dog.xyz.ourdomain.se
/ufs/groups/noback15   -alldirs fox.xyz.ourdomain.se elk.xyz.ourdomain.se cat.xyz.ourdomain.se fly.xyz.ourdomain.se dog.xyz.ourdomain.se

/etc/zfs/exports:

Code:
# !!! DO NOT EDIT THIS FILE MANUALLY !!!

/ufs    -sec=krb5p 
/ufs/groups     -sec=krb5p 
/ufs/groups/groups01    -sec=krb5p 
/ufs/groups/groups02    -sec=krb5p 
/ufs/groups/groups03    -sec=krb5p 
/ufs/groups/groups04    -sec=krb5p 
/ufs/groups/groups05    -sec=krb5p 
/ufs/groups/groups06    -sec=krb5p 
/ufs/groups/groups07    -sec=krb5p 
/ufs/groups/groups08    -sec=krb5p 
/ufs/groups/groups09    -sec=krb5p 
/ufs/groups/groups10    -sec=krb5p 
/ufs/groups/groups11    -sec=krb5p 
/ufs/groups/groups12    -sec=krb5p 
/ufs/groups/groups13    -sec=krb5p 
/ufs/groups/groups14    -sec=krb5p 
/ufs/groups/groups15    -sec=krb5p 
/ufs/groups/groups16    -sec=krb5p 
/ufs/groups/groups17    -sec=krb5p 
/ufs/groups/groups18    -sec=krb5p 
/ufs/groups/groups19    -sec=krb5p 
/ufs/groups/manual01    -sec=krb5p 
/ufs/groups/manual02    -sec=krb5p 
/ufs/groups/manual04    -sec=krb5p 
/ufs/groups/manual05    -sec=krb5p 
/ufs/groups/manual06    -sec=krb5p 
/ufs/groups/manual07    -sec=krb5p 
/ufs/groups/manual08    -sec=krb5p 
/ufs/groups/manual09    -sec=krb5p 
/ufs/groups/manual10    -sec=krb5p 
/ufs/groups/manual11    -sec=krb5p 
/ufs/groups/manual12    -sec=krb5p 
/ufs/groups/manual13    -sec=krb5p 
/ufs/groups/manual14    -sec=krb5p 
/ufs/groups/manual15    -sec=krb5p 
/ufs/groups/noback01    -sec=krb5p 
/ufs/groups/noback02    -sec=krb5p 
/ufs/groups/noback03    -sec=krb5p 
/ufs/groups/noback04    -sec=krb5p 
/ufs/groups/noback05    -sec=krb5p 
/ufs/groups/noback06    -sec=krb5p 
/ufs/groups/noback07    -sec=krb5p 
/ufs/groups/noback08    -sec=krb5p 
/ufs/groups/noback09    -sec=krb5p 
/ufs/groups/noback10    -sec=krb5p 
/ufs/groups/noback11    -sec=krb5p 
/ufs/groups/noback12    -sec=krb5p 
/ufs/groups/noback13    -sec=krb5p 
/ufs/groups/noback14    -sec=krb5p 
/ufs/groups/noback15    -sec=krb5p 
/ufs/usersz     -sec=krb5p 
/ufs/usersz/users21     -sec=krb5p 
/ufs/usersz/users22     -sec=krb5p 
/ufs/usersz/users23     -sec=krb5p 
/ufs/usersz/users24     -sec=krb5p 
/ufs/usersz/users25     -sec=krb5p 
/ufs/usersz/users26     -sec=krb5p 
/ufs/usersz/users27     -sec=krb5p 
/ufs/usersz/users28     -sec=krb5p 
/ufs/usersz/users29     -sec=krb5p 
/ufs/usersz/users30     -sec=krb5p 
/ufs/usersz/users31     -sec=krb5p 
/ufs/usersz/users32     -sec=krb5p 
/ufs/usersz/users33     -sec=krb5p 
/ufs/usersz/users34     -sec=krb5p 
/ufs/usersz/users35     -sec=krb5p 
/ufs/usersz/users36     -sec=krb5p 
/ufs/usersz/users37     -sec=krb5p 
/ufs/usersz/users38     -sec=krb5p 
/ufs/usersz/users39     -sec=krb5p
 
Don't use the sharenfs feature of ZFS and /etc/exports. Use one or the other, not both. A lot of NFS exports are now doubled, that's going to cause issues.
 
zfs:

Code:
root@sol:~ # zfs list
NAME                                            USED  AVAIL  REFER  MOUNTPOINT
giza                                           21.7T  8.35T    88K  /ufs/giza
giza/necropolis                                21.7T  8.35T    88K  /ufs/giza/necropolis
giza/necropolis/pyramid                        21.7T  8.35T  56.6G  /ufs/giza/necropolis/pyramid
giza/necropolis/pyramid/dumas                  10.4T  8.35T  10.4T  /ufs/giza/necropolis/pyramid/dumas
giza/necropolis/pyramid/hades                  63.5G  8.35T  63.5G  /ufs/giza/necropolis/pyramid/hades
giza/necropolis/pyramid/hades-last             55.6G  8.35T  55.6G  /ufs/giza/necropolis/pyramid/hades-last
giza/necropolis/pyramid/hades_no_passwd_user   9.53G  8.35T  9.53G  /ufs/giza/necropolis/pyramid/hades_no_passwd_user
giza/necropolis/pyramid/hades_no_pdb_user      2.84G  8.35T  2.84G  /ufs/giza/necropolis/pyramid/hades_no_pdb_user
giza/necropolis/pyramid/harare                 68.5G  8.35T  68.5G  /ufs/giza/necropolis/pyramid/harare
giza/necropolis/pyramid/harare-last             118G  8.35T   118G  /ufs/giza/necropolis/pyramid/harare-last
giza/necropolis/pyramid/harare_no_passwd_user  2.28G  8.35T  2.28G  /ufs/giza/necropolis/pyramid/harare_no_passwd_user
giza/necropolis/pyramid/harare_no_pdb_user      207G  8.35T   207G  /ufs/giza/necropolis/pyramid/harare_no_pdb_user
giza/necropolis/pyramid/idefix                  396G  8.35T   396G  /ufs/giza/necropolis/pyramid/idefix
giza/necropolis/pyramid/lambda                  658G  8.35T   658G  /ufs/giza/necropolis/pyramid/lambda
giza/necropolis/pyramid/lambda_2               9.14G  8.35T  9.14G  /ufs/giza/necropolis/pyramid/lambda_2
giza/necropolis/pyramid/lambda_7               14.8G  8.35T  14.8G  /ufs/giza/necropolis/pyramid/lambda_7
giza/necropolis/pyramid/lambda_8               18.9G  8.35T  18.9G  /ufs/giza/necropolis/pyramid/lambda_8
giza/necropolis/pyramid/lambda_no_pdb_user     35.4G  8.35T  35.4G  /ufs/giza/necropolis/pyramid/lambda_no_pdb_user
giza/necropolis/pyramid/mail                   1.35T  8.35T  1.35T  /ufs/giza/necropolis/pyramid/mail
giza/necropolis/pyramid/nets31                 50.8G  8.35T  50.8G  /ufs/giza/necropolis/pyramid/nets31
giza/necropolis/pyramid/obelix                 17.2G  8.35T  17.2G  /ufs/giza/necropolis/pyramid/obelix
giza/necropolis/pyramid/olivia                  207G  8.35T   207G  /ufs/giza/necropolis/pyramid/olivia
giza/necropolis/pyramid/pacman                 3.14T  8.35T   380K  /ufs/giza/necropolis/pyramid/pacman
giza/necropolis/pyramid/pacman/a                221G  8.35T   221G  /ufs/giza/necropolis/pyramid/pacman/a
giza/necropolis/pyramid/pacman/b                188G  8.35T   188G  /ufs/giza/necropolis/pyramid/pacman/b
giza/necropolis/pyramid/pacman/c                113G  8.35T   113G  /ufs/giza/necropolis/pyramid/pacman/c
giza/necropolis/pyramid/pacman/d                111G  8.35T   111G  /ufs/giza/necropolis/pyramid/pacman/d
giza/necropolis/pyramid/pacman/e                178G  8.35T   178G  /ufs/giza/necropolis/pyramid/pacman/e
giza/necropolis/pyramid/pacman/f                130G  8.35T   130G  /ufs/giza/necropolis/pyramid/pacman/f
giza/necropolis/pyramid/pacman/g                133G  8.35T   133G  /ufs/giza/necropolis/pyramid/pacman/g
giza/necropolis/pyramid/pacman/h                191G  8.35T   191G  /ufs/giza/necropolis/pyramid/pacman/h
giza/necropolis/pyramid/pacman/i               59.0G  8.35T  59.0G  /ufs/giza/necropolis/pyramid/pacman/i
giza/necropolis/pyramid/pacman/j                222G  8.35T   222G  /ufs/giza/necropolis/pyramid/pacman/j
giza/necropolis/pyramid/pacman/k                235G  8.35T   235G  /ufs/giza/necropolis/pyramid/pacman/k
giza/necropolis/pyramid/pacman/l                134G  8.35T   134G  /ufs/giza/necropolis/pyramid/pacman/l
giza/necropolis/pyramid/pacman/m                249G  8.35T   249G  /ufs/giza/necropolis/pyramid/pacman/m
giza/necropolis/pyramid/pacman/n               86.9G  8.35T  86.9G  /ufs/giza/necropolis/pyramid/pacman/n
giza/necropolis/pyramid/pacman/o               57.9G  8.35T  57.9G  /ufs/giza/necropolis/pyramid/pacman/o
giza/necropolis/pyramid/pacman/p                109G  8.35T   109G  /ufs/giza/necropolis/pyramid/pacman/p
giza/necropolis/pyramid/pacman/q               5.12G  8.35T  5.12G  /ufs/giza/necropolis/pyramid/pacman/q
giza/necropolis/pyramid/pacman/r               93.3G  8.35T  93.3G  /ufs/giza/necropolis/pyramid/pacman/r
giza/necropolis/pyramid/pacman/s                261G  8.35T   261G  /ufs/giza/necropolis/pyramid/pacman/s
giza/necropolis/pyramid/pacman/t                134G  8.35T   134G  /ufs/giza/necropolis/pyramid/pacman/t
giza/necropolis/pyramid/pacman/u               18.3G  8.35T  18.3G  /ufs/giza/necropolis/pyramid/pacman/u
giza/necropolis/pyramid/pacman/v                177G  8.35T   177G  /ufs/giza/necropolis/pyramid/pacman/v
giza/necropolis/pyramid/pacman/w               63.0G  8.35T  63.0G  /ufs/giza/necropolis/pyramid/pacman/w
giza/necropolis/pyramid/pacman/x               7.56G  8.35T  7.56G  /ufs/giza/necropolis/pyramid/pacman/x
giza/necropolis/pyramid/pacman/y               23.1G  8.35T  23.1G  /ufs/giza/necropolis/pyramid/pacman/y
giza/necropolis/pyramid/pacman/z               20.8G  8.35T  20.8G  /ufs/giza/necropolis/pyramid/pacman/z
giza/necropolis/pyramid/sol                    4.87T  8.35T  4.87T  /ufs/giza/necropolis/pyramid/sol
groups                                         84.5T  35.6T   120K  /ufs/groups
groups/groups01                                2.12T  35.6T  2.12T  /ufs/groups/groups01
groups/groups02                                2.09T  35.6T  2.08T  /ufs/groups/groups02
groups/groups03                                2.05T  35.6T  2.03T  /ufs/groups/groups03
groups/groups04                                3.01T  35.6T  2.82T  /ufs/groups/groups04
groups/groups05                                2.43T  35.6T  2.43T  /ufs/groups/groups05
groups/groups06                                3.95T  35.6T  3.88T  /ufs/groups/groups06
groups/groups07                                2.66T  35.6T  2.65T  /ufs/groups/groups07
groups/groups08                                5.00T  35.6T  4.99T  /ufs/groups/groups08
groups/groups09                                2.93T  35.6T  2.92T  /ufs/groups/groups09
groups/groups10                                4.54T  35.6T  4.42T  /ufs/groups/groups10
groups/groups11                                1.00T  35.6T  1.00T  /ufs/groups/groups11
groups/groups12                                 455G  35.6T   452G  /ufs/groups/groups12
groups/groups13                                1.12T  35.6T  1.12T  /ufs/groups/groups13
groups/groups14                                2.06T  35.6T  2.00T  /ufs/groups/groups14
groups/groups15                                 979G  35.6T   977G  /ufs/groups/groups15
groups/groups16                                 748G  35.6T   745G  /ufs/groups/groups16
groups/groups17                                 794G  35.6T   792G  /ufs/groups/groups17
groups/groups18                                1.45T  35.6T  1.43T  /ufs/groups/groups18
groups/groups19                                1.17T  35.6T  1.12T  /ufs/groups/groups19
groups/manual01                                1.45T  35.6T  1.45T  /ufs/groups/manual01
groups/manual02                                2.34T  35.6T  2.33T  /ufs/groups/manual02
groups/manual03                                2.03T  35.6T  1.97T  /ufs/groups/manual03
groups/manual04                                1.90T  35.6T  1.90T  /ufs/groups/manual04
groups/manual05                                4.16T  35.6T  4.16T  /ufs/groups/manual05
groups/manual06                                1.02T  35.6T   960G  /ufs/groups/manual06
groups/manual07                                 944G  35.6T   944G  /ufs/groups/manual07
groups/manual08                                 484G  35.6T   484G  /ufs/groups/manual08
groups/manual09                                1.02G  35.6T  1.02G  /ufs/groups/manual09
groups/manual10                                2.27M  35.6T    88K  /ufs/groups/manual10
groups/manual11                                2.22M  35.6T    88K  /ufs/groups/manual11
groups/manual12                                2.22M  35.6T    88K  /ufs/groups/manual12
groups/manual13                                2.22M  35.6T    88K  /ufs/groups/manual13
groups/manual14                                2.22M  35.6T    88K  /ufs/groups/manual14
groups/manual15                                2.22M  35.6T    88K  /ufs/groups/manual15
groups/noback01                                4.20T  35.6T  4.20T  /ufs/groups/noback01
groups/noback02                                 201G  35.6T   191G  /ufs/groups/noback02
groups/noback03                                2.03T  35.6T  2.03T  /ufs/groups/noback03
groups/noback04                                13.8T  35.6T  13.8T  /ufs/groups/noback04
groups/noback05                                 595G  35.6T   553G  /ufs/groups/noback05
groups/noback06                                8.02T  35.6T  8.02T  /ufs/groups/noback06
groups/noback07                                 923G  35.6T   923G  /ufs/groups/noback07
groups/noback08                                2.28M  35.6T    88K  /ufs/groups/noback08
groups/noback09                                2.28M  35.6T    88K  /ufs/groups/noback09
groups/noback10                                2.22M  35.6T    88K  /ufs/groups/noback10
groups/noback11                                2.22M  35.6T    88K  /ufs/groups/noback11
groups/noback12                                2.22M  35.6T    88K  /ufs/groups/noback12
groups/noback13                                2.22M  35.6T    88K  /ufs/groups/noback13
groups/noback14                                2.22M  35.6T    88K  /ufs/groups/noback14
groups/noback15                                2.22M  35.6T    88K  /ufs/groups/noback15
usersz                                         17.5T  12.5T    96K  /ufs/usersz
usersz/users21                                  905G  12.5T   875G  /ufs/usersz/users21
usersz/users22                                  940G  12.5T   912G  /ufs/usersz/users22
usersz/users23                                 1.44T  12.5T  1.34T  /ufs/usersz/users23
usersz/users24                                 1.34T  12.5T  1.29T  /ufs/usersz/users24
usersz/users25                                 1.55T  12.5T  1.45T  /ufs/usersz/users25
usersz/users26                                  732G  12.5T   670G  /ufs/usersz/users26
usersz/users27                                  880G  12.5T   813G  /ufs/usersz/users27
usersz/users28                                 1.14T  12.5T  1.10T  /ufs/usersz/users28
usersz/users29                                 1.11T  12.5T  1.07T  /ufs/usersz/users29
usersz/users30                                 1.14T  12.5T  1.10T  /ufs/usersz/users30
usersz/users31                                 1.12T  12.5T  1.08T  /ufs/usersz/users31
usersz/users32                                  921G  12.5T   900G  /ufs/usersz/users32
usersz/users33                                  276G  12.5T   232G  /ufs/usersz/users33
usersz/users34                                  658G  12.5T   602G  /ufs/usersz/users34
usersz/users35                                  771G  12.5T   669G  /ufs/usersz/users35
usersz/users36                                  784G  12.5T   719G  /ufs/usersz/users36
usersz/users37                                  912G  12.5T   866G  /ufs/usersz/users37
usersz/users38                                  500G  12.5T   479G  /ufs/usersz/users38
usersz/users39                                  566G  12.5T   540G  /ufs/usersz/users39
zfsbsdtest                                      580K  7.51T    88K  /zfsbsdtest
zroot                                          44.4G   754G    88K  /zroot
zroot/ROOT                                     17.3G   754G    88K  none
zroot/ROOT/backup-1                               8K   754G  4.10G  /
zroot/ROOT/backup-2                               8K   754G  7.21G  /
zroot/ROOT/backup-3                               8K   754G  7.19G  /
zroot/ROOT/backup-4                               8K   754G  5.56G  /
zroot/ROOT/backup-5                               8K   754G  6.92G  /
zroot/ROOT/backup-6                             360K   754G  6.83G  /
zroot/ROOT/default                             17.3G   754G  10.1G  /
zroot/tmp                                      9.30G   754G  9.30G  /tmp
zroot/ufs                                        88K   754G    88K  /ufs
zroot/usr                                      3.49G   754G    88K  /usr
zroot/usr/home                                 40.9M   754G  40.8M  /usr/home
zroot/usr/ports                                2.80G   754G  2.80G  /usr/ports
zroot/usr/src                                   662M   754G   662M  /usr/src
zroot/var                                      14.3G   754G    88K  /var
zroot/var/audit                                  88K   754G    88K  /var/audit
zroot/var/crash                                14.1G   754G  14.1G  /var/crash
zroot/var/log                                   251M   754G   245M  /var/log
zroot/var/mail                                  120K   754G   120K  /var/mail
zroot/var/tmp                                  3.22M   754G  3.14M  /var/tmp

Code:
root@sol:~ # zpool status
  pool: giza
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
    still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
    the pool may no longer be accessible by software that does not support
    the features. See zpool-features(7) for details.
  scan: resilvered 21.7T in 12h19m with 0 errors on Sat Jul 21 02:10:48 2018
config:

    NAME                         STATE     READ WRITE CKSUM
    giza                         ONLINE       0     0     0
      mirror-0                   ONLINE       0     0     0
        multipath/c_giza_lun20   ONLINE       0     0     0
        multipath/i_giza_lun020  ONLINE       0     0     0
        multipath/j_giza_lun020  ONLINE       0     0     0
      mirror-1                   ONLINE       0     0     0
        multipath/c_giza_lun21   ONLINE       0     0     0
        multipath/i_giza_lun021  ONLINE       0     0     0
        multipath/j_giza_lun021  ONLINE       0     0     0
      mirror-2                   ONLINE       0     0     0
        multipath/c_giza_lun22   ONLINE       0     0     0
        multipath/i_giza_lun022  ONLINE       0     0     0
        multipath/j_giza_lun022  ONLINE       0     0     0
      mirror-3                   ONLINE       0     0     0
        multipath/c_giza_lun23   ONLINE       0     0     0
        multipath/i_giza_lun023  ONLINE       0     0     0
        multipath/j_giza_lun023  ONLINE       0     0     0
      mirror-4                   ONLINE       0     0     0
        multipath/c_giza_lun24   ONLINE       0     0     0
        multipath/i_giza_lun024  ONLINE       0     0     0
        multipath/j_giza_lun024  ONLINE       0     0     0
      mirror-5                   ONLINE       0     0     0
        multipath/c_giza_lun25   ONLINE       0     0     0
        multipath/i_giza_lun025  ONLINE       0     0     0
        multipath/j_giza_lun025  ONLINE       0     0     0
      mirror-6                   ONLINE       0     0     0
        multipath/c_giza_lun26   ONLINE       0     0     0
        multipath/i_giza_lun026  ONLINE       0     0     0
        multipath/j_giza_lun026  ONLINE       0     0     0
      mirror-7                   ONLINE       0     0     0
        multipath/c_giza_lun27   ONLINE       0     0     0
        multipath/i_giza_lun027  ONLINE       0     0     0
        multipath/j_giza_lun027  ONLINE       0     0     0

errors: No known data errors

  pool: groups
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
    still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
    the pool may no longer be accessible by software that does not support
    the features. See zpool-features(7) for details.
  scan: none requested
config:

    NAME                        STATE     READ WRITE CKSUM
    groups                      ONLINE       0     0     0
      multipath/c_groups_lun80  ONLINE       0     0     0
      multipath/c_groups_lun81  ONLINE       0     0     0
      multipath/c_groups_lun82  ONLINE       0     0     0
      multipath/c_groups_lun83  ONLINE       0     0     0
      multipath/c_groups_lun84  ONLINE       0     0     0
      multipath/c_groups_lun85  ONLINE       0     0     0
      multipath/c_groups_lun86  ONLINE       0     0     0
      multipath/c_groups_lun87  ONLINE       0     0     0
      multipath/c_groups_lun88  ONLINE       0     0     0
      multipath/c_groups_lun89  ONLINE       0     0     0
      multipath/c_groups_lun90  ONLINE       0     0     0
      multipath/c_groups_lun91  ONLINE       0     0     0
      multipath/c_groups_lun92  ONLINE       0     0     0
      multipath/c_groups_lun93  ONLINE       0     0     0
      multipath/c_groups_lun94  ONLINE       0     0     0
      multipath/c_groups_lun95  ONLINE       0     0     0
    logs
      mirror-16                 ONLINE       0     0     0
        nvd0p2                  ONLINE       0     0     0
        nvd1p2                  ONLINE       0     0     0

errors: No known data errors

  pool: usersz
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
    still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
    the pool may no longer be accessible by software that does not support
    the features. See zpool-features(7) for details.
  scan: resilvered 2.16T in 38h27m with 0 errors on Sun Jul 22 06:34:04 2018
config:

    NAME                           STATE     READ WRITE CKSUM
    usersz                         ONLINE       0     0     0
      mirror-0                     ONLINE       0     0     0
        multipath/c_usersz_lun40   ONLINE       0     0     0
        multipath/i_usersz_lun040  ONLINE       0     0     0
      multipath/c_usersz_lun41     ONLINE       0     0     0
      multipath/c_usersz_lun42     ONLINE       0     0     0
      multipath/c_usersz_lun43     ONLINE       0     0     0
      multipath/c_usersz_lun44     ONLINE       0     0     0
      multipath/c_usersz_lun45     ONLINE       0     0     0
      multipath/c_usersz_lun46     ONLINE       0     0     0
      multipath/c_usersz_lun47     ONLINE       0     0     0
    logs
      mirror-8                     ONLINE       0     0     0
        nvd0p1                     ONLINE       0     0     0
        nvd1p1                     ONLINE       0     0     0

errors: No known data errors

  pool: zfsbsdtest
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
    still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
    the pool may no longer be accessible by software that does not support
    the features. See zpool-features(7) for details.
  scan: none requested
config:

    NAME               STATE     READ WRITE CKSUM
    zfsbsdtest         ONLINE       0     0     0
      multipath/lun10  ONLINE       0     0     0
      multipath/lun11  ONLINE       0     0     0

errors: No known data errors

  pool: zroot
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
    still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
    the pool may no longer be accessible by software that does not support
    the features. See zpool-features(7) for details.
  scan: scrub repaired 0 in 0h40m with 0 errors on Sat Jun 30 20:03:12 2018
config:

    NAME        STATE     READ WRITE CKSUM
    zroot       ONLINE       0     0     0
      mfid0p3   ONLINE       0     0     0

errors: No known data errors

Code:
root@sol:~ # zpool list
NAME         SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
giza          31T  21.7T  9.32T        -         -    48%    69%  1.00x  ONLINE  -
groups       124T  84.5T  39.5T        -         -    30%    68%  1.00x  ONLINE  -
usersz        31T  17.5T  13.5T        -         -    36%    56%  1.00x  ONLINE  -
zfsbsdtest  7.75T   640K  7.75T        -         -     0%     0%  1.00x  ONLINE  -
zroot        824G  44.5G   780G        -         -     5%     5%  1.00x  ONLINE  -
 
A bunch of sysctl parameters from the server:

Code:
root@sol:~ # sysctl -a | grep vfs.nfs
vfs.nfs.downdelayinitial: 12
vfs.nfs.downdelayinterval: 30
vfs.nfs.defect: 0
vfs.nfs.iodmax: 20
vfs.nfs.iodmin: 0
vfs.nfs.iodmaxidle: 120
vfs.nfs.use_buf_pager: 0
vfs.nfs.fileid_maxwarnings: 10
vfs.nfs.diskless_rootpath: 
vfs.nfs.diskless_valid: 0
vfs.nfs.nfs_ip_paranoia: 1
vfs.nfs.nfs_directio_allow_mmap: 1
vfs.nfs.nfs_keep_dirty_on_error: 0
vfs.nfs.nfs_directio_enable: 0
vfs.nfs.clean_pages_on_close: 1
vfs.nfs.commit_on_close: 0
vfs.nfs.prime_access_cache: 0
vfs.nfs.access_cache_timeout: 60
vfs.nfs.ignore_eexist: 0
vfs.nfs.userhashsize: 100
vfs.nfs.debuglevel: 0
vfs.nfs.callback_addr: 
vfs.nfs.realign_count: 0
vfs.nfs.realign_test: 0
vfs.nfs.suppress_32bits_warning: 0
vfs.nfs.enable_uidtostring: 0
vfs.nfs.skip_wcc_data_onerr: 1
vfs.nfs.nfs3_jukebox_delay: 10
vfs.nfs.reconnects: 0
vfs.nfs.bufpackets: 4
vfs.nfsd.fha.fhe_stats: No file handle entries.
vfs.nfsd.fha.max_reqs_per_nfsd: 0
vfs.nfsd.fha.max_nfsds_per_fh: 8
vfs.nfsd.fha.bin_shift: 22
vfs.nfsd.fha.write: 1
vfs.nfsd.fha.read: 1
vfs.nfsd.fha.enable: 1
vfs.nfsd.request_space_throttle_count: 0
vfs.nfsd.request_space_throttled: 0
vfs.nfsd.request_space_low: 8372464298
vfs.nfsd.request_space_high: 12558696448
vfs.nfsd.request_space_used_highest: 19262812
vfs.nfsd.request_space_used: 0
vfs.nfsd.groups: 1
vfs.nfsd.threads: 4
vfs.nfsd.maxthreads: 4
vfs.nfsd.minthreads: 4
vfs.nfsd.cachetcp: 1
vfs.nfsd.tcpcachetimeo: 1000
vfs.nfsd.udphighwater: 500
vfs.nfsd.tcphighwater: 50000
vfs.nfsd.enable_stringtouid: 0
vfs.nfsd.debuglevel: 0
vfs.nfsd.enable_locallocks: 0
vfs.nfsd.issue_delegations: 0
vfs.nfsd.commit_miss: 0
vfs.nfsd.commit_blks: 0
vfs.nfsd.mirrormnt: 1
vfs.nfsd.async: 0
vfs.nfsd.server_max_nfsvers: 4
vfs.nfsd.server_min_nfsvers: 2
vfs.nfsd.nfs_privport: 0
vfs.nfsd.allowreadforwriteopen: 1
vfs.nfsd.writedelegifpos: 0
vfs.nfsd.v4statelimit: 500000
vfs.nfsd.sessionhashsize: 20
vfs.nfsd.fhhashsize: 20
vfs.nfsd.clienthashsize: 20
vfs.nfsd.statehashsize: 10
vfs.nfsd.enable_nogroupcheck: 1
vfs.nfsd.enable_nobodycheck: 1
vfs.nfsd.enable_checkutf8: 1
 
Don't use the sharenfs feature of ZFS and /etc/exports. Use one or the other, not both. A lot of NFS exports are now doubled, that's going to cause issues.

Hmm, this was confusing to me. I think our exports files are OK: the first file is mostly nfsv3 exports to webservers and the second is nfsv4 exports, so nothing is really doubled is it??

We tried increasing the number of threads to 16, unfortunately the server crashed again today after 3-4 days...
 
On the problem I cannot give much help. Still did not succeed with nfsv4, being home-user.

to your posted files:

in
/etc/exports:
/ufs/groups/groups07
and in
/etc/zfs/exports:
/ufs/groups/groups07
is at least one double shared path, there are some more.

As I just tested
/etc/zfs/exports gets created by FreeBSD when executing
zfs set sharenfs=on zpool/export/dataset

Just like SirDice and ucomp wrote.

So if I understand them right, the advice is something like: set all zfs datasets to sharenfs=off.

As I understand it, the zfs came from solaris, which implemented the sharesmb and sharenfs property.
But the sharing in FreeBSD was implemented differently than in solaris, when zfs came into FreeBSD.
 
On the problem I cannot give much help. Still did not succeed with nfsv4, being home-user.

to your posted files:

in
/etc/exports:
/ufs/groups/groups07
and in
/etc/zfs/exports:
/ufs/groups/groups07
is at least one double shared path, there are some more.

Yes but the path definitions are different:

/etc/zfs/exports:
/ufs/groups/groups07 -sec=krb5p

/etc/exports:
/ufs/groups/groups07 -alldirs fox.xyz.ourdomain.se elk.xyz.ourdomain.se cat.xyz.ourdomain.se fly.xyz.ourdomain.se dog.xyz.ourdomain.se

I think the first line is only valid in an nfsv4 context? If it works that way they are not really doubled.
 
Now, this is just a general comment but I see this happening so many times...

Why all the fine tuning on such a system? When looking at /etc/sysctl.conf and /boot/loader.conf (to a lesser degree) I see several changed system settings of which I wonder why you'd even bother with those?

The one thing I often see from people who migrate from Linux is that they start treating other environments as if it was Linux. The kernel? Obviously that needs to be micro managed because on Linux you can often get a better performance by optimizing your kernel. Only thing is that this doesn't apply to an OS such as FreeBSD. In fact: in most cases you'll get better results from simply using the standard GENERIC kernel. For starters it'll save you the time to build the whole thing.

But this also applies to system settings. Many Linux admins set certain values because they think they obviously need 'm because that's how it used to work on Linux. Or they read about it on the Internet somewhere and assumed it will also apply to them. But why? FreeBSD isn't Linux so don't treat it as such: if it isn't broke, don't try to fix it.

Don't assume that you need to increase the message buffer because of your workload. Test, monitor and then change accordingly if needed. Don't raise cache sizes to "increase your performance", I'll bet you don't even know if the standard performance would be suitable or not in the first place.

Now, don't get me wrong... I'm not saying that all this is the cause of your problems. In fact I'm sharing quite a bit of assumptions myself here as well. But I do seriously wonder if all that tweaking didn't cause (some of) the problems in the first place, or affect it.

If you haven't done this already then I'd seriously consider to double check on why you think you need those system config changes and if you can't directly tie them into actual problems then get rid of those and let the system handle it.
 
Now, this is just a general comment but I see this happening so many times...

Why all the fine tuning on such a system? When looking at /etc/sysctl.conf and /boot/loader.conf (to a lesser degree) I see several changed system settings of which I wonder why you'd even bother with those?

/etc/sysctl.conf:

the "vfs.nfsd...." lines were put there because we had certain problems. The man page for nfsd told us that these lines would solve the problem, and they did!

"If the server has stopped servicing clients and has generated a console
message like "nfsd server cache flooded...", the value for
vfs.nfsd.tcphighwater needs to be increased. This should allow the
server to again handle requests without a reboot. Also, you may want to
consider decreasing the value for vfs.nfsd.tcpcachetimeo to several
minutes (in seconds) instead of 12 hours when this occurs."

I don't know why we use the other settings, maybe I can find out.

The one thing I often see from people who migrate from Linux is that they start treating other environments as if it was Linux. The kernel? Obviously that needs to be micro managed because on Linux you can often get a better performance by optimizing your kernel. Only thing is that this doesn't apply to an OS such as FreeBSD. In fact: in most cases you'll get better results from simply using the standard GENERIC kernel. For starters it'll save you the time to build the whole thing.

For this server we have migrated from Solaris, not Linux!
 
(I know this is a year-old thread but I just came by it and figured someone else might too)...

I notice that you're using Kerberos and NFSv4 - there is a kernel-panic causing problem with the GSS credentials cache in the 11.2 kernel (a race condition) that typically only shows itself when there are many concurrent clients connection (in our case it started happening more frequently around 150 Linux clients).

This bug is fixed in 11.3 or you can build your own kernel with a patch (see the attached file for one version).

With many users you also probably also want to increase the "CLIENT_MAX" define from the default 128 in "/usr/src/sys/rpc/rpcsec_gss/svc_rpcsec_gss.c to something bigger (is tunable in 11.3). I set it to 1024 in the patch. You also need to add a "#define KERN_GSS_DEBUG 38" to /usr/src/sys/sys/sysctl.h for the patch to work/build.
 

Attachments

  • svc_rpcsec_gss.c.patch.txt
    12.7 KB · Views: 185
Back
Top