Hi,
Part of my job is supporting Linux users at a university, and we have recently migrated one of our file servers from Solaris to FreeBSD.
We've had some problems and hence I'm posting here.
HW:
CPU: Intel(R) Xeon(R) CPU E5-2667 v4 @ 3.20GHz (3200.07-MHz K8-class CPU)
real memory = 412304277504 (393204 MB)
The server is running zfs, samba & nfsv4.
FreeBSD 11.2.
/etc/rc.conf:
/etc/nscd.conf:
/etc/nsswitch.conf:
NOTE: We don't run the nfscbd daemon.
320 Linux clients, mostly Redhat Enterprise 7 but also approx. 100 RHEL 6.
Also an unknown number of windows samba clients.
Main applications on Linux: GNOME, Matlab, Firefox.
The server delivers NFSv4 + Samba homedirectories, group file directories and
various nfs shares with software.
Samba is mainly for group file directories, not homedirectories.
Mount flags on RHEL 7 clients:
Flags: rw,nosuid,nodev,relatime,vers=4.1,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=krb5p,clientaddr=x.x.x.x,local_lock=none,addr=x.x.x.x
PROBLEMS:
NOTE: I'm not the admin for this server, just a curious second line support person,
so the description below is going to be a bit vague as I have not been involved
in much of the actual troubleshooting.
The server has been running for approx. six months. The server performed well
during the first months, probably because the weren't many active NFS clients.
Apart from a few isolated incidents samba has caused no problems.
The server runs in a school environment and the problems started to appear when many students at once started
to login to linux computers in classrooms. In the beginning it caused the server to crash & re-boot, it was not able
to start the nfs-service again without some persuasion. The load was not high on the server but nscd & gssd was using a lot
of cpu.
The situation improved when we:
-decreased the number of groups in /etc/group from approx. 155000 to 38000.
-only used "files" for the group entry in nsswitch.conf instead of "cache files"
-installed a "nanny" script that re-starts gssd when it stops working
Now the server is able to start the nfs service again after a crash, in fact hasn't crashed
so much lately, perhaps because students/teachers have given up on using the Linux clients
for labs...;-(
QUESTIONS:
I wonder if there is any configuration flags to nfsd, nfsuserd or nscd that could be used
or set differently?
How many NFSv4 workstation clients should a single server be able to handle?
Threads: how many threads should the nfs daemon use? I've seen conflicting opinions about this...
Currently we have it set to 4, when we used higher numbers like 50 the server would run well but
crash as soon as many nfs clients became active (= students login in to comps.)
Would it be better to use ldap instead of passwd/group files (and skip nscd)?
I would be grateful for any input!
Part of my job is supporting Linux users at a university, and we have recently migrated one of our file servers from Solaris to FreeBSD.
We've had some problems and hence I'm posting here.
HW:
CPU: Intel(R) Xeon(R) CPU E5-2667 v4 @ 3.20GHz (3200.07-MHz K8-class CPU)
real memory = 412304277504 (393204 MB)
The server is running zfs, samba & nfsv4.
FreeBSD 11.2.
/etc/rc.conf:
Code:
zfs_enable="YES"
# NFS
nfs_server_enable="YES"
nfs_server_flags="-u -t -n 4" # Flags to nfsd (if enabled).
nfsv4_server_enable="YES" # Enable support for NFSv4
nfsuserd_enable="YES" # NFSv4 user/group name mapping daemon
nfsuserd_flags="-domain ourdomain.se -usermax 5000 20"
gssd_enable="YES" # Run the gssd daemon (or NO).
gssd_flags=""
#Linux compat
linux_enable="YES"
#Autofs
autofs_enable="YES"
autounmountd_flags="-r 200 -t 200 -v"
automountd_flags="-v"
# Samba (in /usr/local)
samba_server_enable="YES"
winbindd_enable="YES"
nmbd_enable="NO"
# nscd
nscd_enable="YES"
#TSM
dsmcad_enable="YES"
# PF Firewall (from OpenBSD 4.5) see /etc/pf.conf
pf_enable="YES"
# inetd (needed for rquouta)
inetd_enable="YES"
/etc/nscd.conf:
Code:
#
# Default caching daemon configuration file
# $FreeBSD: releng/11.2/etc/nscd.conf 172742 2007-10-18 08:26:20Z bushman $
#
#
enable-cache passwd yes
enable-cache group yes
enable-cache hosts yes
enable-cache services yes
enable-cache protocols yes
enable-cache rpc yes
enable-cache networks yes
#
suggested-size passwd 19997
suggested-size group 19997
keep-hot-count passwd 163840
keep-hot-count group 163840
/etc/nsswitch.conf:
Code:
#
# nsswitch.conf(5) - name service switch configuration file
# $FreeBSD: releng/11.2/etc/nsswitch.conf 301711 2016-06-09 01:28:44Z markj $
#
group: files
hosts: files dns
netgroup: compat
networks: files
passwd: cache files
shells: files
services: files
protocols: files
rpc: files
NOTE: We don't run the nfscbd daemon.
Code:
# wc -l /etc/group
34371 /etc/group
# wc -l /etc/passwd
124167 /etc/passwd
320 Linux clients, mostly Redhat Enterprise 7 but also approx. 100 RHEL 6.
Also an unknown number of windows samba clients.
Main applications on Linux: GNOME, Matlab, Firefox.
The server delivers NFSv4 + Samba homedirectories, group file directories and
various nfs shares with software.
Samba is mainly for group file directories, not homedirectories.
Mount flags on RHEL 7 clients:
Flags: rw,nosuid,nodev,relatime,vers=4.1,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=krb5p,clientaddr=x.x.x.x,local_lock=none,addr=x.x.x.x
PROBLEMS:
NOTE: I'm not the admin for this server, just a curious second line support person,
so the description below is going to be a bit vague as I have not been involved
in much of the actual troubleshooting.
The server has been running for approx. six months. The server performed well
during the first months, probably because the weren't many active NFS clients.
Apart from a few isolated incidents samba has caused no problems.
The server runs in a school environment and the problems started to appear when many students at once started
to login to linux computers in classrooms. In the beginning it caused the server to crash & re-boot, it was not able
to start the nfs-service again without some persuasion. The load was not high on the server but nscd & gssd was using a lot
of cpu.
The situation improved when we:
-decreased the number of groups in /etc/group from approx. 155000 to 38000.
-only used "files" for the group entry in nsswitch.conf instead of "cache files"
-installed a "nanny" script that re-starts gssd when it stops working
Now the server is able to start the nfs service again after a crash, in fact hasn't crashed
so much lately, perhaps because students/teachers have given up on using the Linux clients
for labs...;-(
QUESTIONS:
I wonder if there is any configuration flags to nfsd, nfsuserd or nscd that could be used
or set differently?
How many NFSv4 workstation clients should a single server be able to handle?
Threads: how many threads should the nfs daemon use? I've seen conflicting opinions about this...
Currently we have it set to 4, when we used higher numbers like 50 the server would run well but
crash as soon as many nfs clients became active (= students login in to comps.)
Would it be better to use ldap instead of passwd/group files (and skip nscd)?
I would be grateful for any input!