FreeBSD 10 maxproc error every night during periodic daily

romeor · Mar 13, 2014

Hi,

I've been looking for information on this using this forum and google. Nothing useful found.
So I'd better ask here to save time.

Every night there is a periodic job in /etc/crontab, which runs through lots of scripts to generate reports of different kinds. One of them is /etc/periodic/security/100.chksetuid.
This script runs two commands: first one is find, the other one is a function ( check_diff()) that could be found in /etc/periodic/security/security.functions . So the first part seems to run smoothly and the find results in this:

Code:

# find -sx / /dev/null -type f \( -perm -u+x -or -perm -g+x -or -perm -o+x \) \( -perm -u+s -or -perm -g+s \) -exec ls -liTd \{\} +
 19903508 -r-sr-xr-x  1 root  wheel      19912 Jan 17 00:40:07 2014 /bin/rcp
113883305 -r-sr-x---  1 root  operator    9880 Jan 17 00:40:33 2014 /sbin/mksnap_ffs
113883265 -r-sr-xr-x  1 root  wheel      28024 Jan 17 00:40:34 2014 /sbin/ping
113883340 -r-sr-xr-x  1 root  wheel      36496 Jan 17 00:40:34 2014 /sbin/ping6
113883322 -r-sr-x---  2 root  operator   15656 Jan 17 00:40:34 2014 /sbin/poweroff
113883322 -r-sr-x---  2 root  operator   15656 Jan 17 00:40:34 2014 /sbin/shutdown
 58747459 -r-sr-xr-x  4 root  wheel      28528 Jan 17 00:41:00 2014 /usr/bin/at
 58747459 -r-sr-xr-x  4 root  wheel      28528 Jan 17 00:41:00 2014 /usr/bin/atq
 58747459 -r-sr-xr-x  4 root  wheel      28528 Jan 17 00:41:00 2014 /usr/bin/atrm
 58747459 -r-sr-xr-x  4 root  wheel      28528 Jan 17 00:41:00 2014 /usr/bin/batch
 58747664 -r-xr-sr-x  1 root  kmem       12544 Jan 17 00:41:01 2014 /usr/bin/btsockstat
 58747551 -r-sr-xr-x  6 root  wheel      22376 Jan 17 00:41:02 2014 /usr/bin/chfn
 58747551 -r-sr-xr-x  6 root  wheel      22376 Jan 17 00:41:02 2014 /usr/bin/chpass
 58747551 -r-sr-xr-x  6 root  wheel      22376 Jan 17 00:41:02 2014 /usr/bin/chsh
 58747681 -r-sr-xr-x  1 root  wheel      32624 Jan 17 00:41:24 2014 /usr/bin/crontab
 58747500 -r-sr-xr-x  1 root  wheel      11440 Jan 17 00:41:07 2014 /usr/bin/lock
 58747523 -r-sr-xr-x  1 root  wheel      24752 Jan 17 00:41:07 2014 /usr/bin/login
 58747616 -r-sr-sr-x  1 root  daemon     32896 Jan 17 00:41:27 2014 /usr/bin/lpq
 58747513 -r-sr-sr-x  1 root  daemon     38248 Jan 17 00:41:27 2014 /usr/bin/lpr
 58747707 -r-sr-sr-x  1 root  daemon     31064 Jan 17 00:41:27 2014 /usr/bin/lprm
 58747558 -r-xr-sr-x  1 root  kmem      141288 Jan 17 00:41:09 2014 /usr/bin/netstat
 58747462 -r-sr-xr-x  1 root  wheel       7104 Jan 17 00:41:09 2014 /usr/bin/opieinfo
 58747538 -r-sr-xr-x  1 root  wheel      13584 Jan 17 00:41:09 2014 /usr/bin/opiepasswd
 58747417 -r-sr-xr-x  2 root  wheel       8296 Jan 17 00:41:09 2014 /usr/bin/passwd
 58747746 -r-sr-xr-x  1 root  wheel      15648 Jan 17 00:41:10 2014 /usr/bin/quota
 58747475 -r-sr-xr-x  1 root  wheel      15608 Jan 17 00:41:10 2014 /usr/bin/rlogin
 58747629 -r-sr-xr-x  1 root  wheel      11432 Jan 17 00:41:10 2014 /usr/bin/rsh
 58747560 -r-sr-xr-x  1 root  wheel      17216 Jan 17 00:41:11 2014 /usr/bin/su
 58747541 -r-xr-sr-x  1 root  tty        16088 Jan 17 00:41:14 2014 /usr/bin/wall
 58747458 -r-xr-sr-x  1 root  tty        11880 Jan 17 00:41:15 2014 /usr/bin/write
 58747551 -r-sr-xr-x  6 root  wheel      22376 Jan 17 00:41:02 2014 /usr/bin/ypchfn
 58747551 -r-sr-xr-x  6 root  wheel      22376 Jan 17 00:41:02 2014 /usr/bin/ypchpass
 58747551 -r-sr-xr-x  6 root  wheel      22376 Jan 17 00:41:02 2014 /usr/bin/ypchsh
 58747417 -r-sr-xr-x  2 root  wheel       8296 Jan 17 00:41:09 2014 /usr/bin/yppasswd
 58836124 -r-xr-sr-x  1 root  smmsp     676064 Jan 17 00:41:34 2014 /usr/libexec/sendmail/sendmail
 58750673 -r-sr-xr-x  1 root  wheel      32824 Jan 17 00:40:38 2014 /usr/libexec/ssh-keysign
 58750669 -r-sr-xr-x  1 root  wheel       6000 Jan 17 00:40:05 2014 /usr/libexec/ulog-helper
 58929023 -rwsr-xr-x  1 root  wheel     340384 Dec 12 19:52:03 2013 /usr/local/bin/screen
 58929022 -rwxr-sr-x  1 root  kmem      125176 Mar  6 09:05:48 2014 /usr/local/sbin/lsof
 58750485 -r-sr-sr-x  2 root  authpf     24160 Jan 17 00:41:18 2014 /usr/sbin/authpf
 58750485 -r-sr-sr-x  2 root  authpf     24160 Jan 17 00:41:18 2014 /usr/sbin/authpf-noip
 58750423 -r-xr-sr-x  1 root  daemon     55584 Jan 17 00:41:27 2014 /usr/sbin/lpc
 58750497 -r-sr-x---  1 root  network   407920 Jan 17 00:41:32 2014 /usr/sbin/ppp
 58750483 -r-sr-xr-x  1 root  wheel      20792 Jan 17 00:41:34 2014 /usr/sbin/timedc
 58750639 -r-sr-xr-x  1 root  wheel      28296 Jan 17 00:41:34 2014 /usr/sbin/traceroute
 58750519 -r-sr-xr-x  1 root  wheel      23920 Jan 17 00:41:34 2014 /usr/sbin/traceroute6
 58750582 -r-xr-sr-x  1 root  kmem       11552 Jan 17 00:41:35 2014 /usr/sbin/trpt

and it seem, that right after find was successful, i get the
maxproc limit exceeded by uid 0 (pid 14825); see tuning(7) and login.conf(5)
error and system becomes not usable until hard restart.

This is the fresh install of FreeBSD10 with gmirror and ufs.

And the result of top during this action:

Code:

13469 processes:1 running, 13468 sleeping
CPU:  0.1% user,  0.0% nice,  7.1% system,  0.3% interrupt, 92.6% idle
Mem: 4274M Active, 193M Inact, 1937M Wired, 820M Buf, 1468M Free
Swap: 8192M Total, 8192M Free

  PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
 2194 root          1  24    0 17576K  3236K fork    2   0:02   9.18% bash
 1143 root          1  26    0 56632K 38364K CPU6    6   0:04   7.57% top
 1137 root          1  24    0 56632K 38924K select  3   0:04   4.79% top
16392 root          1  52    0 16988K  2364K wait    3   0:00   0.68% sh
16331 root          1  52    0 16988K  2364K wait    5   0:00   0.68% sh
16323 root          1  52    0 16988K  2352K wait    2   0:00   0.68% sh
16302 root          1  52    0 16988K  2352K wait    2   0:00   0.68% sh
16219 root          1  52    0 16988K  2352K wait    2   0:00   0.68% sh
16270 root          1  52    0 16988K  2352K wait    2   0:00   0.68% sh
16293 root          1  52    0 16988K  2352K wait    2   0:00   0.68% sh
16290 root          1  52    0 16988K  2352K wait    2   0:00   0.68% sh
16228 root          1  52    0 16988K  2352K wait    2   0:00   0.68% sh
16312 root          1  52    0 16988K  2352K wait    2   0:00   0.68% sh
16226 root          1  52    0 16988K  2352K wait    2   0:00   0.68% sh
16330 root          1  52    0 16988K  2352K wait    2   0:00   0.68% sh
16224 root          1  52    0 16988K  2352K wait    2   0:00   0.68% sh
16244 root          1  52    0 16988K  2352K wait    2   0:00   0.68% sh
16238 root          1  52    0 16988K  2352K wait    2   0:00   0.68% sh
16314 root          1  52    0 16988K  2352K wait    2   0:00   0.68% sh
16231 root          1  52    0 16988K  2352K wait    2   0:00   0.68% sh
16276 root          1  52    0 16988K  2352K wait    2   0:00   0.68% sh
16251 root          1  52    0 16988K  2352K wait    2   0:00   0.68% sh
16235 root          1  52    0 16988K  2352K wait    2   0:00   0.68% sh
16294 root          1  52    0 16988K  2352K wait    4   0:00   0.68% sh
16245 root          1  52    0 16988K  2352K wait    6   0:00   0.68% sh
16329 root          1  52    0 16988K  2352K wait    6   0:00   0.68% sh
16315 root          1  52    0 16988K  2352K wait    6   0:00   0.68% sh
16275 root          1  52    0 16988K  2352K wait    7   0:00   0.68% sh
16239 root          1  52    0 16988K  2352K wait    4   0:00   0.68% sh
16232 root          1  52    0 16988K  2352K wait    6   0:00   0.68% sh
16325 root          1  52    0 16988K  2352K wait    1   0:00   0.68% sh
16227 root          1  52    0 16988K  2352K wait    0   0:00   0.68% sh
16220 root          1  52    0 16988K  2352K wait    7   0:00   0.68% sh
16269 root          1  52    0 16988K  2352K wait    4   0:00   0.68% sh
16303 root          1  52    0 16988K  2352K wait    5   0:00   0.68% sh
16317 root          1  52    0 16988K  2352K wait    7   0:00   0.68% sh
16277 root          1  52    0 16988K  2352K wait    4   0:00   0.68% sh
16271 root          1  52    0 16988K  2352K wait    5   0:00   0.68% sh
16229 root          1  52    0 16988K  2352K wait    1   0:00   0.68% sh
16230 root          1  52    0 16988K  2352K wait    7   0:00   0.68% sh
16316 root          1  52    0 16988K  2352K wait    4   0:00   0.68% sh
16252 root          1  52    0 16988K  2352K wait    0   0:00   0.68% sh
16324 root          1  52    0 16988K  2352K wait    5   0:00   0.68% sh
16313 root          1  52    0 16988K  2352K wait    7   0:00   0.68% sh
16295 root          1  52    0 16988K  2352K wait    6   0:00   0.68% sh
16225 root          1  52    0 16988K  2352K wait    6   0:00   0.68% sh
16301 root          1  52    0 16988K  2352K wait    4   0:00   0.68% sh
16284 root          1  52    0 16988K  2352K wait    6   0:00   0.68% sh

Any ideas or hints how to fix that?

tingo · Mar 13, 2014

Re: FreeBSD 10 maxproc error every night during periodic dai

Well, did you read the tuning(7) man page?

wblock@ · Mar 13, 2014

Re: FreeBSD 10 maxproc error every night during periodic dai

What do you have in /boot/loader.conf, /etc/rc.conf, and /etc/sysctl.conf?

romeor · Mar 14, 2014

Re: FreeBSD 10 maxproc error every night during periodic dai

tingo
Sure I did. And it has nothing about it. Default FreeBSD installation with Generic kernel has enough resources.

Well, this topic could be closed now. In my /etc/rc.conf I had two lines:

Code:

firewall_type="OPEN"
firewall_script="/etc/ipfw"

So during one of these failures (I changed time to run periodic) I've seen, that these sh processes are /bin/sh /sbin/ipfw -q -f flush. So I removed the firewall_type="OPEN" line and renamed IPFW script to /etc/ipfw/ipfw.fw (I made the change in /etc/rc.conf also).
After that it runs smoothly. Pretty ridiculous thing, but well, that was the way it worked :OOO . Something spawned lots of /bin/sh /sbin/ipfw -q -f flush commands from periodic runs.

FreeBSD 10 maxproc error every night during periodic daily

romeor

tingo

wblock@

romeor