Reconfiguration of Squid with many authorization helpers.

Hi every one. I using squid version 3.1 at my work for Internet access. My squid doesn't want to run with more than 127 (exactly 127) authorization helpers. I have try to fix it by myself, but I can't anymore. I have approximately 500 users and some times squid simply goes down.
It shows me this in it logs.

Code:
FATAL: Too many queued ntlmauthenticator requests
Squid Cache (Version 3.1.0.14): Terminated abnormally.
CPU Usage: 5233.065 seconds = 4597.012 user + 636.053 sys
Maximum Resident Size: 352560 KB
Page faults with physical i/o: 0

Well, the problem is when i try to increase an amount of helpers,
it runs well for the first time, but when i try to reconfigure it, It stop working. And shows me this

Code:
2009/11/15 11:45:52| Adding domain test.tv from /etc/resolv.conf
2009/11/15 11:45:52| Adding nameserver 10.159.4.2 from /etc/resolv.conf
2009/11/15 11:45:52| Adding nameserver 10.159.7.34 from /etc/resolv.conf
2009/11/15 11:45:52| helperOpenServers: Starting 250/250
'ntlm_smb_lm_auth' processes
2009/11/15 11:45:59| helperOpenServers: Starting 20/20 'ntlm_auth' processes
2009/11/15 11:46:00| Accepting  HTTP connections at 192.168.7.10:3128, FD 1062.
2009/11/15 11:46:00| HTCP Disabled.
2009/11/15 11:46:00| Accepting SNMP messages on 0.0.0.0:3401, FD 1063.
2009/11/15 11:46:00| Loaded Icons.
2009/11/15 11:46:00| Ready to serve requests.
2009/11/15 11:46:00| Select loop Error. Retry

After this I need to rerun it.

I'm sure that the problem is in my environment or system settings. Take a look at my system parameters

Code:
[B]bash-2.05b# top | head[/B]
last pid: 24842;  load averages:  1.27,  1.35,  1.36  up 84+22:17:30    15:08:54
201 processes: 2 running, 198 sleeping, 1 zombie

Mem: 528M Active, 194M Inact, 221M Wired, 49M Cache, 111M Buf, 3740K Free
Swap: 1024M Total, 12M Used, 1012M Free, 1% Inuse

[B]bash-2.05b# uname -a[/B]
FreeBSD proxy.tsb.kz 6.2-RELEASE FreeBSD 6.2-RELEASE #0: Fri Jan 12 11:05:30 UTC 2007 
root@dessler.cse.buffalo.edu:/usr/obj/usr/src/sys/SMP  i386
[B]bash-2.05b# gcc -v[/B]
Using built-in specs.
Configured with: FreeBSD/i386 system compiler
Thread model: posix
gcc version 3.4.6 [FreeBSD] 20060305
[B]bash-2.05b# sysctl kern | grep max[/B]
kern.maxvnodes: 69979
kern.maxproc: 6164
kern.maxfiles: 65535
kern.argmax: 262144
kern.maxfilesperproc: 32768
kern.maxprocperuid: 16384
kern.ipc.maxsockbuf: 2097152
kern.ipc.somaxconn: 128
kern.ipc.max_linkhdr: 16
kern.ipc.max_protohdr: 60
kern.ipc.max_hdr: 76
kern.ipc.max_datalen: 132
kern.ipc.maxpipekva: 16777216
kern.ipc.msgmax: 16384
kern.ipc.shmmax: 33554432
kern.ipc.maxsockets: 25600
kern.iov_max: 1024
kern.cam.cd.changer.max_busy_seconds: 15
kern.kq_calloutmax: 4096
kern.maxusers: 384
kern.threads.max_threads_per_proc: 40000
kern.threads.max_groups_per_proc: 40000
kern.threads.max_threads_hits: 0
kern.smp.maxcpus: 16
[B]bash-2.05b# limits [/B]
Resource limits (current):
  cputime          infinity secs
  filesize         infinity kB
  datasize           524288 kB
  stacksize           65536 kB
  coredumpsize     infinity kB
  memoryuse        infinity kB
  memorylocked     infinity kB
  maxprocesses        16384
  openfiles           32768
  sbsize           infinity bytes
  vmemoryuse       infinity kB

# [b]squid -v[/b]
Code:
Squid Cache: Version 3.1.0.14
configure options:  '--with-default-user=squid' '--bindir=/usr/local/sbin' '--sbindir=/usr/local/sbin' '--datadir=/usr/local/etc/squid' 
'--libexecdir=/usr/local/libexec/squid' '--localstatedir=/usr/local/squid' '--sysconfdir=/usr/local/etc/squid' '--with-logdir=/usr/local/squid
/logs' '--with-pidfile=/usr/local/squid/squid.pid' '--enable-removal-policies=lru heap' '--disable-linux-netfilter' '--disable-linux-tproxy' 
'--disable-epoll' '--disable-translation' '--with-filedescriptors=32768' '--enable-auth=basic digest negotiate ntlm' '--enable-basic-
auth-helpers=DB NCSA PAM MSNT SMB squid_radius_auth LDAP SASL YP' '--enable-digest-auth-helpers=password ldap' '--enable-external-
acl-helpers=ip_user session unix_group wbinfo_group ldap_group' '--enable-ntlm-auth-helpers=smb_lm' '--enable-negotiate-
auth-helpers=squid_kerb_auth' '--enable-storeio=ufs diskd aufs' '--enable-disk-io=AIO Blocking DiskDaemon DiskThreads' '--disable-ipv6' 
'--enable-delay-pools' '--enable-ssl' '--with-openssl=/usr' '--enable-cache-digests' '--enable-arp-acl' '--enable-ipfw-transparent' '--enable-
follow-x-forwarded-for' '--enable-kqueue' '--with-large-files' '--enable-stacktraces' '--prefix=/usr/local' '--mandir=/usr/local/man' 
'--infodir=/usr/local/info/' '--build=i386-portbld-freebsd6.2' 'build_alias=i386-portbld-freebsd6.2' 'CC=cc' 'CFLAGS=-O2 -fno-strict-aliasing 
-pipe -I/usr/local/include -I/usr/local/include  -I/usr/include -g -DLDAP_DEPRECATED' 'LDFLAGS= -L/usr/local/lib -L/usr/local/lib -rpath=/usr
/lib:/usr/local/lib -L/usr/lib' 'CPPFLAGS=-I/usr/local/include' 'CXX=c++' 'CXXFLAGS=-O2 -fno-strict-aliasing -pipe -I/usr/local/include -I/usr
/local/include -I/usr/include -g -DLDAP_DEPRECATED' --with-squid=/usr/ports/www/squid31/work/squid-3.1.0.14 --enable-ltdl-convenience


Here you can see that there is no any limit for fd
Code:
File descriptor usage for squid:
	Maximum number of file descriptors:   32768
	Largest file desc currently in use:    452
	Number of file desc currently in use:  331
	Files queued for open:                   0
	Available number of file descriptors: 32437
	Reserved number of file descriptors:   100
	Store Disk files open:                   6


Maybe there is some parameters that i don't know?

PS: It happens with my home computer (FreeBSD 8.0-RELEASE ) too.

Many Thanks.
 
Well, Are there no any advices? Maybe You will give me direction? I'm trying to find the solution for a few months. =(
 
DutchDaemon said:
You might try a slightly more active approach and visit or post on Squid mailing lists or forums. This is not a specific FreeBSD problem, and I assume that the combination FreeBSD / Squid / NTLM is not exactly commonplace around here.

http://www.squid-cache.org/Support/mailing-lists.dyn
http://old.nabble.com/Squid-Web-Proxy-Cache-f22481.html

I have already asked it at the Squid Mailing list, but no result. Amos have sent me his cache.log and some information that I show You before. I have build squid with exactly the same parameters, but it doesn't resolve my problem.

Why you so sure that it's not FreeBSD problem?
 
Levenson said:
Well the problem is in the squid experimental feature - kqueue.

So specifically the interaction kqueue < - > authorization helpers? Because I think I've used the kqueue option (without problems) ever since it became available, but never with any authorization helpers.
 
Back
Top