Apache: Help with rogue processes

[biscuits]

Ive got a FreeBSD 7.0 machine running Apache 2.0 in our office that serves a large web application by everyone in the office. Normally this server will run about 60 odd processes but lately its been spiking at 300+ processes about 2-3 times a week randomly. (This exceeds the 250 max connections and crashes the HTML App).

Top basically looks like:

last pid: 40136; load averages: 0.63, 0.33, 0.26 up 11+01:02:16 15:01:05
300 processes: 1 running, 299 sleeping
CPU states: 1.7% user, 0.0% nice, 1.1% system, 0.0% interrupt, 97.2% idle
Mem: 302M Active, 2908M Inact, 495M Wired, 8508K Cache, 214M Buf, 186M Free
Swap: 4096M Total, 80K Used, 4096M Free


I've tried setting up "apachectl fullstatus" on a test machine (apache 1.3, freebsd 7.0) but everytime I run the command in the CLI all I get is:

Index of /server-status

[1]Name [2]Last modified [3]Size [4]Descriptio
n
__________________________________________________ _________________________

[DIR] [5]Parent Directory 02-Feb-2009 15:35 -
[ ] [6]test.htm 02-Feb-2009 16:09 1k
__________________________________________________ ________________


Apache/1.3.37 Server at Virtual_BSD.my.domain Port 80

References

1. http://Virtual_BSD.my.domain/server-status/?N=D
2. http://Virtual_BSD.my.domain/server-status/?M=A
3. http://Virtual_BSD.my.domain/server-status/?S=A
4. http://Virtual_BSD.my.domain/server-status/?D=A
5. http://Virtual_BSD.my.domain/
6. http://Virtual_BSD.my.domain/server-status/test.htm


I'm guessing I've set this up wrong because its only showing the data/service-status/ directory instead of all the folder sin data/. Either way, its not telling me much.

Is there any way to see what page all these httpd processes are getting hung up on? I've tried a ps auxww | grep httpd but that didnt tell me much either.

Any ideas what else to try (or where I've gone wrong maybe?)

[Mel_Flynn]

Your test machine doesn't have those requests. The server-status module shows whatever the server is doing now. In case a process is idle, it will show the last served request.
Run it on the live server when you have the load spike and you'll see all 300 active connections. Of course, tail -f /var/log/httpd-access.log will give you an indication as well.

[r-c-e]

what does http://server.com/server-status show you?

[biscuits]

> what does http://server.com/server-status show you?

nothing as I havent got an internal domain setup in our office so only CLI command is available unless I go and set that up. (have no GUI setup on this server box either). Accessing http://192.168.1.1/server-status just gives me the directory contents (guessing it actually needs a domain in there for apache to parse correctly).

So is there no other way of finding out what 100-odd httpd processes that have sitting dormant for say the last hour or two in 'top' are doing / where they came from? Do they all sit there purely to manage sessions maybe?

Cheers,
Dave.

[r-c-e]

There is a way, you're just not enabling it

turn extendedstatus on

also you can do some tracing


mount -t procfs /proc /proc
truss -p xxxx


where xxxx is the pid

[biscuits]

I figured out why it wasnt working - I mistakenly created a directory called "server-status" in my www/ folder ... whoops :o)

Anyway, apachectl fullstatus now works properly - and im now chasing leads in keepalive settings in order to solve my maxclients issue.

Cheers!
Dave.