Heavy load with apache22 (ufs state in `top`)

Hello!

After several minutes of operation my Apache is getting stuck and almost totally unresponsive and hanging on the ufs state greatly, as diagnosed by `top` tool.

I'd like to know what files exactly is it accessing at the moment it hangs ?

Is there such tool that will tell me. I tried `lsof` and `fstat` but i would like to see real-time reads/writes on the HDD.


Site is CakePHP + MySQL. Begins to operate normally and after some time it is getting stuck on ufs state.
 
Try something like [cmd=]procstat -f -w 1 pid_of_httpd[/cmd] or run [cmd=]top -U www -m io[/cmd] to see disk activity. Assuming 'www' is your Apache user.
 
Thanks, these are quite useful. In the meantime I enabled noatime flag on /home and /var, seems to have helped a bit.

Any more details on successfully using ktrace?

Regards
 
Run [cmd=]ktrace -f ktrace.out -p pid_of_httpd &[/cmd] and let it run for a while (not too much as it produces huge output).
Then run [cmd=]ktrace -C[/cmd] to stop tracing the process.
Now you can examine the output with [cmd=]kdump -f ktrace.out | less[/cmd]
 
Thanks DutchDaemon and ale!

ktrace showed me where is the problem. I have set session lifetime for quite long time (1 day), now the site is heavily visited, so after a day the directory contained 100k session files!

Somehow and somewhen apache or httpd was listing the directory and, whats worse, stating each of 100k files :|

Now I should probably find another session storage not to kill my server anymore.

Imagine stating 100k files concurrently by 10 processes, no wonder I had problems.
 
solution

I ran into same, in our case it the solution was dirhashing:

sysctl -a | grep dirhash

will show you current state, and most likely increasing vfs.ufs.dirhash_maxmem will solve your problems.
 
Code:
systat -vm
gives you a general overview what happens on your disks. If you have thousands of directories and "vfs.ufs.dirhash_mem" is nearly as big as "vfs.ufs.dirhash_maxmem" then increasing "vfs.ufs.dirhash_maxmem" will help a lot. On a mail server i have set it to 30MB and it consumes nearly all of it :)
 
Back
Top