Heavy load with apache22 (ufs state in `top`)

balrog2000pl

New Member


Messages: 4

Hello!

After several minutes of operation my Apache is getting stuck and almost totally unresponsive and hanging on the ufs state greatly, as diagnosed by `top` tool.

I'd like to know what files exactly is it accessing at the moment it hangs ?

Is there such tool that will tell me. I tried `lsof` and `fstat` but i would like to see real-time reads/writes on the HDD.


Site is CakePHP + MySQL. Begins to operate normally and after some time it is getting stuck on ufs state.
 

DutchDaemon

Administrator
Staff member
Administrator
Moderator
Developer

Reaction score: 2,928
Messages: 11,341

Try something like [cmd=]procstat -f -w 1 pid_of_httpd[/cmd] or run [cmd=]top -U www -m io[/cmd] to see disk activity. Assuming 'www' is your Apache user.
 
OP
B

balrog2000pl

New Member


Messages: 4

Thanks, these are quite useful. In the meantime I enabled noatime flag on /home and /var, seems to have helped a bit.

Any more details on successfully using ktrace?

Regards
 

ale

Daemon

Reaction score: 166
Messages: 1,059

Run [cmd=]ktrace -f ktrace.out -p pid_of_httpd &[/cmd] and let it run for a while (not too much as it produces huge output).
Then run [cmd=]ktrace -C[/cmd] to stop tracing the process.
Now you can examine the output with [cmd=]kdump -f ktrace.out | less[/cmd]
 

DutchDaemon

Administrator
Staff member
Administrator
Moderator
Developer

Reaction score: 2,928
Messages: 11,341

I think ktrace(1) and kdump(1) (and/or ale) tell you all you need to know. Be sure to have [cmd=]ktrace -C[/cmd] ready :)
 
OP
B

balrog2000pl

New Member


Messages: 4

Thanks DutchDaemon and ale!

ktrace showed me where is the problem. I have set session lifetime for quite long time (1 day), now the site is heavily visited, so after a day the directory contained 100k session files!

Somehow and somewhen apache or httpd was listing the directory and, whats worse, stating each of 100k files :|

Now I should probably find another session storage not to kill my server anymore.

Imagine stating 100k files concurrently by 10 processes, no wonder I had problems.
 

tuxick

New Member


Messages: 1

solution

I ran into same, in our case it the solution was dirhashing:

sysctl -a | grep dirhash

will show you current state, and most likely increasing vfs.ufs.dirhash_maxmem will solve your problems.
 

User23

Well-Known Member

Reaction score: 67
Messages: 495

Code:
systat -vm
gives you a general overview what happens on your disks. If you have thousands of directories and "vfs.ufs.dirhash_mem" is nearly as big as "vfs.ufs.dirhash_maxmem" then increasing "vfs.ufs.dirhash_maxmem" will help a lot. On a mail server i have set it to 30MB and it consumes nearly all of it :)
 
Top