Apache quits responding - Exits on Signal 10

R

ryanm

Hi Everyone,


We have Apache 2.2.13 running on FreeBSD 7.2 (inside a jail). Yesterday it quit responding completely to any requests. Although it's not responding, I use [cmd=]ps auxc | grep httpd[/cmd] to check if it's running and it is. So I restart it with [cmd=]apachectl restart[/cmd] and everything works fine. I start investigating logs and graphs and everything appears to be normal. But then when I check dmesg, I got a slew of lines that say:

Code: 
pid ##### (httpd), uid 80: exited on signal 10

I saw that message for each httpd process.

I've looked through some documentation, googled quite a bit, and the only thing I can find is that it's a BUS error. The forum I read stated that it can either mean you have bad hardware or the program you're using has bad code.

Is this accurate? Can anyone tell me what is really going on or point me to some documentation that explains more about signal 10 and how to handle it?
 
Yeah, we're using PHP and running that command works fine. This server runs our company's money-making website and has a current uptime of 450+ days. We haven't made any structural changes to the server lately (i.e. upgrades) frankly because management is afraid to do so at the risk of breaking something (which has happened to us before, which is why we're still on freebsd FreeBSD 7.2. The 'bad experience' happened on a server that is still currently running on 4.7....).

My point being that nothing significant has changed except for one thing. We added another jail to the base host. One suspicion that I have is concerning memory usage, which when I read a little bit more about SIGBUS (signal 10), many fingers point towards memory being the issue. Our server has 16 GB of RAM. Our primary jail (the one having the issue) is configured to reserve 6 GB of that to MySQL. We copied that jail and made two additional jails, but the second only reserves 4GB for MySQL and the third reserves only 1GB. So at this point we're using much more memory than we have in the past. A couple of sites I found referred to a memtest86 (I think) utility to check the memory. I'm assuming I couldn't run memory tests while the server is in use and the memory is full, right? So my hope is that someone out there has faced a similar issue and can perhaps give guidance on what steps I should take next. I think executive management would see taking the site offline for several hours to run memory tests out of the question. But if I could get some additional opinions on whether this is going to be worth my time and effort, it would go a long way.

Here's a picture of our munin memory stats.
mem_usage.png
 
That's very interesting because in the middle of my dmesg output where all the apache processes are dying, you can see this:

Code: 
...
...
pid 23358 (httpd), uid 80: exited on signal 10
pid 23438 (httpd), uid 80: exited on signal 10
pid 23433 (httpd), uid 80: exited on signal 10
pid 23386 (httpd), uid 80: exited on signal 10
pid 23339 (httpd), uid 80: exited on signal 10
Limiting closed port RST response from 281 to 200 packets/sec
Limiting closed port RST response from 209 to 200 packets/sec
Limiting closed port RST response from 259 to 200 packets/sec
Limiting closed port RST response from 228 to 200 packets/sec
Limiting closed port RST response from 223 to 200 packets/sec
pid 75352 (httpd), uid 80: exited on signal 10
pid 75602 (httpd), uid 80: exited on signal 10
pid 75529 (httpd), uid 80: exited on signal 10
pid 75527 (httpd), uid 80: exited on signal 10
pid 74812 (httpd), uid 80: exited on signal 10
pid 75707 (httpd), uid 80: exited on signal 10
pid 75538 (httpd), uid 80: exited on signal 10
...
...

Apparently that happens when there are far too many requests made to a closed port. I initially thought perhaps it could have been from a denial of service attack, but I quickly discarded the idea when I looked at our network monitors and saw that traffic was within normal operational ranges.

This seems like the best starting point. I'll propose an upgrade of Apache.
 
I see that you are using munin to monitor your server performance which is very good. Munin also comes with some very nice plugins for apache & mysql.

Those graphs can give you a clear picture if something is wrong in regards to a httpd DOS attack. I suggest that you activate them, if you haven't already.

Now, like SirDice said you have to upgrade apache because there are some nasty scripts that can cause the issues you describe.

Also, if I were in your position, I would be thoroughly monitoring my firewall and apache logs, (always keep time in synch) for a week at least.

Regards,
George
 
It doesn't happen often but Apache can have bugs. I suggest installing something like ports-mgmt/portaudit and/or signing up for the announce@ mailinglist of Apache.
 
You must log in or register to reply here.
Back
Top