Apache/PHP segfaults regularly

Hello,

I'm experiencing massive problems with my Apache/PHP webserver. It's running FreeBSD 8.1, i386 branch with GENERIC kernel.

The apache segfaults more or less regularly, however, only "Joomla" is affected. Other PHP applications deployed on the machine can be accessed as normal during such outages. (However, I can't confirm this on a long term scale, as I have no test machine available.)

Symptoms of such a situation are the "white screen of death" when trying to access a Joomla-based website, and in the following message logged:
Code:
Jan 19 18:15:58 SERVER kernel: pid 79045 (httpd), uid 80: exited on signal 11
Running "httpd -V" returns:
Code:
Server version: Apache/2.2.17 (FreeBSD)
Server built:   Oct 22 2010 20:50:07
Server's Module Magic Number: 20051115:25
Server loaded:  APR 1.4.2, APR-Util 1.3.10
Compiled using: APR 1.4.2, APR-Util 1.3.10
Architecture:   32-bit
Server MPM:     Prefork
  threaded:     no
    forked:     yes (variable process count)
Server compiled with....
 -D APACHE_MPM_DIR="server/mpm/prefork"
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
 -D APR_USE_FLOCK_SERIALIZE
 -D APR_USE_PTHREAD_SERIALIZE
 -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D DYNAMIC_MODULE_LIMIT=128
 -D HTTPD_ROOT="/usr/local"
 -D SUEXEC_BIN="/usr/local/bin/suexec"
 -D DEFAULT_PIDLOG="/var/run/httpd.pid"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_LOCKFILE="/var/run/accept.lock"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D AP_TYPES_CONFIG_FILE="etc/apache22/mime.types"
 -D SERVER_CONFIG_FILE="etc/apache22/httpd.conf"
The situation can only be cleared by a complete restart of the apache service.

After a thorough search on the web and especially some FreeBSD resources, I was already able to rule out the ZEND multibyte problem. As recommended by several sources, apache was configured to produce a coredump, shown below.

Code:
This GDB was configured as "i386-marcel-freebsd"...
Core was generated by `httpd'.
Program terminated with signal 11, Segmentation fault.
{...}
Reading symbols from /libexec/ld-elf.so.1...done.
Loaded symbols for /libexec/ld-elf.so.1
#0  0x28a57eb9 in do_bind_function ()
   from /usr/local/libexec/apache22/libphp5.so
[New Thread 2c1abf40 (LWP 100202)]
[New Thread 28501140 (LWP 100230)]
(gdb) bt
#0  0x28a57eb9 in do_bind_function ()
   from /usr/local/libexec/apache22/libphp5.so
#1  0x28a94a04 in ZEND_DECLARE_FUNCTION_SPEC_HANDLER ()
   from /usr/local/libexec/apache22/libphp5.so
#2  0x28a958a9 in execute () from /usr/local/libexec/apache22/libphp5.so
#3  0x28a71734 in zend_execute_scripts ()
   from /usr/local/libexec/apache22/libphp5.so
#4  0x28a1f1c9 in php_execute_script ()
   from /usr/local/libexec/apache22/libphp5.so
#5  0x28af414e in php_handler () from /usr/local/libexec/apache22/libphp5.so
#6  0x08076b19 in ap_run_handler (r=0x2c226058) at config.c:157
#7  0x08079dee in ap_invoke_handler (r=0x2c226058) at config.c:376
#8  0x08084eb0 in ap_process_request (r=0x2c226058) at http_request.c:282
#9  0x0808201b in ap_process_http_connection (c=0x29cf41f0) at http_core.c:190
#10 0x0807de09 in ap_run_process_connection (c=0x29cf41f0) at connection.c:43
#11 0x08089791 in child_main (child_num_arg=Variable "child_num_arg" is not available.
) at prefork.c:662
#12 0x08089a77 in make_child (s=0x28510f10, slot=0) at prefork.c:763
#13 0x0808a427 in ap_mpm_run (_pconf=0x2850f018, plog=0x2853d018, s=0x28510f10)
    at prefork.c:898
#14 0x08064195 in main (argc=676384792, argv=0x29cf2018) at main.c:739

The backtrace confirms, in my opinion, that the problem originates within the PHP module for apache and especially during the creation of new threads in order to handle an incoming request. However, this is just my guess. (I've only got minimal experience with gdb debugging).

This led me to another possible solution, found in the httpd documentation at apache.org, inserting

ServerLimit 512
MaxClients 512


as new directives in httpd.conf within the mpm_prefork complex. However, changing these values (shown are the standard values) only produced more segfaults.

The problems only occured after upgrading to PHP 5.3 from the 5.2 branch.

I have absolutely no further idea what to do. Has anyone else experienced that kind of problem and was able to solve it? Or did I miss anything?

Thank you in advance for any input or idea. If you need any further information, please don't hesitate to ask, I'll provide it as fast as possible.

Regards,
waldmeister
 
Did you manage to solve it? I have exactly the same situation. Firstly I decided that it could be connected to some memory leak and began to gather stats for the memory use, but the latest segfault occurred at night with minimum of visitors and no sudden growth in memory usage.
 
Some generic hints:

You're using the mod_php5 library. I know this is a minor holy war, but IMHO using mod_anylanguage is an increasingly bad idea.
The mod_lang were originally created to solve speed issues associated with CGI, and this worked reasonably well.
In the meanwhile, however, FastCGI was developed, which solves the same problem in a much better way.
As I've said, not everyone agrees, but one large advantage of using FastCGI is that whenever one PHP process crashes, it doesn't take every other PHP process and the entire webserver down with it. Instead, the user will get a 500 error, and the PHP process will be restarted: much more graceful!

You didn't mention which PHP version you're using, but try upgrading to the latest version possible. The latest version at the time of writing is PHP 5.3.8.

You also didn't mention which Joomla version you're using, but you can try upgrading that if possible.
 
I dealt a similar issue with a php application a few months ago. Eventually the server would restart after a while. After some digging it turned out to be an exploit in the application causing some kind of DOS.

Search for a pattern in error.log & access.log. You could find some weird PHP calls which eventually you can block access to either by using apache directives or mod_security.

Inform the application developers about this so they can issue a fix.

@Carpetsmoker, I agree with you but unfortunately some applications don't work very well this way.

Regards,
George
 
Hi all,

sorry for my late reaction on this.

@extra: I've built a quick and very dirty workaround - monitoring the apache with a Nagios check and restarting it if the check fails. This reduces the pain, but is of course no real resolution as the problem itself persists.

@Carpetsmoker: We're currently on 5.3.8 with no change in the behaviour so far. Our SSO system relies on Shibboleth2 - that, as far as I know, currently doesn't work with FastCGI (well, the documentation says it is, but the last time I looked into this topic, noone seemed to have managed to configure it properly). Our Joomla is on the latest 1.5 branch release.

@gkontos: I found nothing unusual or weird there...

At this point, I think it might have something to do with mod_shib2 but I cannot provide any evidence for this theory, except that one of the last logged actions prior to the segfault is a login.

I'll try to find out whether an update to Joomla 1.7 is planned, maybe this helps to stop the segfaults.

Regards,
waldmeister
 
In fact in my case it's not exactly the Joomla, but a custom php script with a "log in" form. I guess segfaults happen in other cases as well but that's the only script that generates them more frequently.

I've recompiled apache and php with debug options:

Code:
httpd -V
Server version: Apache/2.2.21 (FreeBSD)
Server built:   Dec  1 2011 13:01:04
Server's Module Magic Number: 20051115:30
Server loaded:  APR 1.4.5, APR-Util 1.3.12
Compiled using: APR 1.4.5, APR-Util 1.3.12
Architecture:   64-bit
Server MPM:     Prefork
  threaded:     no
    forked:     yes (variable process count)
Server compiled with....
 -D APACHE_MPM_DIR="server/mpm/prefork"
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
 -D APR_USE_FLOCK_SERIALIZE
 -D APR_USE_PTHREAD_SERIALIZE
 -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D DYNAMIC_MODULE_LIMIT=128
 -D HTTPD_ROOT="/usr/local"
 -D SUEXEC_BIN="/usr/local/sbin/suexec"
 -D DEFAULT_PIDLOG="/var/run/httpd.pid"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_LOCKFILE="/var/run/accept.lock"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D AP_TYPES_CONFIG_FILE="etc/apache22/mime.types"
 -D SERVER_CONFIG_FILE="etc/apache22/httpd.conf"

Code:
PHP 5.3.8 with Suhosin-Patch (cli) (built: Dec  5 2011 11:46:57) (DEBUG)
Copyright (c) 1997-2011 The PHP Group
Zend Engine v2.3.0, Copyright (c) 1998-2011 Zend Technologie

The coredump file is more detailed now, however I can't get anything from here :) :

Code:
(gdb) bt
#0  0x00000008045346dd in do_bind_function (opline=0x8096ddd10, function_table=0x804cb8830, compile_time=0 '\0') at 
/usr/ports/lang/php5/work/php-5.3.8/Zend/zend_compile.c:2977
#1  0x00000008045903ab in ZEND_DECLARE_FUNCTION_SPEC_HANDLER (execute_data=0x80b30a200) at zend_vm_execute.h:586
#2  0x000000080458d2ea in execute (op_array=0x8073bb3e0) at zend_vm_execute.h:107
#3  0x0000000804553a5d in zend_execute_scripts (type=8, retval=0x0, file_count=3) at /usr/ports/lang/php5/work/php-5.3.8/Zend/zend.c:1308
#4  0x00000008044d0ae2 in php_execute_script (primary_file=0x7fffffffe440) at /usr/ports/lang/php5/work/php-5.3.8/main/main.c:2299
#5  0x000000080464da85 in php_handler (r=0x8073d7ae8) at /usr/ports/lang/php5/work/php-5.3.8/sapi/apache2handler/sapi_apache2.c:673
#6  0x000000000043634a in ap_run_handler (r=0x8073d7ae8) at config.c:157
#7  0x0000000000439582 in ap_invoke_handler (r=0x8073d7ae8) at config.c:376
#8  0x0000000000443d5a in ap_internal_redirect (new_uri=Variable "new_uri" is not available.) at http_request.c:554
#9  0x00000008041b4c80 in handler_redirect (r=0x8073c90a0) at mod_rewrite.c:4838
#10 0x000000000043634a in ap_run_handler (r=0x8073c90a0) at config.c:157
#11 0x0000000000439582 in ap_invoke_handler (r=0x8073c90a0) at config.c:376
#12 0x0000000000443ece in ap_process_request (r=0x8073c90a0) at http_request.c:282
#13 0x00000000004411b8 in ap_process_http_connection (c=0x8073bf290) at http_core.c:190
#14 0x000000000043d242 in ap_run_process_connection (c=0x8073bf290) at connection.c:43
#15 0x0000000000448731 in child_main (child_num_arg=Variable "child_num_arg" is not available.) at prefork.c:667
#16 0x00000000004489d4 in make_child (s=0x801619708, slot=1) at prefork.c:768
#17 0x0000000000449264 in ap_mpm_run (_pconf=Variable "_pconf" is not available.) at prefork.c:903
#18 0x0000000000423a3a in main (argc=2, argv=0x7fffffffea48) at main.c:739

Here's the request example which _sometimes_ produces segfaults (captured by mod_whatkilledus):

Code:
[Mon Dec  5 19:25:27 2011] pid 81291 mod_whatkilledus sig 11 crash
[Mon Dec  5 19:25:27 2011] pid 81291 mod_whatkilledus active connection: 93.189.56.229:54329->93.189.56.230:81 (conn_rec 8073bf290)
[Mon Dec  5 19:25:27 2011] pid 81291 mod_whatkilledus active request (request_rec 8073e70a0):
POST /manager/index.php HTTP/1.0|Host:www.domain.com|X-Forwarded-For:x.x.x.x, x.x.x.x|X-Forwarded-Proto:http|X-Real-
IP:x.x.x.x|Connection:close|User-Agent:Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv%3a1.9.2.24) Gecko/20111103
Firefox/3.6.24|Accept:text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8|Accept-Language:en-us,en;q=0.5|Accept-
Encoding:gzip,deflate|Accept-Charset:ISO-8859-1,utf-
8;q=0.7,*;q=0.7|Referer:http%3a//www.domain.com/manager/|Cookie:client_id=x.x.x.1031319539699; PHPSESSID=d6ioqda6p82tp99d2qspf12dn2; 
session_id=x.x.x.1031323098718|Cache-Control:max-age=0|Content-Type:application/x-www-form-urlencoded|Content-Length:104
[Mon Dec  5 19:25:27 2011] pid 81291 mod_whatkilledus end of report

I've only hidden the actual URLs and ip-addresses. Where would it better to dig now?
 
extra said:
I've only hidden the actual URLs and ip-addresses. Where would it better to dig now?

You should email the developers and congratulate them because their coding is crashing the server.
 
Back
Top