Apache or php problem (i think) (http.core dump)

Hi all !

I have been running some old php websites without problem until now. I dont know what the problem is exactly just that sometimes the box grinds to a halt under huge load (20+). Ssh barely responds.
It is a vmware esx 4.1 guest running 8.1 with php52 + apache22 + mysql in JAIL. I have disabled suhosin and multibyte.

httpd.conf:
CoreDumpDirectory /tmp/apache2-dump
Code:
# gdb /usr/local/sbin/httpd  httpd.core
....
Loaded symbols for /usr/local/lib/libpthread-stubs.so.0
Reading symbols from /usr/lib/librpcsvc.so.4...done.
Loaded symbols for /usr/lib/librpcsvc.so.4
Reading symbols from /usr/local/lib/php/20060613/mysqli.so...done.
Loaded symbols for /usr/local/lib/php/20060613/mysqli.so
Reading symbols from /libexec/ld-elf.so.1...done.
Loaded symbols for /libexec/ld-elf.so.1
#0  0x0000000804d11bb0 in _zend_mm_free_int () from /usr/local/libexec/apache22/libphp5.so
[New Thread 8018041c0 (LWP 100219)]
(gdb) bt
#0  0x0000000804d11bb0 in _zend_mm_free_int () from /usr/local/libexec/apache22/libphp5.so
#1  0x0000000804d1f0e5 in free_zend_constant () from /usr/local/libexec/apache22/libphp5.so
#2  0x0000000804d378f2 in zend_hash_apply_deleter () from /usr/local/libexec/apache22/libphp5.so
#3  0x0000000804d379f9 in zend_hash_reverse_apply () from /usr/local/libexec/apache22/libphp5.so
#4  0x0000000804d209b1 in shutdown_executor () from /usr/local/libexec/apache22/libphp5.so
#5  0x0000000804d2cdff in zend_deactivate () from /usr/local/libexec/apache22/libphp5.so
#6  0x0000000804ceae2b in php_request_shutdown () from /usr/local/libexec/apache22/libphp5.so
#7  0x0000000804db1815 in php_handler () from /usr/local/libexec/apache22/libphp5.so
#8  0x0000000000435e4a in ap_run_handler (r=0x8057464c0) at config.c:157
#9  0x0000000000439082 in ap_invoke_handler (r=0x8057464c0) at config.c:376
#10 0x00000000004435aa in ap_internal_redirect (new_uri=Variable "new_uri" is not available.
) at http_request.c:549
#11 0x00000008049bcc60 in handler_redirect (r=0x80574c0a0) at mod_rewrite.c:4831
#12 0x0000000000435e4a in ap_run_handler (r=0x80574c0a0) at config.c:157
#13 0x0000000000439082 in ap_invoke_handler (r=0x80574c0a0) at config.c:376
#14 0x000000000044371e in ap_process_request (r=0x80574c0a0) at http_request.c:282
#15 0x0000000000440a58 in ap_process_http_connection (c=0x805731290) at http_core.c:190
#16 0x000000000043cd42 in ap_run_process_connection (c=0x805731290) at connection.c:43
#17 0x00000000004479a1 in child_main (child_num_arg=Variable "child_num_arg" is not available.
) at prefork.c:662
#18 0x0000000000447c44 in make_child (s=0x80181c708, slot=13) at prefork.c:763
#19 0x00000000004484d4 in ap_mpm_run (_pconf=Variable "_pconf" is not available.
) at prefork.c:898
#20 0x000000000042374a in main (argc=2, argv=0x7fffffffead8) at main.c:739
(gdb) quit

Suggestions ?
 
What means not curable? o O
billgates i think you should try to debug by turning off virtualhosts one by one.. With huge load do you have cpu idle?
 
Not curable means not curable. PHP (in this case - mod_php5, but this situation is typical for cgi, fast-cgi and fpm too) catches segfault on exit, when request was processed and client recieved its result. On some kind of freeing memory (_zend_mm_free_int, look gogle for many bugs). I saw it on very different virtual hosts, site engines, different php code.
 
@Alt, what do you mean "with huge load do you have cpu idle" ? Is this possible?

@mievmk, my httpd processes do not die, is this your experience too ?
 
billgates said:
@Alt, what do you mean "with huge load do you have cpu idle" ? Is this possible?
Sure, cpu load is cpu load. While load averages is number of waitng processes. So if 100 processes waiting disk i/o, LA will be 100 while cpu load is low
 
Ok, so i ran truss -p httpdpid on a pid that i think "hangs" and i see :
Code:
....
gettimeofday({1289992042.474568 },0x0)           = 0 (0x0)
gettimeofday({1289992042.485542 },0x0)           = 0 (0x0)
gettimeofday({1289992042.490529 },0x0)           = 0 (0x0)
gettimeofday({1289992042.497070 },0x0)           = 0 (0x0)
gettimeofday({1289992042.504518 },0x0)           = 0 (0x0)
gettimeofday({1289992042.507055 },0x0)           = 0 (0x0)
gettimeofday({1289992042.514551 },0x0)           = 0 (0x0)
gettimeofday({1289992042.522065 },0x0)           = 0 (0x0)
gettimeofday({1289992042.543116 },0x0)           = 0 (0x0)
gettimeofday({1289992042.553166 },0x0)           = 0 (0x0)
gettimeofday({1289992042.558047 },0x0)           = 0 (0x0)
gettimeofday({1289992042.563117 },0x0)           = 0 (0x0)
gettimeofday({1289992042.568216 },0x0)           = 0 (0x0)
gettimeofday({1289992042.570552 },0x0)           = 0 (0x0)
gettimeofday({1289992042.578012 },0x0)           = 0 (0x0)
gettimeofday({1289992042.597010 },0x0)           = 0 (0x0)
gettimeofday({1289992042.607025 },0x0)           = 0 (0x0)
gettimeofday({1289992042.613119 },0x0)           = 0 (0x0)
gettimeofday({1289992042.618052 },0x0)           = 0 (0x0)
gettimeofday({1289992042.623148 },0x0)           = 0 (0x0)
gettimeofday({1289992042.628184 },0x0)           = 0 (0x0)
gettimeofday({1289992042.633143 },0x0)           = 0 (0x0)
gettimeofday({1289992042.635553 },0x0)           = 0 (0x0)
gettimeofday({1289992042.640519 },0x0)           = 0 (0x0)
gettimeofday({1289992042.640639 },0x0)           = 0 (0x0)
gettimeofday({1289992042.645606 },0x0)           = 0 (0x0)
gettimeofday({1289992042.648172 },0x0)           = 0 (0x0)
gettimeofday({1289992042.653071 },0x0)           = 0 (0x0)
....LOT more of gettimeofday....

Is this normal ? (this is on vmware esx in a Jail)

Best regards
 
Back
Top