White Screen of Death: apache/PHP/Client Certificate

gessel

gessel

I occasionally get a White Screen Of Death when connecting to a MediaWiki install running behind client-cert authentication. Once I get a WSoD, I get nothing but WSoD - that is restarting the browser, switching computers, etc. is to no avail. The WSoD state seems resolutely server inflicted.

I've done all the recommended php WSoD fixes I've found to no avail (memory limit, time zone set, error reporting on (no errors reported)). From the logs, the WSoD seems to be related to the client-cert authentication process. What is odd is that it works for a while after apachectl restart, sometimes for hours, sometimes for weeks. I could cron an hourly restart and probably have no issues, but it shouldn't require that kind of patch.

Code: 
Apache/2.2.19 (FreeBSD 8.2-Release) mod_ssl/2.2.20 OpenSSL/0.9.8q DAV/2 PHP/5.3.8

The WSoD Log is below, the full log, showing a successful connection after restart is attached.

Code: 
[Tue Sep 13 00:04:52 2011] [info] [client 12.23.45.78] Connection to child 12 established (server www.server.org:443)
[Tue Sep 13 00:04:52 2011] [info] Seeding PRNG with 144 bytes of entropy
[Tue Sep 13 00:04:52 2011] [debug] ssl_engine_kernel.c(1866): OpenSSL: Handshake: start
[Tue Sep 13 00:04:52 2011] [debug] ssl_engine_kernel.c(1874): OpenSSL: Loop: before/accept initialization
[Tue Sep 13 00:04:52 2011] [debug] ssl_engine_io.c(1897): OpenSSL: read 11/11 bytes from BIO#8068714e0 [mem: 80688e000] (BIO dump follows)
[Tue Sep 13 00:04:52 2011] [debug] ssl_engine_io.c(1830): +-------------------------------------------------------------------------+
[Tue Sep 13 00:04:52 2011] [debug] ssl_engine_io.c(1869): | 0000: th is bi to fd at am ig-ht be ov er sh ar ei ng  redacted........ |
[Tue Sep 13 00:04:52 2011] [debug] ssl_engine_io.c(1873): | 0011 - <SPACES/NULS>
[Tue Sep 13 00:04:52 2011] [debug] ssl_engine_io.c(1875): +-------------------------------------------------------------------------+
[Tue Sep 13 00:04:52 2011] [debug] ssl_engine_io.c(1897): OpenSSL: read 109/109 bytes from BIO#8068714e0 [mem: 80688e00b] (BIO dump follows)
[Tue Sep 13 00:04:52 2011] [debug] ssl_engine_io.c(1830): +-------------------------------------------------------------------------+
[Tue Sep 13 00:04:52 2011] [debug] ssl_engine_io.c(1869): | 0000: th is bi to fd at am ig-ht be ov er sh ar ei ng  redacted........ |
[Tue Sep 13 00:04:52 2011] [debug] ssl_engine_io.c(1869): | 0000: th is bi to fd at am ig-ht be ov er sh ar ei ng  redacted........ |
[Tue Sep 13 00:04:52 2011] [debug] ssl_engine_io.c(1869): | 0000: th is bi to fd at am ig-ht be ov er sh ar ei ng  redacted........ |
[Tue Sep 13 00:04:52 2011] [debug] ssl_engine_io.c(1869): | 0000: th is bi to fd at am ig-ht be ov er sh ar ei ng  redacted........ |
[Tue Sep 13 00:04:52 2011] [debug] ssl_engine_io.c(1869): | 0000: th is bi to fd at am ig-ht be ov er sh ar ei ng  redacted........ |
[Tue Sep 13 00:04:52 2011] [debug] ssl_engine_io.c(1869): | 0000: th is bi to fd at am ig-ht be ov er sh ar ei ng  redacted........ |
[Tue Sep 13 00:04:52 2011] [debug] ssl_engine_io.c(1869): | 0000: th is bi to fd at am ig-ht be ov er sh ar ei ng  redacted........ |
[Tue Sep 13 00:04:52 2011] [debug] ssl_engine_io.c(1873): | 0109 - <SPACES/NULS>
[Tue Sep 13 00:04:52 2011] [debug] ssl_engine_io.c(1875): +-------------------------------------------------------------------------+
[Tue Sep 13 00:04:52 2011] [debug] ssl_scache_shmcb.c(393): ssl_scache_shmcb_retrieve (0x49 -> subcache 9)
[Tue Sep 13 00:04:52 2011] [debug] ssl_scache_shmcb.c(680): possible match at idx=0, data=0
[Tue Sep 13 00:04:52 2011] [debug] ssl_scache_shmcb.c(697): shmcb_subcache_retrieve returning matching session
[Tue Sep 13 00:04:52 2011] [debug] ssl_scache_shmcb.c(408): leaving ssl_scache_shmcb_retrieve successfully
[Tue Sep 13 00:04:52 2011] [debug] ssl_engine_kernel.c(1732): Inter-Process Session Cache: request=GET status=FOUND id=49DDbitsremoved1FA (session reuse)
[Tue Sep 13 00:04:52 2011] [debug] ssl_engine_kernel.c(1987): [client 12.23.45.78] SSL virtual host for servername www.server.org found
[Tue Sep 13 00:04:52 2011] [debug] ssl_engine_kernel.c(1874): OpenSSL: Loop: SSLv3 read client hello A
[Tue Sep 13 00:04:52 2011] [debug] ssl_engine_kernel.c(1874): OpenSSL: Loop: SSLv3 write server hello A
[Tue Sep 13 00:04:52 2011] [debug] ssl_engine_kernel.c(1874): OpenSSL: Loop: SSLv3 write change cipher spec A
[Tue Sep 13 00:04:52 2011] [debug] ssl_engine_kernel.c(1874): OpenSSL: Loop: SSLv3 write finished A
[Tue Sep 13 00:04:52 2011] [debug] ssl_engine_kernel.c(1874): OpenSSL: Loop: SSLv3 flush data
[Tue Sep 13 00:04:52 2011] [debug] ssl_engine_io.c(1897): OpenSSL: read 5/5 bytes from BIO#8068714e0 [mem: 80688e000] (BIO dump follows)
[Tue Sep 13 00:04:52 2011] [debug] ssl_engine_io.c(1830): +-------------------------------------------------------------------------+
[Tue Sep 13 00:04:52 2011] [debug] ssl_engine_io.c(1869): | 0000: th is bi to fd at am ig-ht be ov er sh ar ei ng  redacted........ |
[Tue Sep 13 00:04:52 2011] [debug] ssl_engine_io.c(1875): +-------------------------------------------------------------------------+
[Tue Sep 13 00:04:52 2011] [debug] ssl_engine_io.c(1897): OpenSSL: read 1/1 bytes from BIO#8068714e0 [mem: 80688e005] (BIO dump follows)
[Tue Sep 13 00:04:52 2011] [debug] ssl_engine_io.c(1830): +-------------------------------------------------------------------------+
[Tue Sep 13 00:04:52 2011] [debug] ssl_engine_io.c(1869): | 0000: th is bi to fd at am ig-ht be ov er sh ar ei ng  redacted........ |
[Tue Sep 13 00:04:52 2011] [debug] ssl_engine_io.c(1875): +-------------------------------------------------------------------------+
[Tue Sep 13 00:04:52 2011] [debug] ssl_engine_io.c(1897): OpenSSL: read 5/5 bytes from BIO#8068714e0 [mem: 80688e000] (BIO dump follows)
[Tue Sep 13 00:04:52 2011] [debug] ssl_engine_io.c(1830): +-------------------------------------------------------------------------+
[Tue Sep 13 00:04:52 2011] [debug] ssl_engine_io.c(1869): | 0000: th is bi to fd at am ig-ht be ov er sh ar ei ng  redacted........ |
[Tue Sep 13 00:04:52 2011] [debug] ssl_engine_io.c(1875): +-------------------------------------------------------------------------+
[Tue Sep 13 00:04:52 2011] [debug] ssl_engine_io.c(1897): OpenSSL: read 64/64 bytes from BIO#8068714e0 [mem: 80688e005] (BIO dump follows)
[Tue Sep 13 00:04:52 2011] [debug] ssl_engine_io.c(1830): +-------------------------------------------------------------------------+
[Tue Sep 13 00:04:52 2011] [debug] ssl_engine_io.c(1869): | 0000: th is bi to fd at am ig-ht be ov er sh ar ei ng  redacted........ |
[Tue Sep 13 00:04:52 2011] [debug] ssl_engine_io.c(1869): | 0000: th is bi to fd at am ig-ht be ov er sh ar ei ng  redacted........ |
[Tue Sep 13 00:04:52 2011] [debug] ssl_engine_io.c(1869): | 0000: th is bi to fd at am ig-ht be ov er sh ar ei ng  redacted........ |
[Tue Sep 13 00:04:52 2011] [debug] ssl_engine_io.c(1869): | 0000: th is bi to fd at am ig-ht be ov er sh ar ei ng  redacted........ |
[Tue Sep 13 00:04:52 2011] [debug] ssl_engine_io.c(1875): +-------------------------------------------------------------------------+
[Tue Sep 13 00:04:52 2011] [debug] ssl_engine_kernel.c(1874): OpenSSL: Loop: SSLv3 read finished A
[Tue Sep 13 00:04:52 2011] [debug] ssl_engine_kernel.c(1870): OpenSSL: Handshake: done
[Tue Sep 13 00:04:52 2011] [info] Connection: Client IP: 12.23.45.78, Protocol: SSLv3, Cipher: DHE-RSA-CAMELLIA256-SHA (256/256 bits)
[Tue Sep 13 00:04:52 2011] [info] [client 12.23.45.78] No acceptable peer certificate available
[Tue Sep 13 00:04:52 2011] [info] [client 12.23.45.78] Connection closed to child 12 with abortive shutdown (server www.server.org:443)
white screen of death served:(
(attached file is the log with happy serving after restart):)
 

Attachments

  • BSoD_client_cert.zip
    4.5 KB · Views: 148
gessel,

I am going on the assumption that you have a LCD flat panel monitor. The screen going white on these devices whether it is a TV or monitor are common. The usual reason is that the capacitors on the power supply are failing and not supplying the proper voltage to the display and all you see is the florescent light. If the capacitors have a bulge in them they are suspect. Replacing them is the only fix if this is the problem. You stated that it sometimes runs for weeks without any problems could indicate a hardware malfunction. Have you tried a different monitor? Just an idea that may not even be in the ball park.

Just a thought!

Keith
 
Hello,

You have a PHP warning you should fix, did you update some part of PHP or Apache, usually mod_ssl version is same as the Apache one, e.g. Apache 2.2.17 with mod_ssl 2.2.17.

From your attached file Apache seems to be version 2.2.20 not 2.2.19 compiled with OpenSSL/1.0.0e not 0.9.8q. Perhaps you paste the info from the server-status?

Anyway, please paste your httpd-ssl.conf and server-status beginning lines.

Code: 
No acceptable peer certificate available
from the attached file indicates some problem with the config, your server may stop sending the CA list/chains due to some reason (high load perhaps because next line is that server start to kill child processes).

Can you also provide the error log of Apache, and output from $ top when white screen of death happens (I mean in what state Apache stuck when this happens).
 
Crotalus,

The WSoD is in the browser window only - it is a phrase to describe a fairly common PHP-related server error a bit like BSoD describes the common windows error.

Quintessence,
I apologize for the delayed response. Much travel happened, AFK (and machines).

server-status returns WSoD (until apachectl restart)

top attached
httpd-ssl.conf attached

server-status output:
Code: 
Server Version: Apache/2.2.21 (FreeBSD) mod_ssl/2.2.21 OpenSSL/0.9.8q DAV/2 PHP/5.3.8 with Suhosin-Patch
Server Built: Sep 20 2011 20:19:42

SSL/TLS Session Cache Status:
cache type: SHMCB, shared memory: 512000 bytes, current sessions: 1
subcaches: 32, indexes per subcache: 133
time left on oldest entries' SSL sessions: avg: 3600 seconds, (range: 3600...3600)
index usage: 0%, cache usage: 0%
total sessions stored since starting: 1
total sessions expired since starting: 0
total (pre-expiry) sessions scrolled out of the cache: 0
total retrieves since starting: 0 hit, 1 miss
total removes since starting: 0 hit, 0 miss

I did one update since the original post, but the problem is the same.
 

Attachments

  • top.txt
    3.1 KB · Views: 239
  • httpd-ssl.txt
    11 KB · Views: 253
Hello,

Is the output is from when white screen of death occurs?

Are HostnameLookups is set to Off in /usr/local/etc/apache22/extra/httpd-default.conf? What makes Bind to has such CPU usage?
 
Yes, it is during a WSoD state. 24 hours after restarting apache, it is still serving pages, so it may be a day or a few days before it starts WSoDing again.

HostnameLookups is off

Not sure what is making Bind crazy. That's a pretty obvious indicator of problem. It is high now at 54%WCPU. I'll try to track that down and report.
 
OK, named error was dnssec issue (resolution still pending, but DNSSec options disabled for now). I believe independent of the WSoD issue.
 
You must log in or register to reply here.
Back
Top