Apache 2.2.17 'accepts' connections, never responds (accf_http problem?)

N

NuclearDog

Hey folks,

I'm running Apache 2.2.17 with the peruser mpm installed from ports (www/apache22-peruser-mpm) on FreeBSD 8.2-RC1.

My problem is that, although Apache is running and listening for connections, it's not actually responding to any requests once connected (or showing them in the log at all).

If I telnet to 127.0.0.1 80, it connects, and I can type out an HTTP request:

Code: 
GET / HTTP/1.1
Host: asdfasdf

And hit enter a couple of times - I never receive any reply. The connection stays open until I kill Apache. Similar results sniffing the packets of a browser's request.

The only output in the error log (with LogLevel debug):

Code: 
[Mon Jan 10 12:56:58 2011] [notice] Apache/2.2.17 (FreeBSD) configured -- resuming normal operations
[Mon Jan 10 12:56:58 2011] [info] Server built: Jan  9 2011 16:23:07
[Mon Jan 10 12:56:58 2011] [debug] peruser.c(2454): AcceptMutex: flock (default: flock)
[Mon Jan 10 12:57:07 2011] [info] removed PID file /var/run/httpd.pid (pid=1091)
[Mon Jan 10 12:57:07 2011] [notice] caught SIGTERM, shutting down

The access log stays empty:

Code: 
root:/var/log# wc httpd-access.log 
       0       0       0 httpd-access.log
root:/var/log#

I've tried with accf_http and accf_data both enabled and disabled, and with both the stock configuration and my customized config. I also tried uninstalling apache22-peruser-mpm and just installing straight apache22... Still no luck. I tried removing all of the LoadModule lines from httpd.conf and just re-enabled the ones that were necessary to parse the config. Ended up with only the following loaded:

Code: 
root:/usr/local/etc/apache22# /usr/local/sbin/apachectl -M
Loaded Modules:
 core_module (static)
 mpm_peruser_module (static)
 http_module (static)
 so_module (static)
 authz_host_module (shared)
 log_config_module (shared)
 alias_module (shared)
Syntax OK
root:/usr/local/etc/apache22#

Same results.

Apache is definitely what's listening on port 80:

Code: 
root:/usr/local/etc/apache22# sockstat -4 | grep httpd
root     httpd      43789 3  tcp4 6 *:80                  *:*
root     httpd      43789 4  tcp4   *:*                   *:*
root:/usr/local/etc/apache22#

And I know it's not a firewall issue as there is nothing running locally, and connecting from the local box to 127.0.0.1:80 results in the same issue.

I ran httpd with the -X option (debug) through truss. It never seems to actually receive any connection request from the kernel (which fits the error log lacking any connections). Only problem I'm able to see is that it keeps trying to call wait on a non-existent process id... No idea what, if anything it could mean, though.

For the interest: the output of truss.

I'm posting this on the FreeBSD forum as, given that Apache is never receiving the connection request, I don't think it's an Apache issue.

I've been banging my head against this for a few days - I've tried StackOverflow, I've tried the #freebsdhelp IRC. Any and all suggestions are VERY appreciated.

-- Adam
 
(Sorry for replying to myself, I can't find the edit button.)

According to another gentleman I talked to, the errors with wait in the truss output aren't a problem - simply a red herring, and a byproduct of how Apache manages its threads.

As well, just to be sure, I ran:

Code: 
while true; do nc -l 80 < tempfile; done

And telnet'd to it both locally and remotely. The connection completed and the file was transferred without issue.
 
Further update:

I grabbed a copy of Apache 2.2.17 from Apache directly, then applied all of the patches in /usr/ports/www/apache22/files and /usr/ports/www/apache22-peruser-mpm/files that I found specified in EXTRA_PATCHES in either Makefile, and those with names beginning 'patch-'. Compiled fine, had a bit of trouble with the install (due to the patch changing the Makefile) but got enough installed and was able to start httpd. Even copied the config I was using for the other install to be sure - starts and responds to requests just fine.

No idea what the difference is between the two at this point. Guess I'll try and find out.
 
NuclearDog said:
I'm posting this on the FreeBSD forum as, given that Apache is never receiving the connection request, I don't think it's an Apache issue.
Click to expand...

Very odd. Actually, it does seem like an Apache build/configuration issue, right? You tried the peruser and prefork MPMs (from Ports), and ran into problems. You built the source from Apache (patched / peruser MPM?), and it worked.

Grasping at straws at the moment, but:
  • Anything exotic in your /etc/make.conf?
  • Is /var/log/httpd-access.log writable by the user the Apache web server is running as? (Double check.)
  • Have you investigated at the packet level using tcpdump(1) yet? I'd be curious to know whether you're seeing normal tcp conversation - at least up to the point of a proper handshake.
 
anomie said:
Anything exotic in your /etc/make.conf?
Click to expand...

Just the standard USE_PERL line.

anomie said:
Is /var/log/httpd-access.log writable by the user the Apache web server is running as? (Double check.)
Click to expand...

They're not, but I think they're being opened before Apache drops permissions. The truss output shows it receiving what appears to be a valid file handle when it attempts to open it.

anomie said:
Have you investigated at the packet level using tcpdump(1) yet? I'd be curious to know whether you're seeing normal tcp conversation - at least up to the point of a proper handshake.
Click to expand...

I haven't, but neither telnet nor a browser raises any complaints about connecting or sending data. I have to assume it's successful.

anomie said:
Very odd. Actually, it does seem like an Apache build/configuration issue, right? You tried the peruser and prefork MPMs (from Ports), and ran into problems. You built the source from Apache (patched / peruser MPM?), and it worked.
Click to expand...

I ran all the patches that were listed in EXTRA_PATCHES, and anything starting with patch- (to my understanding, what ports does). I'm not sure it actually built with peruser, but certainly got prefork out. I did run into the same problems with www/apache22 though.

I ran a diff -r between both the ports directory and my own build directory, but unfortunately it spit out way too much for me to sift through by hand.

As much as I don't want to, I think I'm just going to have to install Apache from source - this is for a work server that I need to have up by Wednesday, and I still need to get a bunch of other stuff installed and some testing done. I've, unfortunately, been messing around with this for about 4 days already! :O

It's on Amazon EC2, so I'll make a copy of the volume if anyone is interested in continuing debugging for the sake of debugging, but for now we can consider this issue resolved for me. :)

Thanks for taking a look and taking the time to reply, anomie.
 
NuclearDog said:
They're not, but I think they're being opened before Apache drops permissions. The truss output shows it receiving what appears to be a valid file handle when it attempts to open it.
Click to expand...

Ah, of course. (Next question is: is it writable by root?? :) Would be worthwhile to absolutely confirm using cat(1), vi(1), or whatever.)

NuclearDog said:
It's on Amazon EC2, so I'll make a copy of the volume if anyone is interested in continuing debugging for the sake of debugging, but for now we can consider this issue resolved for me. :)
Click to expand...

If you pursue it at a later date, I'd be curious to know what you find. Not sure I've seen these exact symptoms before.
 
You must log in or register to reply here.
Back
Top