Apache 2.4 Reverse Proxy error 22 (invalid argument)

G

GTeley

Hello,

My FreeBSD server (14.1-RELEASE-p5) suddenly started to give Apache error "503 Service Unavailable" for every website I run on it in a separate jail.
These are various websites, each running in a 'Bastille' jail (RoR, PHP, etc)
Each site can be reached with
Code: 
curl -v http://<local ip address:port>

After adding 'LogLevel debug' to the Apache configuration, the only error I see is '(22)Invalid argument':

Code: 
[Tue Oct 22 15:40:09.245375 2024] [ssl:debug] [pid 11308] ssl_engine_kernel.c(415): [client <ipv6 address>] AH02034: Initial (No.1) HTTPS request received for child 6 (server /<my URL>:443), referer: https://<my URL>/cgi-bin/SupportReport.pl
[Tue Oct 22 15:40:09.245401 2024] [authz_core:debug] [pid 11308] mod_authz_core.c(843): [client <ipv6 address>] AH01628: authorization result: granted (no directives), referer: https:///<my URL>/cgi-bin/SupportReport.pl
[Tue Oct 22 15:40:09.245424 2024] [proxy:debug] [pid 11308] mod_proxy.c(1465): [client <ipv6 address>] AH01143: Running scheme http handler (attempt 0), referer: https:///<my URL>/cgi-bin/SupportReport.pl
[Tue Oct 22 15:40:09.245430 2024] [proxy:debug] [pid 11308] proxy_util.c(2797): AH00942: http: has acquired connection for (192.168.1.105:80)
[Tue Oct 22 15:40:09.245436 2024] [proxy:debug] [pid 11308] proxy_util.c(3242): [client <ipv6 address>] AH00944: connecting [URL]http://192.168.1.105/cgi-bin/SupportReport.pl[/URL] to 192.168.1.105:80, referer: https:///<my URL>/cgi-bin/SupportReport.pl
[Tue Oct 22 15:40:09.245449 2024] [proxy:debug] [pid 11308] proxy_util.c(2905): [client <ipv6 address>] AH10479: http: 192.168.1.105 resolved to 192.168.1.105:80, referer: https:///<my URL>/cgi-bin/SupportReport.pl
[Tue Oct 22 15:40:09.245455 2024] [proxy:debug] [pid 11308] proxy_util.c(3450): [client <ipv6 address> AH00947: connecting /cgi-bin/SupportReport.pl to 192.168.1.105:80 (192.168.1.105:80), referer: https:///<my URL>/cgi-bin/SupportReport.pl
[Tue Oct 22 15:40:09.245565 2024] [proxy:error] [pid 11308] (22)Invalid argument: AH00957: http: attempt to connect to 192.168.1.105:80 (192.168.1.105:80) failed
[Tue Oct 22 15:40:09.245601 2024] [proxy:debug] [pid 11308] proxy_util.c(2910): AH10480: http: 192.168.1.105 resolved to 192.168.1.105:80
[Tue Oct 22 15:40:09.245606 2024] [proxy_http:error] [pid 11308] [client <ipv6 address>] AH01114: HTTP: failed to make connection to backend: 192.168.1.105, referer: https:///<my URL>/cgi-bin/SupportReport.pl
[Tue Oct 22 15:40:09.245610 2024] [proxy:debug] [pid 11308] proxy_util.c(2813): AH00943: http: has released connection for (192.168.1.105:80)
[Tue Oct 22 15:40:09.245770 2024] [ssl:debug] [pid 11308] ssl_engine_io.c(1150): [client <ipv6 address>] AH02001: Connection closed to child 6 with standard shutdown (server /<my URL>:443)

Above is an example of one particular website (in this case running a Perl script) but the messages are the same for every other site running through reverse proxy.
Sites that are running directly under Apache 2.4 are not affected.

I can't recall if there has been some update specific to the point this started happening.
 
GTeley said:
Code: 
http: attempt to connect to 192.168.1.105:80 (192.168.1.105:80) failed
Click to expand...
The service isn't running on the backend on port 80, or it failed to bind to the correct IP address. In any case, your reverse proxy fails to connect to the backend.
 
SirDice: As I wrote, alle services are running and reachable on their respective addresses.
So the question is: Why is Apache suddenly not able to connect (error (22) invalid argument)?
 
VladiBG You're right!
It says client IPv6 address, although it's an internal IPv4 address.
Strange... I have to dive into that.
Cannot think of something that would trigger this change, because network wise nothing has been changed.
The Apache proxy settings are:
Apache config: 
        ProxyRequests Off
        ProxyPreserveHost On
        ProxyPass / http://192.168.1.105:80/ retry=0 timeout=3600 Keepalive=On
        ProxyPassReverse / http://192.168.1.105:80/
        RequestHeader set X_FORWARDED_PROTO 'https'
This has worked flawlessly for many years now.
As you can see, the reference is to an IPv4 address, not a host name.
 
GTeley said:
It says client IPv6 address, although it's an internal IPv4 address.
Click to expand...
It's certainly possible to have a client connect with IPv6 to the reverse proxy and have the connection to the backend on IPv4. It is a proxy, so this shouldn't be a problem. But perhaps your reverse proxy is on IPv6 only and therefor has no way of connecting to an IPv4 address?
 
Ignore my previous post. It should also work for ipv6 when you server have IPv6 so it can reply to the client.
Check the log file on 192.168.1.105. Do you see the incoming connection there?
 
Did you do any updates - anything in /var/log/messages?

There was an apr update to 1.7.5 - do you upgrade that (not saying there is anything obviously connected to what you are seeing, just something that might have changed.)
 
What else (if anything) was changed at the same time?

It might be apr. But as they’ve said (and from the change log) there’s nothing obvious so it feels like there’s something else going on.
 
You must log in or register to reply here.
Back
Top