Solved nginx https in jail won't work

Hi,

I wanted to add SSL to my webserver, but I am not able to get a proper connection via browser although the SSL-test at ssllabs gave me an "A-".

First things first: I use nginx inside an ezjail on a machine with just one public IP. nginx is reachable and serves http well. So I got me a ssl certificate by cacert.org and "installed" it. (It's not the first time I done this with nginx, but until now I always used Ubuntu)

My nginx.vhost.conf:
Code:
server {
    listen 443 ssl;

    server_name huehnerhose.de www.huehnerhose.de;

    error_log /var/log/nginx-huehnerhose.error.log debug;
    access_log /var/log/nginx-huehnerhose.access.log;
    ssl on;
    ssl_certificate /path/to/cert.crt;
    ssl_certificate_key /path/to/cert.key;

    root /usr/local/www/huehnerhose/www/;

    location / {
        index index.php index.html;
    }

}

On my "host", my pf.conf:

Code:
ext_if="vtnet0"
jail_if="lo1"


IP_PUB="37.120.172.66"

IP_JAIL_NGINX="10.0.0.10"

scrub in all
nat pass on $ext_if from $jail_if:network to any -> $IP_PUB

# Forward: HTTP, HTTPS to nginx-Jail
rdrpass on $ext_if proto tcp from any to $IP_PUB port 80 -> $IP_JAIL_NGINX
rdrpass on $ext_if proto tcp from any to $IP_PUB port 443 -> $IP_JAIL_NGINX

When I try to connect via browser I only get an NET::ERR_CERT_INAVLID. Qualys SSLLabs says everything is "ok" (they don't trust cacert and at the moment i don't have a full certificate chain): https://www.ssllabs.com/ssltest/analyze.html?d=huehnerhose.de

My error.log made me wonder:
Code:
2014/12/17 20:01:41 [info] 44213#0: *1 client closed connection while waiting for request, client: 87.160.83.181, server: 0.0.0.0:443
Every time I connect via browser this line pops up. My only suspicion is that nginx "binds" to the wrong IP or something. I tried to bind nginx to the jail-internal ip but that changed nothing.

Do you have any ideas how to solve this?

Thank and best regards!
 
Try rdr pass on $ext_if inet proto tcp from any to $ext_if port 443 -> $IP_JAIL_NGINX in pf.conf
I'm not sure what jail_if="lo1" means because you put 10.x.x.x range on it and the reserved range for loopback interface is 127.0.0.0/8.
I use aliases on lo0 for my jails with 127.0.x.x addresses.
HTH
 
Is a web server running on the host also? That can be a problem with the host web server running on all the network interfaces, including the jail interface.
 
Thanks for the answers,

HarryE I added an lo1 device and assigned 10.0.0.X IPs to it. So I think the IP assigning is correct. I tried your suggested pf-rule. With that active I can't connect to 443 at all.

wblock@ No there is no other http/https server running anywhere on that server.

But I discovered a very strange behaviour: I worked yesterday on a mac using chrome and safari. Both couldn't connect to my server via https. Today I used Chrome and Internet Explorer on Windows, both were able to connect flawless! Firefox (ESR 17.0, 24) on Windows couldn't connect, either. It said "data connection was interrupted" (rough translation from german version). Firefox 31.3 can connect again.... what the.... ?
Is it possible that I ran into some quirk with ssl-library incompatibilities?
 
I'm able to connect just fine using HTTPS with Firefox on Windows. I do get a certificate error though, the CA isn't trusted. This may be the reason it's not working on the clients. Policies may prevent accepting such a certificate.
 
SirDice Thanks, I just edited my answer when I realized I was using an old version of Firefox. I added cacert to my trusted CAs, allready. https://cacert.org and other cacerted https sites do work without any problems.

Edit: Just tested Firefox 34 on Mac, this works too.
So summary:
Mac:
  • Chrome and Safari don't work
  • Firefox 34, works
Windows
  • Chrome and InternetExplorer do work
  • Firefox 17 and 24 don't work, Firefox 31 works
I think Firefox always uses it's own ssl-lib. Chrome uses the hostsystems. Is there any way to see the cipher, protocol, whatever the current ssl connection tries to use? Can nginx log that somehow?
 
I search a little bit further and recompiled nginx --with-debug. This is how an connection with error looks like:
Code:
2014/12/18 15:18:21 [debug] 39548#0: accept on 0.0.0.0:443, ready: 1
2014/12/18 15:18:21 [debug] 39548#0: posix_memalign: 0000000803D16100:256 @16
2014/12/18 15:18:21 [debug] 39548#0: *7 accept: 87.160.80.51 fd:3
2014/12/18 15:18:21 [debug] 39548#0: posix_memalign: 0000000803D16300:256 @16
2014/12/18 15:18:21 [debug] 39548#0: *7 event timer add: 3: 60000:1418912361796
2014/12/18 15:18:21 [debug] 39548#0: *7 reusable connection: 1
2014/12/18 15:18:21 [debug] 39548#0: *7 kevent set event: 3: ft:-1 fl:0025
2014/12/18 15:18:21 [debug] 39548#0: accept on 0.0.0.0:443, ready: 1
2014/12/18 15:18:21 [debug] 39548#0: posix_memalign: 0000000803D16400:256 @16
2014/12/18 15:18:21 [debug] 39548#0: *8 accept: 87.160.80.51 fd:12
2014/12/18 15:18:21 [debug] 39548#0: posix_memalign: 0000000803D16500:256 @16
2014/12/18 15:18:21 [debug] 39548#0: *8 event timer add: 12: 60000:1418912361797
2014/12/18 15:18:21 [debug] 39548#0: *8 reusable connection: 1
2014/12/18 15:18:21 [debug] 39548#0: *8 kevent set event: 12: ft:-1 fl:0025
2014/12/18 15:18:21 [debug] 39548#0: *7 http check ssl handshake
2014/12/18 15:18:21 [debug] 39548#0: *7 http recv(): 1
2014/12/18 15:18:21 [debug] 39548#0: *7 https ssl handshake: 0x16
2014/12/18 15:18:21 [debug] 39548#0: *7 SSL server name: "huehnerhose.de"
2014/12/18 15:18:21 [debug] 39548#0: *7 SSL NPN advertised
2014/12/18 15:18:21 [debug] 39548#0: *7 SSL_do_handshake: -1
2014/12/18 15:18:21 [debug] 39548#0: *7 SSL_get_error: 2
2014/12/18 15:18:21 [debug] 39548#0: *7 reusable connection: 0
2014/12/18 15:18:22 [debug] 39548#0: *8 http check ssl handshake
2014/12/18 15:18:22 [debug] 39548#0: *8 http recv(): 1
2014/12/18 15:18:22 [debug] 39548#0: *8 https ssl handshake: 0x16
2014/12/18 15:18:22 [debug] 39548#0: *8 SSL server name: "huehnerhose.de"
2014/12/18 15:18:22 [debug] 39548#0: *8 SSL NPN advertised
2014/12/18 15:18:22 [debug] 39548#0: *8 SSL_do_handshake: -1
2014/12/18 15:18:22 [debug] 39548#0: *8 SSL_get_error: 2
2014/12/18 15:18:22 [debug] 39548#0: *8 reusable connection: 0
2014/12/18 15:18:22 [debug] 39548#0: *7 SSL handshake handler: 0
2014/12/18 15:18:22 [debug] 39548#0: *7 SSL_do_handshake: 1
2014/12/18 15:18:22 [debug] 39548#0: *7 SSL: TLSv1.2, cipher: "ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD"
2014/12/18 15:18:22 [debug] 39548#0: *7 reusable connection: 1
2014/12/18 15:18:22 [debug] 39548#0: *7 http wait request handler
2014/12/18 15:18:22 [debug] 39548#0: *7 malloc: 0000000803C06800:1024
2014/12/18 15:18:22 [debug] 39548#0: *7 SSL_read: 0
2014/12/18 15:18:22 [debug] 39548#0: *7 SSL_get_error: 5
2014/12/18 15:18:22 [debug] 39548#0: *7 peer shutdown SSL cleanly
2014/12/18 15:18:22 [info] 39548#0: *7 client closed connection while waiting for request, client: 87.160.80.51, server: 0.0.0.0:443
2014/12/18 15:18:22 [debug] 39548#0: *7 close http connection: 3
2014/12/18 15:18:22 [debug] 39548#0: *7 SSL_shutdown: 1
2014/12/18 15:18:22 [debug] 39548#0: *7 event timer del: 3: 1418912361796
2014/12/18 15:18:22 [debug] 39548#0: *7 reusable connection: 0
2014/12/18 15:18:22 [debug] 39548#0: *7 free: 0000000803C06800
2014/12/18 15:18:22 [debug] 39548#0: *7 free: 0000000803D16100, unused: 2
2014/12/18 15:18:22 [debug] 39548#0: *7 free: 0000000803D16300, unused: 40
2014/12/18 15:18:22 [debug] 39548#0: *8 SSL handshake handler: 0
2014/12/18 15:18:22 [debug] 39548#0: *8 SSL_do_handshake: 1
2014/12/18 15:18:22 [debug] 39548#0: *8 SSL: TLSv1.2, cipher: "ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD"
2014/12/18 15:18:22 [debug] 39548#0: *8 reusable connection: 1
2014/12/18 15:18:22 [debug] 39548#0: *8 http wait request handler
2014/12/18 15:18:22 [debug] 39548#0: *8 malloc: 0000000803C06800:1024
2014/12/18 15:18:22 [debug] 39548#0: *8 SSL_read: 0
2014/12/18 15:18:22 [debug] 39548#0: *8 SSL_get_error: 5
2014/12/18 15:18:22 [debug] 39548#0: *8 peer shutdown SSL cleanly
2014/12/18 15:18:22 [info] 39548#0: *8 client closed connection while waiting for request, client: 87.160.80.51, server: 0.0.0.0:443
2014/12/18 15:18:22 [debug] 39548#0: *8 close http connection: 12
2014/12/18 15:18:22 [debug] 39548#0: *8 SSL_shutdown: 1
2014/12/18 15:18:22 [debug] 39548#0: *8 event timer del: 12: 1418912361797
2014/12/18 15:18:22 [debug] 39548#0: *8 reusable connection: 0
2014/12/18 15:18:22 [debug] 39548#0: *8 free: 0000000803C06800
2014/12/18 15:18:22 [debug] 39548#0: *8 free: 0000000803D16400, unused: 2
2014/12/18 15:18:22 [debug] 39548#0: *8 free: 0000000803D16500, unused: 40

And this is a successful connection:
Code:
2014/12/18 15:25:49 [debug] 39548#0: accept on 0.0.0.0:443, ready: 1
2014/12/18 15:25:49 [debug] 39548#0: posix_memalign: 0000000803D16100:256 @16
2014/12/18 15:25:49 [debug] 39548#0: *10 accept: 87.160.80.51 fd:3
2014/12/18 15:25:49 [debug] 39548#0: posix_memalign: 0000000803D16300:256 @16
2014/12/18 15:25:49 [debug] 39548#0: *10 event timer add: 3: 60000:1418912809088
2014/12/18 15:25:49 [debug] 39548#0: *10 reusable connection: 1
2014/12/18 15:25:49 [debug] 39548#0: *10 kevent set event: 3: ft:-1 fl:0025
2014/12/18 15:25:49 [debug] 39548#0: *10 http check ssl handshake
2014/12/18 15:25:49 [debug] 39548#0: *10 http recv(): 1
2014/12/18 15:25:49 [debug] 39548#0: *10 https ssl handshake: 0x16
2014/12/18 15:25:49 [debug] 39548#0: *10 ssl get session: 63B1F38A:32
2014/12/18 15:25:49 [debug] 39548#0: *10 SSL server name: "huehnerhose.de"
2014/12/18 15:25:49 [debug] 39548#0: *10 SSL NPN advertised
2014/12/18 15:25:49 [debug] 39548#0: *10 SSL_do_handshake: -1
2014/12/18 15:25:49 [debug] 39548#0: *10 SSL_get_error: 2
2014/12/18 15:25:49 [debug] 39548#0: *10 reusable connection: 0
2014/12/18 15:25:49 [debug] 39548#0: *10 SSL handshake handler: 0
2014/12/18 15:25:49 [debug] 39548#0: *10 SSL_do_handshake: 1
2014/12/18 15:25:49 [debug] 39548#0: *10 SSL: TLSv1.2, cipher: "ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD"
2014/12/18 15:25:49 [debug] 39548#0: *10 reusable connection: 1
2014/12/18 15:25:49 [debug] 39548#0: *10 http wait request handler
2014/12/18 15:25:49 [debug] 39548#0: *10 malloc: 0000000803C06800:1024
2014/12/18 15:25:49 [debug] 39548#0: *10 SSL_read: 403
2014/12/18 15:25:49 [debug] 39548#0: *10 SSL_read: -1
2014/12/18 15:25:49 [debug] 39548#0: *10 SSL_get_error: 2
2014/12/18 15:25:49 [debug] 39548#0: *10 reusable connection: 0
2014/12/18 15:25:49 [debug] 39548#0: *10 posix_memalign: 0000000803C90000:4096 @16
2014/12/18 15:25:49 [debug] 39548#0: *10 http process request line
2014/12/18 15:25:49 [debug] 39548#0: *10 http request line: "GET / HTTP/1.1"
2014/12/18 15:25:49 [debug] 39548#0: *10 http uri: "/"
2014/12/18 15:25:49 [debug] 39548#0: *10 http args: ""
2014/12/18 15:25:49 [debug] 39548#0: *10 http exten: ""
2014/12/18 15:25:49 [debug] 39548#0: *10 posix_memalign: 0000000803C95000:4096 @16
2014/12/18 15:25:49 [debug] 39548#0: *10 http process request header line
2014/12/18 15:25:49 [debug] 39548#0: *10 http header: "Host: huehnerhose.de"
2014/12/18 15:25:49 [debug] 39548#0: *10 http header: "User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:34.0) Gecko/20100101 Firefox/34.0"
2014/12/18 15:25:49 [debug] 39548#0: *10 http header: "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"
2014/12/18 15:25:49 [debug] 39548#0: *10 http header: "Accept-Language: en-US,en;q=0.5"
2014/12/18 15:25:49 [debug] 39548#0: *10 http header: "Accept-Encoding: gzip, deflate"
2014/12/18 15:25:49 [debug] 39548#0: *10 http header: "Connection: keep-alive"
2014/12/18 15:25:49 [debug] 39548#0: *10 http header: "If-Modified-Since: Fri, 12 Dec 2014 21:56:02 GMT"
2014/12/18 15:25:49 [debug] 39548#0: *10 http header: "If-None-Match: "548b6472-c""
2014/12/18 15:25:49 [debug] 39548#0: *10 http header: "Cache-Control: max-age=0"
2014/12/18 15:25:49 [debug] 39548#0: *10 http header done

As far as I understand that output, the connection with error actually connects twice. The Cipher is called with different options (which I honestly don't understand). And the only real error I see is SSL_get_error: 5 where the successfull version has SSL_get_error: 2

I'm desperate
 
Mytery solved... I stumbled across the solution having SSL issues with my mail server and Apple Mail... My certificate used a key with 8192bit length... Apparently apple doesn't support this keylength a new certificate with 4096bit key length works.

Thanks for the help!
 
Back
Top