Cloudflare 521 error with PF and acme.sh

J

justinnoor

Hello community,

I'm getting a 521 error after setting up SSL on a domain that is hosted on Cloudflare. This is a FreeBSD-12.0-RELEASE-p10 machine running obhttpd and PF.

I issued certs on a site with acme.sh and the dns challenge was successful. I used the DNS API method:
Code: 
$ sh
$ sudo su
# export CF_Key"sdfsdfsdfljlbjkljlkjsdfoiwje"
# export CF_Email"xxxxxxxxxxxxx"
# acme.sh --issue --dns dns_cf -d mysite.io -d www.mysite.io -w /var/www/htdocs --force > dnschalleng.log
2>&1
Everything works fine on port 80, but after turning on https (which is a button at the Cloudflare panel), the site throws a 521 error. According the Cloudflare documentation, this is almost always a firewall issue, where Cloudflare's proxies are being blocked. That said, I added log to every inbound rule in pf.conf, and fired up pflog0 to search for the proxies, which are listed on Cloudflare's site. It appears that the proxies are not being blocked or rate limited. In fact, they appear to be passing on port 443. Here is a glimpse of pflog0 showing one of the proxies passing:
Code: 
sudo tcpdump -n -e -ttt -v -i pflog0 inbound

    162.158.255.30.33426 > XXX.XX.XXX.XXX.443: Flags [S], cksum 0x6157 (correct), seq
1430716507, win 29200, options [mss 1460,nop,nop,sackOK,nop,wscale 10], length 0
00:00:00.221510 rule 9/0(match): pass in on if0: (tos 0x0, ttl 59, id 24959, offset 0, flags [DF], proto TCP (6), length 52)
Regardless, I created a whitelist for Cloudflare's proxies, even though they were passing without it, and that made no difference. I also checked my webroot location and permissions and they appear to be fine. The webroot is set to /var/www/htdocs in .acme.sh/mysite.io/mysite.io.conf, and all of the static content in the site is readable by the world.

Any thoughts as to why I'm getting a 521 error? Below are my /etc/pf.conf and /usr/local/etc/obhttpd.conf files.

Thank you


/usr/local/etc/obhttpd.conf
Code: 
public_ip = "XXX.XXX.XX.XX"
chroot "/var/www"
server "www.mysite.io" {
    alias mysite.io
    listen on $public_ip tls port 443
    root "/htdocs"
}

/etc/pf.conf
Code: 
ext_if = "if0"
tcp_services= "{ 22 53 123 }"
udp_services= "{ 53 123 }"
web_services= "{ 80 443 }"
icmp4_messages= "{ echoreq }"
table <blackhats> persist
table <crawlers> persist
table <rfc6890> { 0.0.0.0/8 10.0.0.0/8 100.64.0.0/10 127.0.0.0/8 \
                  169.254.0.0/16 172.16.0.0/12 192.0.0.0/24      \
                  192.0.0.0/29 192.0.2.0/24 192.88.99.0/24       \
                  192.168.0.0/16 198.18.0.0/15 198.51.100.0/24   \
                  203.0.113.0/24 240.0.0.0/4 255.255.255.255/32 }

table <cloudflareIPv4> { 173.245.48.0/20 103.21.244.0/22 103.22.200.0/22  \
                         103.31.4.0/22 141.101.64.0/18 108.162.192.0/18   \
                         190.93.240.0/20 188.114.96.0/20 197.234.240.0/22 \
                         198.41.128.0/17 162.158.0.0/15 104.16.0.0/12     \
                         172.64.0.0/13 131.0.72.0/22 }

set skip on lo0
scrub in all
antispoof log quick for $ext_if
block in log quick on $ext_if from <rfc6890>
block return out log quick on egress to <rfc6890>

block log all

pass in log on $ext_if from <cloudflareIPv4>
pass in log proto tcp to port { 443 } \
    keep state (max-src-conn 100, max-src-conn-rate 3/1, \
        overload <crawlers> flush global) tag WEB
pass in log proto tcp to port { 22 } \
    keep state (max-src-conn 15, max-src-conn-rate 3/1, \
        overload <blackhats> flush global) tag SSH

pass out proto tcp to port $web_services
pass out proto tcp to port $tcp_services
pass out proto udp to port $udp_services
pass out inet proto icmp icmp-type $icmp4_messages
 
Cloudflare has specific certificates you need to use. You don't need a Let's Encrypt (or any other SSL certificate), Cloudflare already takes care of this. Then install a special Cloudflare SSL certificate on your webserver.
 
SirDice said:
Cloudflare has specific certificates you need to use. You don't need a Let's Encrypt (or any other SSL certificate), Cloudflare already takes care of this. Then install a special Cloudflare SSL certificate on your webserver.
Click to expand...

Hi Sir Dice,

Yes, after a disastrous first attempt (even hit my rate limit) with Let's Encrypt I switched over to Cloudflare's Dedicated SSL service. However it's still throwing a 521 error. With Cloudlfare, there's no certs or keys installed on the server. So then how does httpd.conf work when there's no key file or cert to point to?

My current /etc/httpd.conf looks like this:

Code: 
chroot "/var/www"

server "mysite.io" {
    alias "www.mysite.io"
    listen on * tls port 443
    root "/htdocs/mysite.io"
}
 
Yes, I didn't realize there are two sets of certs and keys in play, one between client and Cloudflare, the other between Cloudflare and origin server. On the former, SSL is turned on at the Cloudflare panel, on the latter, the cert and key are installed on the server. Regardless, I generated the key and cert, installed them on the server, gave them 600 permissions, and it still doesn't work. It's crashing the webserver during the tls handshake.

I built httpd from ports with libressl so the cert and key go in /usr/local/etc/ssl as far as I know. I also tried putting them in /etc/ssl and that made no difference.

Here's my latest httpd.conf:
Code: 
chroot "/var/www"
server "mysite.io" {
    alias "www.mysite.io"
    listen on * tls port 443
    tls certificate "/usr/local/etc/ssl/cert.pem"
    tls key "/usr/local/etc/ssl/private/key.pem"
    root "/htdocs/mysite.io"
}
 
justinnoor said:
Yes, I didn't realize there are two sets of certs and keys in play, one between client and Cloudflare, the other between Cloudflare and origin server.
Click to expand...
Correct. And you don't need to generate the Cloudflare -> Origin certificates, Cloudflare already provides those for you, you just need to download them. I can't login at Cloudflare at the moment or else I would be able to exactly tell you where to get them.
 
SirDice said:
And you don't need to generate the Cloudflare -> Origin certificates, Cloudflare already provides those for you
Click to expand...

Yes, I believe you are refering to the Cloudflare -> SSL/TLS -> Origin Server -> Create Certificate button. I'm almost positive we are talking about the same key, the one that sits between Cloudflare and the origin server. It is a formal Origin CA Certificate issued exclusively to a domain. I believe it has to be generated, though. Here's the instructions I'm referring to. The other key between client and Cloudflare is included with every account and requires no action, other than turning it on at the control panel.
 
Anyways the server crashes at the tls handshake. The 521 error is generated because there's no server running after it crashes. I don't remember the details, but I vaguely recall a having similar issue with ports and LibreSSL in the past. I'm leaning towards rebuilding with packages, and not using LibreSSL. I only used it because I might need some OpenBSD stuff in the future that relies on it. But those can easily be replaced with something else. Besides 12.1-RELEASE has BearSSL now, right?
 
justinnoor said:
Besides 12.1-RELEASE has BearSSL now, right?
Click to expand...
Not for SSL as I understood it.
Code: 
dice@molly:~ % ll /usr/lib/libssl.so
lrwxr-xr-x  1 root  wheel  13 Nov 16 17:12 /usr/lib/libssl.so -> libssl.so.111
dice@molly:~ % openssl version
OpenSSL 1.1.1d-freebsd  10 Sep 2019
dice@molly:~ % uname -a
FreeBSD molly.dicelan.home 12.1-STABLE FreeBSD 12.1-STABLE r354774 GENERIC  amd64

justinnoor said:
Anyways the server is crashing at the tls handshake.
Click to expand...
I'm not familiar with www/obhttpd (never used it) but both Apache and nginx have an error log, I assume obhttpd has one too. You can usually find the reason why it's not working there.

As far as I can remember the server certificate I got from Cloudflare was in the correct format and I could use it in HAProxy. One of the reasons why it might fail is that the certificate and/or key are in the wrong format. Most of the time you can use openssl(1) to convert from one format to another.
 
Yeah there’s nothing in /var/www/logs/error.log. Obhttpd requires PEM format, which I selected when creating the keys. Back to the drawing board. Thanks for all the help.
 
You must log in or register to reply here.
Back
Top