Solved OpenSSL reports my Let's Encrypt certificate chain contains the expired "DST Root CA X3" - it doesn't.

[LSDave]

Hello, I use a certbot to generate Let's Encrypt certs, and anvil to push and pull them around to various jails hosting websites. From the client-side, the certs work as expected--no validity or expiry issues. In Windows, I can inspect the certificate chain and see all is in order.

But locally, when I try to verify the certificate with OpenSSL, I get the dreaded:

# openssl verify -verbose -CAfile /usr/local/etc/ssl/mydomain.com.fullchain.cer /usr/local/etc/ssl/mydomain.com.cer
O = Digital Signature Trust Co., CN = DST Root CA X3
error 10 at 3 depth lookup: certificate has expired
error /usr/local/etc/ssl/dpglawyer.com.fullchain.cer: verification failed

I've read all about this. I know the Let's Encrypt root certificate complained of expired in 2021. But from my inspection from Windows, I confirm the expired certificate is NOT part of the cert chain found in mydomain.com.fullchain.cer. Somehow, OpenSSL is finding it on all my FreeBSD instances.

I read: https://forums.freebsd.org/threads/dst-root-ca-x3-certificate-has-expired.82364/. I followed SirDice's suggestions.

#freebsd-version
13.1-RELEASE-p5

# openssl version
OpenSSL 1.1.1o-freebsd  3 May 2022

#pkg -v
1.19.0

#rm /usr/local/etc/ssl/cert.pem
#pkg install -f ca_root_nss
#certctl rehash

#pkg info | grep ca_root_nss
ca_root_nss-3.86               Root certificate bundle from the Mozilla Project

When I run

#openssl s_client -showcerts   -connect  valid-isrgrootx1.letsencrypt.org:443

I get the expected output--no expired certs reported.

But when I verify my own certificates, OpenSSL continues to complain of expired DST Root CA X3.

The only file in /usr/local/share/certs is ca-root-nss.crt, current to January 2, 2023.

I have two hosts, running five jails that host web and mail services. The hosts and the jails all report the same problem. Even on my certbot jail openssl reports the problem.

I am at a loss and welcome any advice or direction you can offer.

Thanks

 

[cmoerz]

You're mentioning /usr/local/etc/ssl/mydomain.com.fullchain.cer and /usr/local/etc/ssl/mydomain.com.cer but the error message points to /usr/local/etc/ssl/dpglawyer.com.fullchain.cer. I assume this might be a copy/paste issue?

Did you review the chain using openssl? What's at that 3rd spot that openssl points to? Do the certificate hashes openssl reads match what you're seeing from connecting to your host? I.e. you sure you're comparing the same files at the end?
I remember "a friend" going down a similar rabbit hole once and then realizing he had somehow replaced files with old versions. (Would never happen to myself, for sure...)

 

[SirDice]

Is /etc/ssl/cert.pem still a symlink to ../../usr/local/share/certs/ca-root-nss.crt? I've seen some instances where /etc/ssl/cert.pem was a copy of an older ca-root-nss.crt.

# ll /etc/ssl/cert.pem
lrwxr-xr-x  1 root  wheel  43 Jan 11 13:20 /etc/ssl/cert.pem@ -> ../../usr/local/share/certs/ca-root-nss.crt

 

[LSDave]

I've figured out the problem. I was verifying mydomain.com.cer against -CAfile mydomain.com.fullchain.cer. The problem is that mydomain.com.fullchain.cer is three certificates, including the final, end-user (i.e., webserver) certificate that is also found in mydomain.com.cer. So I when I was verifying, I guess OpenSSL would find the end certificate twice, and doesn't like that.

#cp /usr/local/etc/ssl/mydomain.com.fullchain.cer /usr/local/etc/ssl/CA-chain.cer

I vi'd into CA-chain.cer and deleted the last of the three certs, saved, and ran:

#openssl verify -verbose -CAfile /usr/local/etc/ssl/CA-chain.cer /usr/local/etc/ssl/mydomain.com.cer
/usr/local/etc/ssl/mydomain.com.cer: OK

Not exactly the verbose answer I was hoping for, but problem solved, apparently.

I still have no clue why duplicate end-user certificate would cause OpenSSL to report an error about the expired "DST Root CA X3." But I must pick my battles and so will save the mystery for someone else.

Thanks for your help.