Solved DST Root CA X3 certificate has expired

Hi all

One of Let's Encrypts Root Certificates (DST Root CA X3) has expired, as announced a few months back.
I'm using the security/dehydrated pkg to keep my server certificates up to date - which seem to work just fine.

Now I face the issue that client software still complains about expired certificates. How would I fix this?

For example
Code:
thor% openssl s_client -showcerts -host valid-isrgrootx1.letsencrypt.org -port 443
CONNECTED(00000003)
depth=3 O = Digital Signature Trust Co., CN = DST Root CA X3
verify error:num=10:certificate has expired
notAfter=Sep 30 14:01:15 2021 GMT
verify return:1
depth=3 O = Digital Signature Trust Co., CN = DST Root CA X3
notAfter=Sep 30 14:01:15 2021 GMT
verify return:1
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
notAfter=Sep 30 18:14:03 2024 GMT
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
notAfter=Sep 15 16:00:00 2025 GMT
verify return:1
depth=0 CN = valid-isrgrootx1.letsencrypt.org
notAfter=Jan  4 15:00:21 2022 GMT
verify return:1
---
Certificate chain
 0 s:CN = valid-isrgrootx1.letsencrypt.org
   i:C = US, O = Let's Encrypt, CN = R3

Also other software like php-fpm couldn't connect to my local IMAP server anymore. I could fix the issue with specifying:
Code:
openssl.cafile=/usr/local/share/certs/ca-root-nss.crt

I wonder if there is a system wide fix for this behaviour.

Cheers
 
Last edited by a moderator:
Make sure your security/ca_root_nss is up to date. It contains the root CA certificates for clients.

I'm running FreeBSD:13:amd64, quarterly pkgs which uses version 3.63. Last version would be 3.69_1.
So it looks like I've to be patient.

Code:
thor% pkg info | grep ca_root_nss
ca_root_nss-3.63               Root certificate bundle from the Mozilla Project
 
Last edited by a moderator:
A new quarterly was recently made, that's busy getting built.
 
Hi again

So today I was able to install the quarterly pkg upgrade. Now ca_root_nss is bumped to 3.69_1.

Code:
thor% pkg info | grep ca_root_nss
ca_root_nss-3.69_1             Root certificate bundle from the Mozilla Project

Nevertheless my openssl still reports an expired certificate.

Code:
thor% openssl s_client -showcerts -host valid-isrgrootx1.letsencrypt.org -port 443
CONNECTED(00000003)
depth=3 O = Digital Signature Trust Co., CN = DST Root CA X3
verify error:num=10:certificate has expired
notAfter=Sep 30 14:01:15 2021 GMT
verify return:1
depth=3 O = Digital Signature Trust Co., CN = DST Root CA X3
notAfter=Sep 30 14:01:15 2021 GMT
verify return:1
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
notAfter=Sep 30 18:14:03 2024 GMT
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
notAfter=Sep 15 16:00:00 2025 GMT
verify return:1
depth=0 CN = valid-isrgrootx1.letsencrypt.org
notAfter=Jan  4 15:00:21 2022 GMT
verify return:1
---

Anything else I could do? OpenSSL is the base one, my /etc/ssl/openssl.conf is untouched, as well as /usr/share/certs/trusted/*.
I noticed that the trusted directory contains the DST_Root_CA_X3.pem file.
 
Last edited by a moderator:
Yes, I think both, system and pkg are up to date.

Code:
thor% freebsd-version -kru
13.0-RELEASE-p4
13.0-RELEASE-p4
13.0-RELEASE-p4

Do you get a different output with the following command?
openssl s_client -showcerts -host valid-isrgrootx1.letsencrypt.org -port 443
 
Last edited by a moderator:
Verifies just fine on Freebsd 12.2
Code:
$ openssl version
OpenSSL 1.1.1h-freebsd  22 Sep 2020
$ openssl s_client -showcerts -host valid-isrgrootx1.letsencrypt.org -port 443
CONNECTED(00000003)
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = valid-isrgrootx1.letsencrypt.org
verify return:1
...
    Start Time: 1633890795
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
---
closed
What does which openssl return on your system?
 
When you test with :
Code:
openssl s_client -showcerts   -connect  valid-isrgrootx1.letsencrypt.org:443
You receive the full certificate chain in the response, so it should not matter which root certificates you have installed.

The wrong certificate you are seeing in the response is actually what is included in package ca_root_nss:

Code:
From /etc/ssl/cert.pem

Serial Number:
            44:af:b0:80:d6:a3:27:ba:89:30:39:86:2e:f8:40:6b
        Signature Algorithm: sha1WithRSAEncryption
        Issuer: O = Digital Signature Trust Co., CN = DST Root CA X3
        Validity
            Not Before: Sep 30 21:12:19 2000 GMT
            Not After : Sep 30 14:01:15 2021 GMT


Maybe you have some special config somewhere that tells openssl to ignore fullchain response and look at local files instead?
I have no idea if that is even possible.
 
You receive the full certificate chain in the response, so it should not matter which root certificates you have installed.

Hmm yeah. I just wonder why two different systems (Ubuntu / FreeBSD) come up with two different certificate chains and finally conclusions on the validity.

Ubuntu 21.04

Code:
ubuntu@farmer:~$ openssl version
OpenSSL 1.1.1j  16 Feb 2021
ubuntu@farmer:~$ openssl s_client -showcerts   -connect  valid-isrgrootx1.letsencrypt.org:443
CONNECTED(00000003)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = valid-isrgrootx1.letsencrypt.org
verify return:1
---
Certificate chain
 0 s:CN = valid-isrgrootx1.letsencrypt.org
   i:C = US, O = Let's Encrypt, CN = R3
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
---
Server certificate
subject=CN = valid-isrgrootx1.letsencrypt.org

issuer=C = US, O = Let's Encrypt, CN = R3

---
No client certificate CA names sent
Peer signing digest: SHA512
Peer signature type: RSA
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 3369 bytes and written 450 bytes
Verification: OK
---

FreeBSD 13.0

Code:
thor% openssl version
OpenSSL 1.1.1k-freebsd  24 Aug 2021
thor% openssl s_client -showcerts   -connect  valid-isrgrootx1.letsencrypt.org:443
CONNECTED(00000003)
depth=3 O = Digital Signature Trust Co., CN = DST Root CA X3
verify error:num=10:certificate has expired
notAfter=Sep 30 14:01:15 2021 GMT
verify return:1
depth=3 O = Digital Signature Trust Co., CN = DST Root CA X3
notAfter=Sep 30 14:01:15 2021 GMT
verify return:1
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
notAfter=Sep 30 18:14:03 2024 GMT
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
notAfter=Sep 15 16:00:00 2025 GMT
verify return:1
depth=0 CN = valid-isrgrootx1.letsencrypt.org
notAfter=Jan  4 15:00:21 2022 GMT
verify return:1
---
Certificate chain
 0 s:CN = valid-isrgrootx1.letsencrypt.org
   i:C = US, O = Let's Encrypt, CN = R3
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
---
Server certificate
subject=CN = valid-isrgrootx1.letsencrypt.org

issuer=C = US, O = Let's Encrypt, CN = R3

---
No client certificate CA names sent
Peer signing digest: SHA512
Peer signature type: RSA
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 3369 bytes and written 460 bytes
Verification error: certificate has expired
---

Ultimately I have at least www/matomo which uses a Python script to import some apache log files which fails since weeks with the following Error:

Code:
 File "/usr/local/lib/python3.8/ssl.py", line 1309, in do_handshake
   self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: [B]certificate has expired[/B] (_ssl.c:1136)

I'm more than happy for any hints ...
 
Try updating your security/ca_root_nss once more, quarterly should have 3.69_1 and it works with that. Builds have been done but I'm not sure if everything has been synced to the package mirrors yet.

I haven't updated this system yet and it just happens to have the same version as quarterly has.
Code:
dice@williscorto:~ % pkg version -vRx ca_root
ca_root_nss-3.69_1                 <   needs updating (remote has 3.71)
dice@williscorto:~ % openssl s_client -showcerts -connect valid-isrgrootx1.letsencrypt.org:443
CONNECTED(00000003)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = valid-isrgrootx1.letsencrypt.org
verify return:1
---
Certificate chain
 0 s:CN = valid-isrgrootx1.letsencrypt.org
   i:C = US, O = Let's Encrypt, CN = R3
-----BEGIN CERTIFICATE-----
MIIFRjCCBC6gAwIBAgISA311SO/woQ7N3paQNcXYRxZcMA0GCSqGSIb3DQEBCwUA
MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQD
EwJSMzAeFw0yMTEwMDYxNTAwMjJaFw0yMjAxMDQxNTAwMjFaMCsxKTAnBgNVBAMT
IHZhbGlkLWlzcmdyb290eDEubGV0c2VuY3J5cHQub3JnMIIBIjANBgkqhkiG9w0B
AQEFAAOCAQ8AMIIBCgKCAQEA5VClDVNyJ6G9lssK4vw/hZ2EAfG5N46GdjDWt2OQ
GB8VVPlFOTSN9z2I2vArjRotBqrpJ5nAhU2vY/VJo1/zHxEgNbVcsSRiOeQd5vdI
OYR8MP2clIxBhs3Xlphy6GKO6D4SDEuLnp039ZqoSJ3vtIqbkimtSzJIAnNE2fYu
m4Qq58h7DSTLGADOK6Xpxbi4/152Hh3TkkBgXQQH0g7dD4yRaM4EgK0GoYQ8px+D
4aTS0tSEGvHPROGlA55ImgYzMnCOSdnfTCI+IGPh48nCimGY9Smw+enZc2xH96JE
o6i++MDW8z29qSvXq4u+3sZv3zY8EGDW2ZkhU2NzHdbQzwIDAQABo4ICWzCCAlcw
DgYDVR0PAQH/BAQDAgWgMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAM
BgNVHRMBAf8EAjAAMB0GA1UdDgQWBBRxEysX1wVqSxDuWJG28FkLpimKIDAfBgNV
HSMEGDAWgBQULrMXt1hWy65QCUDmH6+dixTCxjBVBggrBgEFBQcBAQRJMEcwIQYI
KwYBBQUHMAGGFWh0dHA6Ly9yMy5vLmxlbmNyLm9yZzAiBggrBgEFBQcwAoYWaHR0
cDovL3IzLmkubGVuY3Iub3JnLzArBgNVHREEJDAigiB2YWxpZC1pc3Jncm9vdHgx
LmxldHNlbmNyeXB0Lm9yZzBMBgNVHSAERTBDMAgGBmeBDAECATA3BgsrBgEEAYLf
EwEBATAoMCYGCCsGAQUFBwIBFhpodHRwOi8vY3BzLmxldHNlbmNyeXB0Lm9yZzCC
AQQGCisGAQQB1nkCBAIEgfUEgfIA8AB2AG9Tdqwx8DEZ2JkApFEV/3cVHBHZAsEA
KQaNsgiaN9kTAAABfFZVmlcAAAQDAEcwRQIhAJNzn1iyjao4wrycwtHYnFNv2P9S
Pw6nTbHmQ+H9/SEaAiAb9EJAAQyWjskm3GLpFcQ38K5d+LQ2MBT7GhmvIzpA8QB2
AEalVet1+pEgMLWiiWn0830RLEF0vv1JuIWr8vxw/m1HAAABfFZVm2kAAAQDAEcw
RQIgWfHl/CWqZynfi2d71hbqPEbbB1AAhxuUY5XmGyzv3+kCIQCgDZOcDFTTo8vg
TL99JWRQvAUKiQK7MRqlz8dgSzGwOzANBgkqhkiG9w0BAQsFAAOCAQEAgyZYOYJs
pYX/spl0yeotRTGjeR4o56jHCV6TR8qAd7SXM3Wx4FaxF9gAfGMRDBjuXmUx82+G
mBecn/yLR3mCU3Ud+M3Xn7WBjD9UM6vfFiQwswkBmu+eIPH98a1snEKqg2kaESla
9FGBGij/TPLbN8IK+FK9gulLfhA6yKFmQZ3fUH+B3LbuHgp44U02T5iu6iVCdQEK
X9Eq1JGcrPKT6wNWxBWsvzJcVKFg4qPXe6jiFgHbsI/7v3gLopY3wxis7JCg45FY
s/FwZDgBtA/zct/o3nz5BM2XBqDqcd6FIWXV1sF8RLhQVAETUpaj6BoQtWK0xwPI
ZDXlqKi8W2H8/g==
-----END CERTIFICATE-----
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
-----BEGIN CERTIFICATE-----
MIIFFjCCAv6gAwIBAgIRAJErCErPDBinU/bWLiWnX1owDQYJKoZIhvcNAQELBQAw
TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh
cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwHhcNMjAwOTA0MDAwMDAw
WhcNMjUwOTE1MTYwMDAwWjAyMQswCQYDVQQGEwJVUzEWMBQGA1UEChMNTGV0J3Mg
RW5jcnlwdDELMAkGA1UEAxMCUjMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK
AoIBAQC7AhUozPaglNMPEuyNVZLD+ILxmaZ6QoinXSaqtSu5xUyxr45r+XXIo9cP
R5QUVTVXjJ6oojkZ9YI8QqlObvU7wy7bjcCwXPNZOOftz2nwWgsbvsCUJCWH+jdx
sxPnHKzhm+/b5DtFUkWWqcFTzjTIUu61ru2P3mBw4qVUq7ZtDpelQDRrK9O8Zutm
NHz6a4uPVymZ+DAXXbpyb/uBxa3Shlg9F8fnCbvxK/eG3MHacV3URuPMrSXBiLxg
Z3Vms/EY96Jc5lP/Ooi2R6X/ExjqmAl3P51T+c8B5fWmcBcUr2Ok/5mzk53cU6cG
/kiFHaFpriV1uxPMUgP17VGhi9sVAgMBAAGjggEIMIIBBDAOBgNVHQ8BAf8EBAMC
AYYwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMBMBIGA1UdEwEB/wQIMAYB
Af8CAQAwHQYDVR0OBBYEFBQusxe3WFbLrlAJQOYfr52LFMLGMB8GA1UdIwQYMBaA
FHm0WeZ7tuXkAXOACIjIGlj26ZtuMDIGCCsGAQUFBwEBBCYwJDAiBggrBgEFBQcw
AoYWaHR0cDovL3gxLmkubGVuY3Iub3JnLzAnBgNVHR8EIDAeMBygGqAYhhZodHRw
Oi8veDEuYy5sZW5jci5vcmcvMCIGA1UdIAQbMBkwCAYGZ4EMAQIBMA0GCysGAQQB
gt8TAQEBMA0GCSqGSIb3DQEBCwUAA4ICAQCFyk5HPqP3hUSFvNVneLKYY611TR6W
PTNlclQtgaDqw+34IL9fzLdwALduO/ZelN7kIJ+m74uyA+eitRY8kc607TkC53wl
ikfmZW4/RvTZ8M6UK+5UzhK8jCdLuMGYL6KvzXGRSgi3yLgjewQtCPkIVz6D2QQz
CkcheAmCJ8MqyJu5zlzyZMjAvnnAT45tRAxekrsu94sQ4egdRCnbWSDtY7kh+BIm
lJNXoB1lBMEKIq4QDUOXoRgffuDghje1WrG9ML+Hbisq/yFOGwXD9RiX8F6sw6W4
avAuvDszue5L3sz85K+EC4Y/wFVDNvZo4TYXao6Z0f+lQKc0t8DQYzk1OXVu8rp2
yJMC6alLbBfODALZvYH7n7do1AZls4I9d1P4jnkDrQoxB3UqQ9hVl3LEKQ73xF1O
yK5GhDDX8oVfGKF5u+decIsH4YaTw7mP3GFxJSqv3+0lUFJoi5Lc5da149p90Ids
hCExroL1+7mryIkXPeFM5TgO9r0rvZaBFOvV2z0gp35Z0+L4WPlbuEjN/lxPFin+
HlUjr8gRsI3qfJOQFy/9rKIJR0Y/8Omwt/8oTWgy1mdeHmmjk7j1nYsvC9JSQ6Zv
MldlTTKB3zhThV1+XWYp6rjd5JW1zbVWEkLNxE7GJThEUG3szgBVGP7pSWTUTsqX
nLRbwHOoq7hHwg==
-----END CERTIFICATE-----
---
Server certificate
subject=CN = valid-isrgrootx1.letsencrypt.org

issuer=C = US, O = Let's Encrypt, CN = R3

---
No client certificate CA names sent
Peer signing digest: SHA512
Peer signature type: RSA
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 3369 bytes and written 460 bytes
Verification: OK
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES128-GCM-SHA256
    Session-ID: 1B9AC6512E46064132C4973639EB35AACB7222A86DDBBE405C5EA688112E3DBD
    Session-ID-ctx:
    Master-Key: 0DB703CE2AD846793A449880895BBE271C4F958CBCCE4B50335AD848366A7D6580E4944D3E01FA12D55A545EA6AE030F
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 86400 (seconds)
    TLS session ticket:
    0000 - 2f e2 41 e8 bd 32 36 4d-95 bf 83 fe 38 5c 10 7a   /.A..26M....8\.z
    0010 - 92 33 9c 28 4b 5a 95 e7-da fd 7c 7d 26 c6 e1 d5   .3.(KZ....|}&...
    0020 - 50 c3 a2 fa 5e 41 50 86-a2 ff 72 6b 0c 3f 54 3b   P...^AP...rk.?T;
    0030 - fc 03 fc 7b 93 fd 53 c8-70 de 05 71 26 7f c8 d1   ...{..S.p..q&...
    0040 - 0f 42 69 89 98 25 75 d8-f7 fd 27 a5 a6 5d ac eb   .Bi..%u...'..]..
    0050 - 69 d4 b3 ac 8a 15 00 6f-45 98 59 63 db e4 2f 00   i......oE.Yc../.
    0060 - f6 e2 87 64 e6 3d 96 e7-4e 01 f5 eb ab ee 3b 3c   ...d.=..N.....;<
    0070 - 68 69 eb 51 2c 93 ff d3-29 ff a7 ad 04 29 dc 58   hi.Q,...)....).X
    0080 - e9 48 31 81 60 e0 e7 08-8c d0 e2 f8 67 91 78 e0   .H1.`.......g.x.
    0090 - 87 f4 db 64 64 bf ed 27-62 42 8c fe 56 c0 51 6f   ...dd..'bB..V.Qo
    00a0 - 9e 3f 64 34 e1 4f bb 51-47 4c 69 3b 65 77 ca 3e   .?d4.O.QGLi;ew.>
    00b0 - 98 16 d2 c8 80 07 c0 f1-7b 3b 84 ec 8d 3a 83 06   ........{;...:..
    00c0 - 6b 44 f0 da 7c b1 9a 6b-65 37 bf b2 13 fd f3 b9   kD..|..ke7......

    Start Time: 1634022625
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
---
DONE
dice@williscorto:~ %
 
gldecurtins
I agree it is wierd, I have freebsd and linux systems working without issues, it seems to be some speciality on your setup.
I do notice however that on all mine servers, SirDice's output and your linux there is:

Code:
ubuntu@farmer:~$ openssl s_client -showcerts   -connect  valid-isrgrootx1.letsencrypt.org:443
CONNECTED(00000003)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1

That is depth=2, but on you freebsd, depth=3

Code:
thor% openssl version
OpenSSL 1.1.1k-freebsd  24 Aug 2021
thor% openssl s_client -showcerts   -connect  valid-isrgrootx1.letsencrypt.org:443
CONNECTED(00000003)
depth=3 O = Digital Signature Trust Co., CN = DST Root CA X3


This is different:
Code:
depth=3 O = Digital Signature Trust Co., CN = DST Root CA X3
verify error:num=10:certificate has expired
notAfter=Sep 30 14:01:15 2021 GMT
verify return:1

depth=3 O = Digital Signature Trust Co., CN = DST Root CA X3
notAfter=Sep 30 14:01:15 2021 GMT
verify return:1

Any chance you have some secret transparent proxy between your freebsd and valid-isrgrootx1.letsencrypt.org?
 
Try if this helps:
rm /usr/local/etc/ssl/cert.pem
pkg install -f ca_root_nss
 
Hi all

I found the reason, after setting up a virtual machine with FreeBSD 13.0.
All the configuration files were the same, versions of ca_root_nss as well.

The only difference was that the folder /usr/local/share/certs contained my let's encrypt certificates (security/dehydrated) as well.
After moving them to a different folder and executing certctl rehash it started to work as expected.

I thought it was a good idea to put them into the certs folder - looks like I've been proven wrong. Where do you store them?

Thanks for all your proposals and patience, appreciate!
 
Back
Top