pkg & Private Certificate Authority

schrodinger

schrodinger

I have a poudriere build system and pkgs are hosted via Nginx over HTTPS using an SSL cert signed by an internal CA.

Certs are installed onto hosts in /usr/local/etc/ssl/certs/ and works fine - until security/ca_nss_root is installed. When this is installed /etc/ssl/cert.pem takes precedence and I get SSL verify errors from pkg. To resolve the issue I have to append the internal CA root and intermediate certificates to /etc/ssl/cert.pem but this is reverted every time ca_nss_root is upgraded and I get verify errors again from pkg.

My question is: Where do I place my CA certs so that pkg will see them and I don't have to manually intervene?
 
Yes.

Code: 
# certctl list | grep -i hyperion
1c6e8955.0    Hyperion XNet Intermediate CA
1d8e0e4e.0    Hyperion XNet Root CA
 
The issue is with the certificate presented during the fetch that pkg performs not with the key used to sign packages.

Code: 
# pkg update
Updating konundrum_server repository catalogue...
Certificate verification failed for /C=IE/ST=Leinster/L=Dublin/O=Hyperion/OU=Pegasus WWW/CN=pegasus.hyperion.xnet/emailAddress=root@hyperion.xnet
34378686464:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1921:
 
Code: 
# pkg install -f security/ca_root_nss
Updating konundrum_server repository catalogue...
[host1.hyperion.xnet] Fetching meta.conf: 100%    163 B   0.2kB/s    00:01
[host1.hyperion.xnet] Fetching packagesite.pkg: 100%  183 KiB 186.9kB/s    00:01
Processing entries: 100%
konundrum_server repository update completed. 659 packages processed.
All repositories are up to date.
Checking integrity... done (0 conflicting)
The following 1 package(s) will be affected (of 0 checked):

Installed packages to be REINSTALLED:
    ca_root_nss-3.87

Number of packages to be reinstalled: 1

Proceed with this action? [y/N]: y
[host1.hyperion.xnet] [1/1] Reinstalling ca_root_nss-3.87...
[host1.hyperion.xnet] [1/1] Extracting ca_root_nss-3.87: 100%
# fetch https://pkgs.hyperion.xnet
Certificate verification failed for /C=IE/ST=Leinster/L=Dublin/O=Hyperion/OU=Pegasus WWW/CN=pegasus.hyperion.xnet/emailAddress=root@hyperion.xnet
34389172224:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1921:
fetch: https://pkgs.hyperion.xnet: Authentication error
# cat /usr/local/etc/ssl/certs/Hyperion_Xnet_Root.pem >> /etc/ssl/cert.pem
# cat /usr/local/etc/ssl/certs/Hyperion_Xnet_Intermediate.pem >> /etc/ssl/cert.pem
# fetch https://pkgs.hyperion.xnet
pkgs.hyperion.xnet                                    6178  B   51 MBps    00s
#
 
After certctl rehash your CA x509 hash should be created in /etc/ssl/certs then if you try with openssl s_client -connect pkgs.hyperion.xnet:443 | grep verify it should return "1"

Edit:
Try to put both the CA cert and the Intermediate certificate in one .pem file
 
Code: 
# openssl s_client -connect pkgs.hyperion.xnet:443 | grep verify
depth=2 C = IE, ST = Leinster, L = Dublin, O = Hyperion, OU = Hyperion XNet Root CA, CN = Hyperion XNet Root CA, emailAddress = schrodinger@hyperion.xnet
verify return:1
depth=1 C = IE, ST = Leinster, O = Hyperion, OU = Hyperion XNet Intermediate CA, CN = Hyperion XNet Intermediate CA, emailAddress = schrodinger@hyperion.xnet
verify return:1
depth=0 C = IE, ST = Leinster, L = Dublin, O = Hyperion, OU = Pegasus WWW, CN = pegasus.hyperion.xnet, emailAddress = root@hyperion.xnet
verify return:1
 
There is a bug report discussing this very issue which was marked as "Closed FIXED" but then commented as not merged and I wonder if this was ever really addressed.

193871 – Certificates in /etc/ssl/certs not considered by pkg and fetch

bugs.freebsd.org bugs.freebsd.org

And just also to show that the certs were properly hashed:

Code: 
# certctl list | grep -i hyperion
1c6e8955.0    Hyperion XNet Intermediate CA
1d8e0e4e.0    Hyperion XNet Root CA
# ls -l /etc/ssl/certs/1c6e8955.0
lrwxr-xr-x  1 root  wheel  63 Jan 21 19:59 /etc/ssl/certs/1c6e8955.0 -> ../../../usr/local/etc/ssl/certs/Hyperion_Xnet_Intermediate.pem
# ls -l /etc/ssl/certs/1d8e0e4e.0
lrwxr-xr-x  1 root  wheel  55 Jan 21 19:59 /etc/ssl/certs/1d8e0e4e.0 -> ../../../usr/local/etc/ssl/certs/Hyperion_Xnet_Root.pem
#
 
Thanks a lot. Would really appreciate that. Everything I read, and done in the past, I have certs where they should be and I would consider this a fairly standard practise, to use an internal CA, but fetch seems unhappy to look for certs outside of /etc/ssl/cert.pem
 
Ok, this works for pkg but doesn't fix fetch, that's not a problem for me though - for now (Maybe for things like curl later on.

I added the following to the pkg.conf for my repo /usr/local/etc/pkg/repos/konundrum.conf

Added:

Code: 
 ENV: {
        SSL_CA_CERT_PATH: "/etc/ssl/certs"
    }
 
In FreeBSD 14.0 is working as expected. I will test it in FreeBSD 13.1 also and will post back

Code: 
# openssl s_client appro01.XXXXXX.com:443 | grep verify
depth=0 CN = APPRO01.XXXXXX.com
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = APPRO01.XXXXXX.com
verify error:num=21:unable to verify the first certificate
verify return:1
depth=0 CN = APPRO01.XXXXXX.com
verify return:1
Verification error: unable to verify the first certificate
    Verify return code: 21 (unable to verify the first certificate)
^C
# fetch https://appro01.XXXXXX.com
Certificate verification failed for /CN=APPRO01.XXXXXX.com
46236512817152:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1921:
fetch: https://appro01.XXXXXX.com: Authentication error

# openssl x509 -inform der -in XXX-CA.cer -out XXX-CA.pem
# cp XXX-CA.pem /usr/local/etc/ssl/certs/
# certctl rehash
Scanning /usr/share/certs/untrusted for certificates...
Scanning /usr/share/certs/trusted for certificates...
Scanning /usr/local/etc/ssl/certs for certificates...
# openssl s_client appro01.XXXXXX.com:443 | grep verify
depth=1 DC = com, DC = XXXXXX, CN = XXX-CA
verify return:1
depth=0 CN = APPRO01.XXXXXX.com
verify return:1
^C
# fetch https://appro01.XXXXXX.com
appro01.XXXXXX.com                            1495  B 3803 kBps    00s
# uname -a
FreeBSD btest 14.0-CURRENT FreeBSD 14.0-CURRENT #0 main-n260062-7583c633e0aa: Sat Jan 14 03:36:09 UTC 2023     root@releng1.nyi.freebsd.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64

Edit:
It's also working in FreeBSD 13.1. I'm thinking that your Intermediate (chain) certificate is the issue as it's not in the one file with your trusted root certificate.
Try to create a new PEM certificate which contains both CA + chain certificate in single file.

cat /usr/local/etc/ssl/certs/Hyperion_Xnet_Intermediate.pem /usr/local/etc/ssl/certs/Hyperion_Xnet_Root.pem > /usr/local/etc/ssl/certs/Hyperion_Xnet_CA_chain.pem
chmod 444 /usr/local/etc/ssl/certs/Hyperion_Xnet_CA_chain.pem
Then make a backup of your CA pem and intermediate pem and remove them from /usr/local/etc/ssl/certs

Code: 
# openssl s_client appro01.XXX.com:443 | grep verify
depth=0 CN = APPRO01.XXX.com
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = APPRO01.XXX.com
verify error:num=21:unable to verify the first certificate
verify return:1
depth=0 CN = APPRO01.XXX.com
verify return:1
Verification error: unable to verify the first certificate
    Verify return code: 21 (unable to verify the first certificate)
^C
# fetch https://appro01.XXX.com
Certificate verification failed for /CN=APPRO01.XXX.com
34389172224:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1921:
fetch: https://appro01.XXX.com: Authentication error

# openssl x509 -inform der -in XXX-CA.cer -out XXX-CA.pem
# mkdir -p /usr/local/etc/ssl/certs
# cp XXX-CA.pem /usr/local/etc/ssl/certs/
# certctl rehash
Scanning /usr/share/certs/blacklisted for certificates...
Scanning /usr/share/certs/trusted for certificates...
Scanning /usr/local/etc/ssl/certs for certificates...
# openssl s_client appro01.XXX.com:443 | grep verify
depth=1 DC = com, DC = XXX, CN = XXX-CA
verify return:1
depth=0 CN = APPRO01.XXX.com
verify return:1
^C
# fetch https://appro01.XXX.com
appro01.XXX.com                            1495  B 6919 kBps    00s
# uname -a
FreeBSD btest 13.1-RELEASE FreeBSD 13.1-RELEASE releng/13.1-n250148-fc952ac2212 GENERIC amd64
 
No, it's clean install without any software.
There's no /etc/ssl/cert.pem in place

Edit:
i've installed ca_root_nss and the fetch failed with the same error as yours. So when there's /etc/ssl/cert.pem the fetch is picking that one and fails.

This package installs symlinks to support root certificates discovery by
default for software that uses OpenSSL.

This enables SSL Certificate Verification by client software without manual
intervention.

If you prefer to do this manually, replace the following symlinks with
either an empty file or your site-local certificate bundle.

* /etc/ssl/cert.pem
* /usr/local/etc/ssl/cert.pem
* /usr/local/openssl/cert.pem
Click to expand...
 
I guess I always thought that even with cert.pem in place it would still search directories. I will try shortly without cert.pem symlinked but with ca_root_nss still installed.
 
You must log in or register to reply here.
Back
Top