Solved OpenJDK11 - add private CA to cacerts

byrnejb

byrnejb

We have a requirement to use Java SSL authentication to a host that has its certificate created by a private CA. We have added the private intermediate and root certificates to the cacerts keystore in /usr/local/openjdk11/lib/security/cacerts but we still get this error:

Code: 
openssl s_client -connect mx32.harte-lyne.ca:465
CONNECTED(00000003)
depth=2 CN = CA_HLL_ROOT_2016, ST = Ontario, O = Harte & Lyne Limited, OU = Networked Data Services, C = CA, DC = harte-lyne, DC = ca, L = Hamilton
verify return:1
depth=1 CN = CA_HLL_ISSUER_2016, OU = Networked Data Services, O = Harte & Lyne Limited, L = Hamilton, ST = Ontario, C = CA, DC = harte-lyne, DC = ca
verify return:1
depth=0 CN = mx32.harte-lyne.ca, OU = Networked Data Systems, O = Harte & Lyne Limited, L = Hamilton, ST = Ontario, C = CA, DC = hamilton, DC = harte-lyne, DC = ca
verify return:1
. . .

JAVA_VERSION="11" keytool -list -rfc -cacerts > cacerts.txt

grep 'Alias name' cacerts.txt | grep hll
Alias name: hartelyneissuer2016 [hll]
Alias name: hartelyneroot2016 [hll]

JAVA_VERSION="11" java SSLPoke mx32.harte-lyne.ca 465
sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors


Synopsis of solution:

1. First, the importance of the alias value in the keystore file was not evident to me. Where OpenSSL is given specific certificates and searches the provided CA bundle file serially from bottom to top to validate Java searches the keystore for key certificate pairs using the alias value. Once this matter was cleared up the remaining problems were artifacts of previous attempts at a solution.

2. In the end we used an open source application named KeyStore Explorer http://keystore-explorer.org/ o manage the applicaityon keystore . This is considerably less painful means of managing Java key/trust stores than using keytool.

3. It is possible that having multiple versions of OpenJDK installed contributed to confusing the solution to the problem.
 
Last edited:
Jose said:
Maybe you're running into this?

stackoverflow.com

How to add certificate chain to keystore?

I have file with chain of certificates - certificate.cer: subject=/C... issuer=/C=US/O=VeriSign, Inc... -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- subject=/C=US/O=VeriSign, Inc...
stackoverflow.com stackoverflow.com
Click to expand...
I do not think so. This is how I attempted to add the private CA:

Code: 
# Copy the original cacerts
cp -p /usr/local/openjdk11/lib/security/cacerts
usr/local/openjdk11/lib/security/cacerts.save

cp -p /usr/local/openjdk11/lib/security/cacerts.save /root/hll_jdk11_cacerts

# Add the ROOT CA cert to keystore
JAVA_VERSION="11" keytool -import \
  -file /usr/local/etc/pki/tls/certs/CA_HLL_ROOT_2016.crt \
  -alias 'hartelyneroot2016 [hll]'  \
  -keystore /root/hll_jdk11_cacerts
Enter keystore password:  [changeit]
. . .
Trust this certificate? [no]:  yes
Certificate was added to keystore


# Add the intermediary CA cert to keystore
JAVA_VERSION="11" keytool -import \
  -file /usr/local/etc/pki/tls/certs/CA_HLL_ISSUER_2016.crt \
  -alias 'hartelyneissuer2016 [hll]'  \
  -keystore /root/hll_jdk11_cacerts
Enter keystore password:  [changeit]
Certificate was added to keystore

# Note that the intermediate certificate is Trusted by keytool

# verify keystore
JAVA_VERSION="11" keytool -list \
  -rfc \
  -keystore /root/hll_jdk11_cacerts \
  > /root/cacerts_hll.txt

grep hll /root/cacerts_hll.txt
Alias name: hartelyneissuer2016 [hll]
Alias name: hartelyneroot2016 [hll]

# Replace /usr/local/openjdk11/lib/security/cacerts
cp /root/hll_jdk11_cacerts /usr/local/openjdk11/lib/security/cacert

# Test certificate authority
openssl s_client -connect mx32.harte-lyne.ca:465
CONNECTED(00000003)
depth=2 CN = CA_HLL_ROOT_2016, ST = Ontario, O = Harte & Lyne Limited, OU =
Networked Data Services, C = CA, DC = harte-lyne, DC = ca, L = Hamilton
verify return:1
depth=1 CN = CA_HLL_ISSUER_2016, OU = Networked Data Services, O = Harte & Lyne
Limited, L = Hamilton, ST = Ontario, C = CA, DC = harte-lyne, DC = ca
verify return:1
depth=0 CN = mx32.harte-lyne.ca, OU = Networked Data Systems, O = Harte & Lyne
Limited, L = Hamilton, ST = Ontario, C = CA, DC = hamilton, DC = harte-lyne, DC
= ca
verify return:1
---
Certificate chain
. . .
---
read R BLOCK
220 mx32.harte-lyne.ca ESMTP Postfix

JAVA_VERSION="11" java SSLPoke mx32.harte-lyne.ca 465
sun.security.validator.ValidatorException: PKIX path building failed:
sun.security.provider.certpath.SunCertPathBuilderException: unable to find
valid certification path to requested target
 
acheron said:
Is it related to PR 229329?
Click to expand...
In a way. The fundamental issue is the same, getting a private CA into a file in a format that Java11 recognizes. There are multiple discussions of this all over the Internet and no two seem to solve it the same way. As far as I can tell, what I have done should work. Either I have misunderstood something, or something respecting cacerts is broken.

The interesting thing about PR 229329 is that it mentions that Java14 just uses a pem file as provided with ca_root_nss. SO I will try this with openjdk14
 
Last edited by a moderator:
byrnejb said:
We have a requirement to use Java SSL authentication to a host that has its certificate created by a private CA. We have added the private intermediate and root certificates to the cacerts keystore in /usr/local/openjdk11/lib/security/cacerts but we still get this error:

Code: 
openssl s_client -connect mx32.harte-lyne.ca:465
CONNECTED(00000003)
depth=2 CN = CA_HLL_ROOT_2016, ST = Ontario, O = Harte & Lyne Limited, OU = Networked Data Services, C = CA, DC = harte-lyne, DC = ca, L = Hamilton
verify return:1
depth=1 CN = CA_HLL_ISSUER_2016, OU = Networked Data Services, O = Harte & Lyne Limited, L = Hamilton, ST = Ontario, C = CA, DC = harte-lyne, DC = ca
verify return:1
depth=0 CN = mx32.harte-lyne.ca, OU = Networked Data Systems, O = Harte & Lyne Limited, L = Hamilton, ST = Ontario, C = CA, DC = hamilton, DC = harte-lyne, DC = ca
verify return:1
. . .

JAVA_VERSION="11" keytool -list -rfc -cacerts > cacerts.txt

grep 'Alias name' cacerts.txt | grep hll
Alias name: hartelyneissuer2016 [hll]
Alias name: hartelyneroot2016 [hll]

JAVA_VERSION="11" java SSLPoke mx32.harte-lyne.ca 465
sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors

Why?
Click to expand...
Hi,

Run your java program with -Djavax.net.debug=ssl you should see which certificate is missing.
Look where upcoming handshake states: server finished (if you have other SSL in your program) and errors starts, you can share the log file if you want.
 
Last edited by a moderator:
I have done a lot of work on this and cannot get a solution. I did discover that keytool will only install the first certificate of a certificate bundle, which means that every certificate has to be individually loaded into a keystore. I have read, but have not verified, that the order in which the certificates are added is significant. I have added out private CA root and intermediate certificates into /usr/local/openjdk11/lib/security/cacert:

Code: 
keytool -keystore  /usr/local/openjdk11/lib/security/cacerts -storepass changeit -list | grep hll
hartelyneissuer2016 [hll], Dec 14, 2020, trustedCertEntry,
hartelyneroot2016 [hll], Dec 14, 2020, trustedCertEntry,

But this does not remove the trust anchor error:

Code: 
JAVA_VERSION="11" java -Djavax.net.ssl.keyStorePassword=changeit SSLPoke webmail.harte-lyne.ca 443
sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors

How does one get a private CA certificate recognized by openjdk-11 java?
 
I has been suggested to me that the problem may be related to having more than one version of openjdk installed. I have threrefor removed all but openjdk11. However, I find that is if I simply run java from the command line I get an error:
Code: 
java --version
java: error: no suitable JavaVMs found

The man page for javavm says that:
Code: 
     The Java VM wrapper provides a convenient system for switching between
     different Java VMs.  It also provides symbolic links in /usr/local/bin to
     allow the use of the Java executables without having to add the specific
     Java VM executable directories to the PATH environment variable.
. . .
     By default, javavm will select the most "native" and up to date version
     of the Java VM when a given symbolic link is used, invoking and passing
     the arguments to the matching executable within the chosen Java VM.

I checked /usr/local/etc/javavms and see this:
Code: 
cat /usr/local/etc/javavms
/usr/local/openjdk11/bin/java

However, this is the error I get when I run javavm directly:

Code: 
javavm --version
javavm: warning: The use of 'javavm' as a synonym for 'java' is deprecated
java: error: no suitable JavaVMs found

So, what is the proper way to run Java on FreeBSD?
 
On my system

Code: 
java --version
openjdk 11.0.8 2020-07-14
OpenJDK Runtime Environment (build 11.0.8+10-1)
OpenJDK 64-Bit Server VM (build 11.0.8+10-1, mixed mode)

file `which java`                                                                                              
/usr/local/bin/java: symbolic link to /usr/local/bin/javavm

cat /usr/local/etc/javavms                                                                                     
/usr/local/openjdk11/bin/java
/usr/local/openjdk8/bin/java

Check /usr/local/etc/javavm_opts.conf. Is everything commented out in there? Check your environment. Do you have any JAVAVM_OPTS* variables set?
 
Code: 
[root@accounting-2 ~ (master)]# env | grep -i java | sort
[root@accounting-2 ~ (master)]#

Nothing set for JAVA.

Code: 
[root@accounting-2 ~ (master)]# grep '^#.*' /usr/local/etc/javavm_opts.conf | wc -l
      49
[root@accounting-2 ~ (master)]# grep -v '^#.*' /usr/local/etc/javavm_opts.conf | wc -l
       1
[root@accounting-2 ~ (master)]# grep -v '^#.*' /usr/local/etc/javavm_opts.conf       

[root@accounting-2 ~ (master)]#

Nothing set in /usr/local/etc/javavm_opts.conf.


What I did do, based on another suggestion given elsewhare, was remove all other versions of openjdk except for one (openjdk12). Now when I run java version is see this:

Code: 
[root@accounting-2 ~ (master)]# java --version
openjdk 12.0.2 2019-07-16
OpenJDK Runtime Environment (build 12.0.2+10-4)
OpenJDK 64-Bit Server VM (build 12.0.2+10-4, mixed mode)

This issue is related to an ongoing problem with an application's use of a private keystore file. So, at the present I do not have the luxury of time to reinstall one or more openjdks to see if that reintroduces the problem.
 
We have gotten to the point where we have established that the Java keystore used by the application contains a complete certificate chain to a trust anchor. The SSLPoke class was used to perform this test:

Code: 
JAVA_VERSION="12" java -Djavax.net.ssl.trustStore=/opt/idempiere/idempiere-server/jettyhome/etc/keystore  -Djavax.net.ssl.trustStorePassword=testing  SSLPoke 192.168.216.32 465
Successfully connected

We have also confirmed this using the KeyStore Explorer:
JAVA_KSE_SSL_MX32.png


Where 192.168.216.32 is our MX host running Postfix as the MTA; and 465 is a port listened on by Postfix that uses SSL.

However, we still have the java/Jetty application rejecting the server certificate:
Code: 
Dec 22 10:10:08 mx32 postfix-p25/smtpd[12694]: SSL_accept:SSLv3/TLS read client hello
Dec 22 10:10:08 mx32 postfix-p25/smtpd[12694]: SSL_accept:SSLv3/TLS write server hello
Dec 22 10:10:08 mx32 postfix-p25/smtpd[12694]: SSL_accept:SSLv3/TLS write change cipher spec
Dec 22 10:10:08 mx32 postfix-p25/smtpd[12694]: SSL_accept:TLSv1.3 write encrypted extensions
Dec 22 10:10:08 mx32 postfix-p25/smtpd[12694]: SSL_accept:SSLv3/TLS write certificate
Dec 22 10:10:08 mx32 postfix-p25/smtpd[12694]: SSL_accept:TLSv1.3 write server certificate verify
Dec 22 10:10:08 mx32 postfix-p25/smtpd[12694]: SSL_accept:SSLv3/TLS write finished
Dec 22 10:10:08 mx32 postfix-p25/smtpd[12694]: SSL_accept:TLSv1.3 early data
Dec 22 10:10:08 mx32 postfix-p25/smtpd[12694]: SSL3 alert read:fatal:certificate unknown
Dec 22 10:10:08 mx32 postfix-p25/smtpd[12694]: SSL_accept:error in error
Dec 22 10:10:08 mx32 postfix-p25/smtpd[12694]: SSL_accept error from accounting-2.internal.harte-lyne.ca[192.168.216.88]: -1
Dec 22 10:10:08 mx32 postfix-p25/smtpd[12694]: warning: TLS library problem: error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown:/usr/src/crypto/openssl/ssl/record/rec_layer_s3.c:1544:SSL alert number 46:

The interesting thing here is that the client appears to be using SSLv3 but this is because Postfix uses SSLv3 as the log prefix of all SSL errors.
 
You must log in or register to reply here.
Back
Top