Issue with mariadb 10.2 and openssl after restart (SSL_CTX_set_default_verify_paths)

MYXOMOP

MYXOMOP

I'm having a difficulty starting mariadb 10.2 (10.2.16-MariaDB-log FreeBSD Ports) with SSL support in a jail after restart of the host.

After I setup the jail and restarted mysql-server without restarting the jail, the SSL setup was OK and clients could connect. Now, after 4 days, I had to physically migrate the host machine, so I turned it off. After turning it on again, the jail does not have SSL functionality for connecting clients.

I'm getting the following log upon starting of mysql-server:

Code: 
...
2018-07-16 16:18:15 34424840192 [Warning] Failed to setup SSL
2018-07-16 16:18:15 34424840192 [Warning] SSL error: SSL_CTX_set_default_verify_paths failed
2018-07-16 16:18:15 34424840192 [Warning] SSL error: error:0200100D:system library:fopen:Permission denied
2018-07-16 16:18:15 34424840192 [Warning] SSL error: error:2006D002:BIO routines:BIO_new_file:system lib
2018-07-16 16:18:15 34424840192 [Warning] SSL error: error:0B084002:x509 certificate routines:X509_load_cert_crl_file:system lib
...

The config (usr/local/etc/my.cnf) contains the following:

Code: 
ssl          = 1
ssl-ca       = /usr/app/mariadb/ssl/ca.crt
ssl-key      = /usr/app/mariadb/ssl/mysql-pkcs1.key
ssl-cert     = /usr/app/mariadb/ssl/mysql.crt
ssl-cipher   = ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-SHA
bind-address = 0.0.0.0

The permissions are like that:

Code: 
ls -lah /usr/app/mariadb/ssl   
total 50
drw-r--r--  2 mysql  mysql    11B Jul 12 14:48 .
drwxr-xr-x  3 mysql  mysql     3B Jul 12 12:09 ..
-rw-r--r--  1 mysql  mysql   3.2K Jul 12 13:52 ca-pkcs1.key
-rw-r--r--  1 mysql  mysql   3.2K Jul 12 13:51 ca-pkcs8.key
-rw-r--r--  1 mysql  mysql   2.2K Jul 12 13:50 ca.crt
-rw-r--r--  1 mysql  mysql   3.2K Jul 12 13:58 mysql-pkcs1.key
-rw-r--r--  1 mysql  mysql   3.2K Jul 12 13:57 mysql-pkcs8.key
-rw-r--r--  1 mysql  mysql   2.2K Jul 12 13:56 mysql.crt
-rw-r--r--  1 mysql  mysql   3.2K Jul 12 14:48 mysqlclient-pkcs1.key
-rw-r--r--  1 mysql  mysql   3.2K Jul 12 14:48 mysqlclient-pkcs8.key
-rw-r--r--  1 mysql  mysql   2.1K Jul 12 14:47 mysqlclient.crt

In the mysql client on the jail itself I'm getting this:
Code: 
MariaDB [(none)]> show global variables like '%ssl%';
+---------------------+-----------------------------------------------------------+

| Variable_name       | Value                                                     |
+---------------------+-----------------------------------------------------------+
| have_openssl        | YES                                                       |
| have_ssl            | DISABLED                                                  |
| ssl_ca              | /usr/app/mariadb/ssl/ca.crt                   |
| ssl_capath          |                                                           |
| ssl_cert            | /usr/app/mariadb/ssl/mysql.crt       |
| ssl_cipher          | ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-SHA            |
| ssl_crl             |                                                           |
| ssl_crlpath         |                                                           |
| ssl_key             | /usr/app/mariadb/ssl/mysql-pkcs1.key |
| version_ssl_library | OpenSSL 1.0.2j-freebsd  26 Sep 2016                       |
+---------------------+-----------------------------------------------------------+

Also, I created a ZFS snapshot after the install of mariadb just to make sure. After getting the SSL connectivity errors I reverted to that snapshot... but the problem persists... Very odd.

Is there anything that I may be missing? From where I'm standing this should not result in a SSL_CTX_set_default_verify_paths error... Can it be that the certificates need to be placed in a special directory?

Any help is greatly appreciated.
 
Well, you must have changed something between those moments. If you take a look at the log you'll see it mention "permission denied" clearly indicating that something couldn't be accessed. Making me believe that something obviously changed between you turning this whole thing on and restarting the machine.

I also think that using /usr/app/mariadb/ssl/ is a rather odd location to put those keys. What made you come up with that? More logical would be something like /usr/local/etc/ or even /usr/local/var/app/mariadb for all that matter.

Even so: are you sure about filepermissions? Maybe an in-between update/update changed things which are now showing themselves.
 
Hmm... I can't think of anything... No one touched the jail. Also, I went back to the moment I snapshot the working configuration and - nothing. The problem persists...

Regarding permissions, the owner is mysql and the permissions of the whole directory are 755. Is there another layer of security on the filesystem in FreeBSD? In linux there are selinux and apparmor... I can't possibly think of anything else, at least with this log. Can I tell mariadb to log more?
 
For the sake of completeness: I deinstalled and installed databases/mariadb102-server to no avail. The OS is FreeBSD 11.1-STABLE.
 
STABLE is a developer snapshot, see this link. Not as bleeding edge as CURRENT but even so, it's not meant for production because of the risk for inconsistencies and bugs which can always sneak in. So honestly I wouldn't even be surprised if your problems resulted from a bug of some sort. Not saying that's the case, but it's also not something easily ruled out, because of the instable nature of the release.

MYXOMOP said:
Regarding permissions, the owner is mysql and the permissions of the whole directory are 755. Is there another layer of security on the filesystem in FreeBSD?
Click to expand...
Not just the filesystem, what to think about the jail? That in itself is limited in its options by default as well. But sure, you also have permissions / properties which you can set on a per-ZFS filesystem (see zfs get all) and also extended permissions (see chflags(1)).

Note that reinstalling is generally a waste of time. This isn't Windows you know ;) Thing is: a package only consists of binaries and example configuration files. Reinstalling doesn't really do much other than replace the binaries. Yet that won't have much consequences, especially if your problem seems to be caused by a permissive problem in the first place.

As to making MariaDB log more... I only have hands on experience with MySQL but it should be easy enough to change the config file and have it log to, for example, syslog while raising the verbosity.
 
Few things, your keys should not be world readable. The whole point of a key is that it's private, making them world readable defeats this. And while the certificate files themselves look readable, their (parent) directories may not be. So this could fail because MySQL is not being able to traverse the (parent) directories.
 
You must log in or register to reply here.
Back
Top