Solved FreeBSD 9.3, OpenSSL segfault in BN_mod_exp_mont_consttime

Hi.

I do not think that what I am seeing is related to this other thread (https://forums.freebsd.org/threads/55430/) but I'm also getting segmentation faults (but not in OpenSSH).

I've been trying to create a self-signed certificate, so (for example) I follow the instructions from here: https://www.freebsd.org/doc/handbook/openssl.html

openssl genrsa -rand -genkey -out cert.key 2048
openssl req -new -x509 -days 365 -key cert.key -out cert.crt -sha256


So I try this (on three different 9.3 machines):

Code:
webarts@fbsd:/tmp % openssl genrsa -rand -genkey -out cert.key 2048
0 semi-random bytes loaded
Generating RSA private key, 2048 bit long modulus
......+++
........+++
e is 65537 (0x10001)
webarts@fbsd:/tmp % openssl req -new -x509 -days 365 -key cert.key -out cert.crt  -sha256
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:NZ
State or Province Name (full name) [Some-State]:
Locality Name (eg, city) []:
Organization Name (eg, company) [Internet Widgits Pty Ltd]:
Organizational Unit Name (eg, section) []:
Common Name (e.g. server FQDN or YOUR name) []:fbsd.webarts.co.nz
Email Address []:
Segmentation fault (core dumped)
This is 9.3, I've used freebsd-update fetch and install, and rebooted. I've tried different variations of the openssl command, tried filling in all the fields, and tried on other 9.3 machines (with OpenSSL from ports installed; it works.)

Code:
webarts@fbsd:/tmp % which openssl
/usr/bin/openssl

webarts@fbsd:/tmp % /usr/bin/openssl version
OpenSSL 0.9.8zh-freebsd 3 Dec 2015

webarts@fbsd:/tmp % uname -a
FreeBSD fbsd.webarts.co.nz 9.3-RELEASE-p33 FreeBSD 9.3-RELEASE-p33 #0: Wed Jan 13 17:55:39 UTC 2016  root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC  amd64

webarts@fbsd:/tmp % pkg info | grep pen
libltdl-2.4.6  System independent dlopen wrapper
php56-openssl-5.6.19  The openssl shared extension for php

webarts@fbsd:/tmp % ldd /usr/bin/openssl
/usr/bin/openssl:
  libssl.so.6 => /usr/lib/libssl.so.6 (0x800879000)
  libcrypto.so.6 => /lib/libcrypto.so.6 (0x800ad0000)
  libc.so.7 => /lib/libc.so.7 (0x800e78000)

webarts@fbsd:/tmp % gdb openssl
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...(no debugging symbols found)...
(gdb) run req -new -x509 -days 365 -key cert.key -out cert.crt -sha256
Starting program: /usr/bin/openssl req -new -x509 -days 365 -key cert.key -out cert.crt -sha256
(no debugging symbols found)...(no debugging symbols found)...(no debugging symbols found)...(no debugging symbols found)...You are about to be asked to enter information that will be incorporated into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:NZ
State or Province Name (full name) [Some-State]:
Locality Name (eg, city) []:
Organization Name (eg, company) [Internet Widgits Pty Ltd]:
Organizational Unit Name (eg, section) []:
Common Name (e.g. server FQDN or YOUR name) []:fbsd.webarts.co.nz
Email Address []:

Program received signal SIGSEGV, Segmentation fault.

0x0000000800bf3438 in BN_mod_exp_mont_consttime () from /lib/libcrypto.so.6
(gdb) bt
#0  0x0000000800bf3438 in BN_mod_exp_mont_consttime () from /lib/libcrypto.so.6
#1  0x0000000800bd135f in RSA_PKCS1_SSLeay () from /lib/libcrypto.so.6
#2  0x0000000800bd22fd in RSA_PKCS1_SSLeay () from /lib/libcrypto.so.6
#3  0x0000000800bac8d9 in RSA_sign () from /lib/libcrypto.so.6
#4  0x0000000800bab11b in EVP_SignFinal () from /lib/libcrypto.so.6
#5  0x0000000800ba90dd in ASN1_item_sign () from /lib/libcrypto.so.6
#6  0x0000000800ba5d61 in X509_sign () from /lib/libcrypto.so.6
#7  0x0000000000432340 in ?? ()
#8  0x000000000042c05b in ?? ()
#9  0x000000000042c70a in ?? ()
#10 0x0000000000413161 in ?? ()
#11 0x000000080067b000 in ?? ()
#12 0x0000000000000000 in ?? ()
The ports version is not installed (see pkg info above).

I thought about having a look in the source code, but when I run subversion I get a segmentation fault in the same place.

I run Apache 2.2, and I get segmentation faults there (haven't checked if exactly the same place, though):

Code:
root@fbsd:/home/webarts # tail /var/log/httpd-error.log
[Tue Mar 08 16:54:30 2016] [notice] child pid 13051 exit signal Segmentation fault (11)
[Tue Mar 08 16:54:30 2016] [notice] child pid 13050 exit signal Segmentation fault (11)
[Tue Mar 08 16:54:31 2016] [notice] child pid 13055 exit signal Segmentation fault (11)
Apache is using the base OpenSSL.

Code:
root@fbsd:/tmp # cat /etc/make.conf
DEFAULT_VERSIONS+=apache=2.2
I think I need to file a bug report, but want to check (a) if anyone else is seeing this (I have searched, but not found anything obvious) and (b) if I've done something wrong. I will be moving to 10.2/10.3 and the port version of OpenSSL over the course of the year, but right now I need to keep 9.3 production servers happy, so I'd like to find a fix for this that (if possible) doesn't involve upgrading to 10.x or rebuilding against the port version.

EDIT: fetch also fails, in the same place. (Not sure if this helps, but a simpler example of it going wrong.)

Code:
webarts@fbsd:/tmp % fetch https://security.FreeBSD.org/patches/SA-16:11/openssl-9.3.patch
Segmentation fault (core dumped)
webarts@fbsd:/tmp % gdb fetch
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...(no debugging symbols found)...
(gdb) run  https://security.FreeBSD.org/patches/SA-16:11/openssl-9.3.patch
Starting program: /usr/bin/fetch https://security.FreeBSD.org/patches/SA-16:11/openssl-9.3.patch
(no debugging symbols found)...(no debugging symbols found)...(no debugging symbols found)...(no debugging symbols found)...(no debugging symbols found)...
Program received signal SIGSEGV, Segmentation fault.
0x0000000800dac438 in BN_mod_exp_mont_consttime () from /lib/libcrypto.so.6
(gdb) bt
#0  0x0000000800dac438 in BN_mod_exp_mont_consttime () from /lib/libcrypto.so.6
#1  0x0000000800d8dab8 in DH_OpenSSL () from /lib/libcrypto.so.6
#2  0x0000000800a631d1 in ssl3_send_client_key_exchange () from /usr/lib/libssl.so.6
#3  0x0000000800a66448 in ssl3_connect () from /usr/lib/libssl.so.6
#4  0x000000080082b3cb in fetch_ssl () from /usr/lib/libfetch.so.6
#5  0x0000000800826816 in http_request () from /usr/lib/libfetch.so.6
#6  0x000000000040272f in ?? ()
#7  0x0000000000404323 in ?? ()
#8  0x0000000000401da1 in ?? ()
#9  0x0000000800622000 in ?? ()
#10 0x0000000000000000 in ?? ()
On another 9.3 box:

Code:
> fetch https://security.FreeBSD.org/patches/SA-16:11/openssl-9.3.patch
Segmentation fault (core dumped)
Both have the same recently-updated-by-freebsd-update libcrypto:

Code:
> ls -l /lib/libcrypto.so.6
-r--r--r--  1 root  wheel  1749632 Mar  8 10:45 /lib/libcrypto.so.6
Thanks.
 
Thank you.

I did search on the internet, and I did look in the bug tracker - but missed it somehow (must have been using the wrong search terms?) Anyway, known issue and hopefully it will be fixed soon.

Thank you for the prompt & helpful answers.
 
I managed to get rid of the segmentation fault in Apache by recompiling it with the following addition to /etc/make.conf:

Code:
WITH_OPENSSL_PORT=yes

You would expect that the base openssl should work fine, but in my case it apparantly did not. Maybe this can help you.
 
I managed to get rid of the segmentation fault in Apache by recompiling it with the following addition to /etc/make.conf:

Code:
WITH_OPENSSL_PORT=yes

You would expect that the base openssl should work fine, but in my case it apparantly did not. Maybe this can help you.

There is a known problem when using ports that directly link with one version of OpenSSL (ports usually) and indirectly link with another (the base system OpenSSL). That causes a symbol conflict on the dynamic linking level and the function calls end up in wrong place causing a crash because the other set of functions is not ABI compatible with the intended set.
 
Back
Top