Accidentally broken libssl

Hi.
I decided to upgrade redmine from ports, did everything as usual ( portsnap fetch, portsnap update first, then make install).
Installation were completed without errors.
After that I discovered apachectl refuses to run with following error:
httpd: Syntax error on line 145 of /usr/local/etc/apache24/httpd.conf: Cannot load libexec/apache24/mod_ssl.so into server: Shared object "libssl.so.9" not found, required by "mod_ssl.so"

There appeared /usr/local/lib/libssl.so.11 and no libssl.so.9 in /usr/local/lib (as I understand, redmine upgraded libssl).
I tried to symlink so.9 to so.11 but in this case mod_ssl fails with
Undefined symbol "get_rfc3526_prime_8192" error. It looks like 9 & 11 are not 100% backward compatible.

I'm lucky that apache is still running, but if I try to restart it, it'll fail, there is no doubt.

Can anyone help?

FreeBSD 10.3-RELEASE-p11 FreeBSD 10.3-RELEASE-p11 #0: Mon Oct 24 18:49:24 UTC 2016 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64
 
What about all depended software? Like apache (specifically mod_ssl)? As I understand they are statically linked to libssl.so.9?

ldd /usr/local/libexec/apache24/mod_ssl.so

produces following:

Bash:
/usr/local/libexec/apache24/mod_ssl.so:
    libssl.so.9 => /usr/local/lib/libssl.so.9 (0x801638000)
    libcrypto.so.9 => /usr/local/lib/libcrypto.so.9 (0x801a00000
    libcrypt.so.5 => /lib/libcrypt.so.5 (0x801ee5000)
    libthr.so.3 => /lib/libthr.so.3 (0x802105000)
    libc.so.7 => /lib/libc.so.7 (0x800821000)

make.conf fragment:

DEFAULT_VERSIONS+=ssl=openssl
 
Or link libssl.so.9 and the others from the new directory to where Apache expects them. Hackey as can be, but bailing wire and duct tape holds most of the world together.
This leads to errors like Undefined symbol 'xxx' as I mentioned in my first post. I tried to link it to libssl.so.11 and libssl.so.7 which I managed to find across the system.
 
Good practice is running a supported FreeBSD version.
Yes, I totally agree, but unfortunately don't feel myself confident enough to perform upgrade on production server, although I previously upgraded it from 9 to 10. It wasn't easy for me at all. Thanks to all computer gods I managed to make it up and running.
More than three years of uptime since that moment)
 
Oh, you have got a production server that is obsolete. Read this:

For what reason do you 'feel not confident enough for upgrading'?
The purpose of this Forum is to assist on such tasks.
 
although I previously upgraded it from 9 to 10. It wasn't easy for me at all.
Obviously your resources were sufficient for that. Don't you know how to start or what exactly is your problem? Did you break your server or is it still functioning?
 
Did you break your server or is it still functioning?

The server is ok now, but it I try to restart for instance apache service it won't start again.
I'm trying to find out whether it's better to restore previous libssl version or do something different. I would appreciate if somebody give me a hint.
 
If I upgrade to FreeBSD 11 will it fix all all my problems?
Probably, but no guaranty.
First make a backup and be sure you can restore from it.
Upgrading from 10 to 11 was not critical here. From 11 to 12 may be more tricky. But my experience says nothing on what may expect you.
You also could setup a new box with 12.1-RELEASE and restore config files and data to it. Some additional work may be necessary, this way you do not need to touch your production server until you have a working new system.
 
OK, I upgraded FreeBSD to 11.3
As I expected apache started to fail on launch. The error it produces is

httpd: Syntax error on line 145 of /usr/local/etc/apache24/httpd.conf: Cannot load libexec/apache24/mod_ssl.so into server: Shared object "libssl.so.9" not found, required by "mod_ssl.so"

I did

# portsnap fetch update
# pkg-static install -f pkg
# portmaster -afG


and finally

/usr/sbin/freebsd-update install
 
Is it ok I upgraded from 10.3 right to 11.3? or I should first upgrade to all version between 10.3 and 11.3?
 
What about all depended software? Like apache (specifically mod_ssl)? As I understand they are statically linked to libssl.so.9?
They're not statically linked. They're just built with that version, and different major numbers on shared libs are not supposed to be compatible.
If I upgrade to FreeBSD 11 will it fix all all my problems?
It will fix no problems, it will just be a version that gets fixes for known exploits, and most people seem to like that.
Is it ok I upgraded from 10.3 right to 11.3? or I should first upgrade to all version between 10.3 and 11.3?
That should work.

As far as I see, your main shortcoming is that you have a prod machine, but no test machine - where you could run a backup, run an upgrade, then restore the backup, and whatever else until you see how it all works out.

The next fancy question is how you obtained that apache24 in the first place. Because mine is locally built, and on 11.3 it runs with libssl.so.8 (from base). Probably your apache24 is built to use ssl from ports (which afaik is not the default). So probably somebody configured that (probably because they needed a newer ssl version at that time). Which makes me wonder how you usually update your apache (on a prod server there is some reason to update it occasionally).

And anyway, after upgrading, it might be a good idea to run pkg check -d -n -a to see what else is amiss.
 
The next fancy question is how you obtained that apache24 in the first place. Because mine is locally built, and on 11.3 it runs with libssl.so.8 (from base). Probably your apache24 is built to use ssl from ports (which afaik is not the default). So probably somebody configured that (probably because they needed a newer ssl version at that time). Which makes me wonder how you usually update your apache (on a prod server there is some reason to update it occasionally).
i had apache24 installed via pkg. Just now I tried to build it from ports, but got compilation errors.
after upgrade to 11.3 ldd /usr/local/libexec/apache24/mod_ssl.so says mod_ssl linked to libssl.so.8 (on 10.3 it was linked to libssl.so.9)

ldd /usr/local/libexec/apache24/mod_ssl.so
/usr/local/libexec/apache24/mod_ssl.so:
libssl.so.8 => /usr/lib/libssl.so.8 (0x80123b000)
libcrypto.so.8 => /lib/libcrypto.so.8 (0x801600000)
libcrypt.so.5 => /lib/libcrypt.so.5 (0x801a76000)
libthr.so.3 => /lib/libthr.so.3 (0x801c95000)
libc.so.7 => /lib/libc.so.7 (0x800825000)




And anyway, after upgrading, it might be a good idea to run pkg check -d -n -a to see what else is amiss.

here is the output of pkg check -d -n -a :

pkg: Warning: Major OS version upgrade detected. Running "pkg bootstrap -f" recommended
Checking all packages: 100%
freetds-msdblib is missing a required shared library: libiconv.so.3
 
LMAO I gave you an option that most likely would have fixed your problem, and if it did or didn't, it posed essentially no risk. You didn't listen to advice before.

What did I do wrong? I followed your recommendations.
 
What finally helped me to make my server up and running again.
1. I deinstalled openssl installed previously with pkg util ( pkg delete openssl)
2. Just in case I deinstalled openssl port and built it again
cd /usr/ports/security/openssl/
make deinstall clean
make install clean


3. Reinstalled Apache Portability Library from ports (without this step I had no success)
cd /usr/ports/devel/apr1
make deinstall clean
make install clean


4. Reinstalled apache from ports
cd /usr/ports/www/apache24/
make deinstall clean
make install clean


This fixed all static linking issues. Thanks to all digital gods there is no more symlink workarounds in my system. This gives me a hope next OS upgrade won't be so hard.
 
This fixed all static linking issues.
Please learn the difference between static and dynamic linking. There is no static linking involved here.


 
What finally helped me to make my server up and running again.
1. I deinstalled openssl installed previously with pkg util ( pkg delete openssl)
2. Just in case I deinstalled openssl port and built it again
cd /usr/ports/security/openssl/
make deinstall clean
make install clean


3. Reinstalled Apache Portability Library from ports (without this step I had no success)
cd /usr/ports/devel/apr1
make deinstall clean
make install clean


4. Reinstalled apache from ports
cd /usr/ports/www/apache24/
make deinstall clean
make install clean


This fixed all static linking issues. Thanks to all digital gods there is no more symlink workarounds in my system. This gives me a hope next OS upgrade won't be so hard.

You mixed binaries which were not built against each other and this created a binary mismatch.
Your Apache mod_ssl.so was installed via a binary that was built by FreeBSD and your libssl.so was compiled by you via the port. In this case mod_ssl.so calls into libssl.so and expects a certain binary interface (ABI) but libssl.so has an incompatible one.
Both the dependency and the dependent binary need to be built against each other (either install both via pkg, or build both of them from ports).
Of course, the same applies for everything that depends on libssl.so - recursively! This might be quite a long list.

You could also try to build the libssl port and configure it identically to the FreeBSD build server (what you download via pkg). Unfortunatelly I have not done this and I can't tell you exactly how to do it.
 
Unless you changed /etc/make.conf I don't think you've done what you think you've done.

You've built Apache and the APR against the system's OpenSSL, and you've separately installed the port OpenSSL.

But easier to use the system OpenSSL (and it should make future upgrades easier) and if you're back in business then that's good.
 
Back
Top