Accidentally broken libssl

heximal

Member

Reaction score: 2
Messages: 60

Hi.
I decided to upgrade redmine from ports, did everything as usual ( portsnap fetch, portsnap update first, then make install).
Installation were completed without errors.
After that I discovered apachectl refuses to run with following error:
httpd: Syntax error on line 145 of /usr/local/etc/apache24/httpd.conf: Cannot load libexec/apache24/mod_ssl.so into server: Shared object "libssl.so.9" not found, required by "mod_ssl.so"

There appeared /usr/local/lib/libssl.so.11 and no libssl.so.9 in /usr/local/lib (as I understand, redmine upgraded libssl).
I tried to symlink so.9 to so.11 but in this case mod_ssl fails with
Undefined symbol "get_rfc3526_prime_8192" error. It looks like 9 & 11 are not 100% backward compatible.

I'm lucky that apache is still running, but if I try to restart it, it'll fail, there is no doubt.

Can anyone help?

FreeBSD 10.3-RELEASE-p11 FreeBSD 10.3-RELEASE-p11 #0: Mon Oct 24 18:49:24 UTC 2016 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64
 
OP
heximal

heximal

Member

Reaction score: 2
Messages: 60

What about all depended software? Like apache (specifically mod_ssl)? As I understand they are statically linked to libssl.so.9?

ldd /usr/local/libexec/apache24/mod_ssl.so

produces following:

Bash:
/usr/local/libexec/apache24/mod_ssl.so:
    libssl.so.9 => /usr/local/lib/libssl.so.9 (0x801638000)
    libcrypto.so.9 => /usr/local/lib/libcrypto.so.9 (0x801a00000
    libcrypt.so.5 => /lib/libcrypt.so.5 (0x801ee5000)
    libthr.so.3 => /lib/libthr.so.3 (0x802105000)
    libc.so.7 => /lib/libc.so.7 (0x800821000)
make.conf fragment:

DEFAULT_VERSIONS+=ssl=openssl
 
OP
heximal

heximal

Member

Reaction score: 2
Messages: 60

Or link libssl.so.9 and the others from the new directory to where Apache expects them. Hackey as can be, but bailing wire and duct tape holds most of the world together.
This leads to errors like Undefined symbol 'xxx' as I mentioned in my first post. I tried to link it to libssl.so.11 and libssl.so.7 which I managed to find across the system.
 
OP
heximal

heximal

Member

Reaction score: 2
Messages: 60

Good practice is running a supported FreeBSD version.
Yes, I totally agree, but unfortunately don't feel myself confident enough to perform upgrade on production server, although I previously upgraded it from 9 to 10. It wasn't easy for me at all. Thanks to all computer gods I managed to make it up and running.
More than three years of uptime since that moment)
 

getopt

Aspiring Daemon

Reaction score: 451
Messages: 735

Oh, you have got a production server that is obsolete. Read this:

For what reason do you 'feel not confident enough for upgrading'?
The purpose of this Forum is to assist on such tasks.
 

getopt

Aspiring Daemon

Reaction score: 451
Messages: 735

although I previously upgraded it from 9 to 10. It wasn't easy for me at all.
Obviously your resources were sufficient for that. Don't you know how to start or what exactly is your problem? Did you break your server or is it still functioning?
 
OP
heximal

heximal

Member

Reaction score: 2
Messages: 60

Did you break your server or is it still functioning?
The server is ok now, but it I try to restart for instance apache service it won't start again.
I'm trying to find out whether it's better to restore previous libssl version or do something different. I would appreciate if somebody give me a hint.
 
OP
heximal

heximal

Member

Reaction score: 2
Messages: 60

If I upgrade to FreeBSD 11 will it fix all all my problems?
 

getopt

Aspiring Daemon

Reaction score: 451
Messages: 735

If I upgrade to FreeBSD 11 will it fix all all my problems?
Probably, but no guaranty.
First make a backup and be sure you can restore from it.
Upgrading from 10 to 11 was not critical here. From 11 to 12 may be more tricky. But my experience says nothing on what may expect you.
You also could setup a new box with 12.1-RELEASE and restore config files and data to it. Some additional work may be necessary, this way you do not need to touch your production server until you have a working new system.
 

pyret

Active Member

Reaction score: 94
Messages: 192

If I upgrade to FreeBSD 11 will it fix all all my problems?


And upgrading would be less of a risk than just compiling a previous version of OpenSSL that has libssl.so.9? Something which would take 30 minutes and poses no risk. It either fixes your problem (likely) or you're in the same state you are now. But, whatever. Knock yourself out.
 
OP
heximal

heximal

Member

Reaction score: 2
Messages: 60

OK, I upgraded FreeBSD to 11.3
As I expected apache started to fail on launch. The error it produces is

httpd: Syntax error on line 145 of /usr/local/etc/apache24/httpd.conf: Cannot load libexec/apache24/mod_ssl.so into server: Shared object "libssl.so.9" not found, required by "mod_ssl.so"

I did

# portsnap fetch update
# pkg-static install -f pkg
# portmaster -afG


and finally

/usr/sbin/freebsd-update install
 
OP
heximal

heximal

Member

Reaction score: 2
Messages: 60

Is it ok I upgraded from 10.3 right to 11.3? or I should first upgrade to all version between 10.3 and 11.3?
 

pyret

Active Member

Reaction score: 94
Messages: 192

LMAO I gave you an option that most likely would have fixed your problem, and if it did or didn't, it posed essentially no risk. You didn't listen to advice before (and I've been doing this for 25 years for Fortune 400 companies, US DoD, and other companies in ecommerce, insurance, securities, transportation, among other industries), so I don't feel any desire to help. Good luck.
 

PMc

Aspiring Daemon

Reaction score: 291
Messages: 781

What about all depended software? Like apache (specifically mod_ssl)? As I understand they are statically linked to libssl.so.9?
They're not statically linked. They're just built with that version, and different major numbers on shared libs are not supposed to be compatible.
If I upgrade to FreeBSD 11 will it fix all all my problems?
It will fix no problems, it will just be a version that gets fixes for known exploits, and most people seem to like that.
Is it ok I upgraded from 10.3 right to 11.3? or I should first upgrade to all version between 10.3 and 11.3?
That should work.

As far as I see, your main shortcoming is that you have a prod machine, but no test machine - where you could run a backup, run an upgrade, then restore the backup, and whatever else until you see how it all works out.

The next fancy question is how you obtained that apache24 in the first place. Because mine is locally built, and on 11.3 it runs with libssl.so.8 (from base). Probably your apache24 is built to use ssl from ports (which afaik is not the default). So probably somebody configured that (probably because they needed a newer ssl version at that time). Which makes me wonder how you usually update your apache (on a prod server there is some reason to update it occasionally).

And anyway, after upgrading, it might be a good idea to run pkg check -d -n -a to see what else is amiss.
 
OP
heximal

heximal

Member

Reaction score: 2
Messages: 60

The next fancy question is how you obtained that apache24 in the first place. Because mine is locally built, and on 11.3 it runs with libssl.so.8 (from base). Probably your apache24 is built to use ssl from ports (which afaik is not the default). So probably somebody configured that (probably because they needed a newer ssl version at that time). Which makes me wonder how you usually update your apache (on a prod server there is some reason to update it occasionally).
i had apache24 installed via pkg. Just now I tried to build it from ports, but got compilation errors.
after upgrade to 11.3 ldd /usr/local/libexec/apache24/mod_ssl.so says mod_ssl linked to libssl.so.8 (on 10.3 it was linked to libssl.so.9)

ldd /usr/local/libexec/apache24/mod_ssl.so
/usr/local/libexec/apache24/mod_ssl.so:
libssl.so.8 => /usr/lib/libssl.so.8 (0x80123b000)
libcrypto.so.8 => /lib/libcrypto.so.8 (0x801600000)
libcrypt.so.5 => /lib/libcrypt.so.5 (0x801a76000)
libthr.so.3 => /lib/libthr.so.3 (0x801c95000)
libc.so.7 => /lib/libc.so.7 (0x800825000)




And anyway, after upgrading, it might be a good idea to run pkg check -d -n -a to see what else is amiss.
here is the output of pkg check -d -n -a :

pkg: Warning: Major OS version upgrade detected. Running "pkg bootstrap -f" recommended
Checking all packages: 100%
freetds-msdblib is missing a required shared library: libiconv.so.3
 
OP
heximal

heximal

Member

Reaction score: 2
Messages: 60

LMAO I gave you an option that most likely would have fixed your problem, and if it did or didn't, it posed essentially no risk. You didn't listen to advice before.
What did I do wrong? I followed your recommendations.
 

pyret

Active Member

Reaction score: 94
Messages: 192

You did not. I told you to get the source for a version of OpenSSL that has libssl.so.9 and build it in a directory like /usr/local/openssl-1.0.2p and then link from there to where it expects to find libssl.so.9, such as,
Code:
ln -s /usr/local/openssl-1.0.2p/lib/libssl.so.9 /usr/lib/libssl.so.9
.

I suggested you do that before you upgraded because it poses no risk even if it doesn't work. But I would have expected it to work because the version it wants would now exist. Now you've not only upgraded but tried to build from ports.

And don't take my gruffness to heart. You just caught me after last week dealing with someone who asked me something and I told him to contact IA, then next thing I know a coworker IM'd me that the guy was asking him the same thing. Apparently he didn't like my answer so thought he would get a different one. Only he got the same one. Then he was back the next day doing the same @#%& again, on something we had gone through N $#@^$#(& times before.
 

richardtoohey2

Member

Reaction score: 34
Messages: 99

If you are now building from ports - do you have to use LibreSSL?

If you build with OpenSSL you might get things back up and running.
 
OP
heximal

heximal

Member

Reaction score: 2
Messages: 60

What finally helped me to make my server up and running again.
1. I deinstalled openssl installed previously with pkg util ( pkg delete openssl)
2. Just in case I deinstalled openssl port and built it again
cd /usr/ports/security/openssl/
make deinstall clean
make install clean


3. Reinstalled Apache Portability Library from ports (without this step I had no success)
cd /usr/ports/devel/apr1
make deinstall clean
make install clean


4. Reinstalled apache from ports
cd /usr/ports/www/apache24/
make deinstall clean
make install clean


This fixed all static linking issues. Thanks to all digital gods there is no more symlink workarounds in my system. This gives me a hope next OS upgrade won't be so hard.
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 8,911
Messages: 33,305

This fixed all static linking issues.
Please learn the difference between static and dynamic linking. There is no static linking involved here.


 

roccobaroccoSC

Aspiring Daemon

Reaction score: 163
Messages: 648

What finally helped me to make my server up and running again.
1. I deinstalled openssl installed previously with pkg util ( pkg delete openssl)
2. Just in case I deinstalled openssl port and built it again
cd /usr/ports/security/openssl/
make deinstall clean
make install clean


3. Reinstalled Apache Portability Library from ports (without this step I had no success)
cd /usr/ports/devel/apr1
make deinstall clean
make install clean


4. Reinstalled apache from ports
cd /usr/ports/www/apache24/
make deinstall clean
make install clean


This fixed all static linking issues. Thanks to all digital gods there is no more symlink workarounds in my system. This gives me a hope next OS upgrade won't be so hard.
You mixed binaries which were not built against each other and this created a binary mismatch.
Your Apache mod_ssl.so was installed via a binary that was built by FreeBSD and your libssl.so was compiled by you via the port. In this case mod_ssl.so calls into libssl.so and expects a certain binary interface (ABI) but libssl.so has an incompatible one.
Both the dependency and the dependent binary need to be built against each other (either install both via pkg, or build both of them from ports).
Of course, the same applies for everything that depends on libssl.so - recursively! This might be quite a long list.

You could also try to build the libssl port and configure it identically to the FreeBSD build server (what you download via pkg). Unfortunatelly I have not done this and I can't tell you exactly how to do it.
 

richardtoohey2

Member

Reaction score: 34
Messages: 99

Unless you changed /etc/make.conf I don't think you've done what you think you've done.

You've built Apache and the APR against the system's OpenSSL, and you've separately installed the port OpenSSL.

But easier to use the system OpenSSL (and it should make future upgrades easier) and if you're back in business then that's good.
 
Top