Apache port won't update; package update includes obsolete OpenSSL

P

phantomflash

I've tried to update www/apache22 and its companion security/openssl via both ports and packages. Nothing works completely. Omitting a lot of detail about false starts and different things I've tried, in the recent past I've used ports, mainly ports-mgmt/portupgrade, in combination with the base system update process. It worked pretty well until this year.

First, using ports, www/apache24 wouldn't update. Eventually I saw a message that it needed a specific version of Perl, and apparently I have a slightly different version. I don't know why portupgrade won't update Perl to the correct version, because I told it to update everything.

While researching this, I came across a description of packages. I realized a binary install would likely be far faster and more convenient, and it also wouldn't have any need of help from Perl to install something unrelated like Apache. So I used pkg, and it did update Apache from version 2.2.29 to 2.2.31, which seems to be the current version. And since it has its own version-coordinated mod_ssl, containing openssl, things seemed OK.

HOWEVER ... I later realized that with Apache had come with a very ancient version of openssl. With Apache 2.2.29 we formerly had an openssl version somewhere north of 1.0.1g -- I don't remember for sure, but it might have been 1.0.2c. Anyway, with the new Apache 2.2.31, it's saying it has mod_ssl 2.2.31 which (it says) contains openssl version 0.9.8zh-freebsd.

We quit using the version 0.9.8 branch at least two years ago. How can a version of Apache that's supposed to be the very latest, be including an obsolete openssl? I tested our web site using an external SSL testing site a few weeks ago we had a grade of A; now it's down to C because we have vulnerabilities. We need to update to the newest openssl to fix these vulnerabilities and meet certification requirements.

Earlier today I tried using pkg and specifying to update openssl. It dutifully reported that it updated it from version 1.0.2_11 to version 1.0.2_12. But this doesn't affect Apache's version. It obviously updated the "system" copy, as opposed to the separate copy bundled with, and used by, Apache.

So why is this happening and what's the best way to get the proper, latest version of openssl for Apache? Sources, binaries, or whatever.
 
Last edited by a moderator:
phantomflash said:
We quit using the version 0.9.8 branch at least two years ago. How can a version of Apache that's supposed to be the very latest, be including an obsolete openssl?
Click to expand...
It's not. It's linked to the OpenSSL in the base OS. Which is version 0.9.8. This version isn't obsolete, it's still supported.
 
FreeBSD security team does backporting of security fixes to the OpenSSL included in FreeBSD 9 that is kept at version 0.9.8 for API/ABI compatibility reasons. If you want to have Apache to use the port version of OpenSSL you can add this to your make.conf(5):

Code: 
WITH_OPENSSL_PORT=yes

You'll have to recompile all of your ports that use OpenSSL after that because it's not recommended to mix the two versions between different installed ports.
 
SirDice said:
It's not. It's linked to the OpenSSL in the base OS. Which is version 0.9.8. This version isn't obsolete, it's still supported.
Click to expand...
I guess it depends on your definition of "supported." Maybe it's supported by the FreeBSD team, but from https://www.openssl.org/news/secadv/20160503.txt we find:

"Support for versions 0.9.8 and 1.0.0 ended on 31st December 2015. Those versions are no longer receiving security updates."

There have been 3 releases of 1.0.1 and 1.0.2 since then, patching approximately 16 security issues. So the 0.9.8 branch is now definitely behind and out of date.

Since the only reason for openssl is security, and 0.9.8 is no longer security-current, and won't pass current certifications (not to mention it downgraded our server from grade A to C), then by most definitions it is obsolete.
 
The document you linked does not apply to the FreeBSD version of 0.9.8 because of the local backported security patches. You would think that someone who have been given the power to rate your systems would be in the "know" about these things but apparently not.
 
If they had ported the more recent patches, they would have bumped the version number, but it's still 0.9.8zh. zh is the December 2015 version suffix. But that's not really the point. The big thing is that the external testing found vulnerabilities. It doesn't care what server you have, or version number it is, but just how vulnerable the server is. And there are more vulnerabilities now, than before I did the binary Apache update.

For some years now, normally openssl has come bundled with Apache in two ways: First, compiled into the Apache program itself; and second, inside mod_ssl which can be optionally loaded by Apache at run time. It keeps openssl up to date, within itself, in that manner. It doesn't seem to make much sense for FreeBSD to change that and impose a different version of openssl and link Apache with that, instead of the version that comes from Apache itself. Linking it into Apache and/or mod_ssl wouldn't cause any incompatibilities with any other program on the system. (Because you can also install Apache from a port, and it's not linked with 0.9.8, and it doesn't cause any incompatibilities either.)

So we have 2 copies of openssl bundled with Apache, plus a separate copy of openssl in its own port. Now you tell me there's a 4th copy, included in the base system? This gets stranger and stranger. (I'm not even counting extra copies of openssl that are included with certain other applications that are used only by those apps.)

But that's kind of a side issue. Doesn't make sense, but if that's the way it is, that's the way it is. The bottom line is that it's clear that the binary update of Apache is not usable, by me anyway. So I'll have to go back and find a way to get the port to work.

But in the meantime, I have a simple question. Where is this 0.9.8 version copy supposed to be? Here's all the system finds:
Code: 
# whereis openssl
openssl: /usr/bin/openssl /usr/local/man/man1/openssl.1.gz /usr/ports/security/openssl
And the version in /usr/bin is the one I updated myself on April 25 of this year (well before I tried using pkg). Unfortunately I don't know of a way to easily tell what version it is.
 
Hmm, wait, are you saying that 0.9.8 is compiled and linked inside every program that uses it, and doesn't appear as a separate file anywhere? I guess that would explain why it doesn't show up separately ..... ?
 
kpa said:
If you want to have Apache to use the port version of OpenSSL you can add this to your make.conf(5):
Code: 
WITH_OPENSSL_PORT=yes
You'll have to recompile all of your ports that use OpenSSL after that because it's not recommended to mix the two versions between different installed ports.
Click to expand...

I actually did that already, over a year ago, right after I updated from 8.4 to 9.3. I don't remember the details of why I had to do it, but after the update something with Apache didn't come out right, and I found mention of that option and tried it, and then things worked.

But, as you indicated, this applies to ports compiled from source. It would not apply to the binary Apache installed by pkg, because that's pre-linked to a certain version of openssl.
 
If I were in your shoes now I would stop using the official packages for FreeBSD 9 because those (as you already guessed) are linked (dynamic linking, not static) against the 0.9.8 version of OpenSSL from the base system and that can not be changed with any method. Instead compile everything yourself using a package builder such as ports-mgmt/synth or ports-mgmt/poudriere with WITH_OPENSSL_PORT=yes set in the builder's make.conf(5). This would give you very up to date OpenSSL (or even LibreSSL if you desire) that is very easy to update when there are security updates.
 
kpa said:
The document you linked does not apply to the FreeBSD version of 0.9.8 because of the local backported security patches. You would think that someone who have been given the power to rate your systems would be in the "know" about these things but apparently not.
Click to expand...
If you want more proof, here's what FreeBSD itself says:
Code: 
# dir /usr/bin/ssh
-r-xr-xr-x 2 root wheel 184672 Apr 25 13:31 /usr/bin/ssh
# ssh -V
OpenSSH_6.6.1p1, OpenSSL 0.9.8zh-freebsd 3 Dec 2015
Note the date on the the 0.9.8zh-freebsd version of OpenSSL. Before any of the over a dozen recent security patches.

No need to be disparaging of the people doing our internal scans. They don't know beans about the details of BSD support; they're just using 3rd-party scanning software -- and we all know software is infallible, right?
 
kpa said:
If I were in your shoes now I would stop using the official packages for FreeBSD 9 because those (as you already guessed) are linked (dynamic linking, not static) against the 0.9.8 version of OpenSSL from the base system and that can not be changed with any method. Instead compile everything yourself using a package builder such as ports-mgmt/synth or ports-mgmt/poudriere with WITH_OPENSSL_PORT=yes set in the builder's make.conf(5). This would give you very up to date OpenSSL (or even LibreSSL if you desire) that is very easy to update when there are security updates.
Click to expand...
That's exactly what I just said, while ago, that I was going to do -- go back to the ports, which is what used to work, until someone (incorrectly) told me Apache had been moved from the ports to the base system. And until, more recently, the Apache port wouldn't update, so I tried the binary package instead of the port (which updated Apache fine, but reverted to the obsolete OpenSSL).

And I've already said I've had WITH_OPENSSL_PORT=yes set for over a year now, ever since updating from 8.4 to 9.3.
 
Yes I know what the various tools such as ssh(1) report. That's a known "pitfall" of the security team's insistence of doing things their way and not updating the dates or the version strings. Believe me, it has been asked on mailing lists and reported as a bug via PRs more than once. If I had any influence on the matter I would certainly do it differently.
 
kpa said:
... the official packages for FreeBSD 9 .. (as you already guessed) are linked (dynamic linking, not static) against the 0.9.8 version of OpenSSL from the base system and that can not be changed with any method.
Click to expand...
When you say "dynamic linking," are you saying there's an openssl file somewhere on the system that's loaded at run time, analogous to a DLL in Windows? If so, where is it? I've already asked, but nobody has been able to tell me so far.
 
phantomflash said:
When you say "dynamic linking," are you saying there's an openssl file somewhere on the system that's loaded at run time, analogous to a DLL in Windows? If so, where is it? I've already asked, but nobody has been able to tell me so far.
Click to expand...

It's split into multiple modules but the main ones are (the versions are the ones in FreeBSD 10, FreeBSD 9 might have different numbers) /usr/lib/libssl.so.7 and (oddly enough in another directory) /lib/libcrypto.so.7. The port version installs all of the shared libraries under /usr/local/lib.
 
OK, thanks. I wasn't thinking about the file names being different. Anyway, those library files (the first two you cited) on our system are dated April 25. So if openssl had been updated in the FreeBSD base system since December, it would be in those files. But we weren't getting the updated version, or else our server grade wouldn't have dropped from A to C. Another proof that these libraries contain the December version of openssl 0.9.8, which is out of date and getting more so all the time.

Hopefully FreeBSD 10.x has one of the currently-supported openssl tracks (1.0.1 or 1.0.2). But they need to rethink this "lowest common denominator" strategy, because that means any binary copies of Apache on version 9.3 are vulnerable and will remain so. They should just let Apache, at least, be linked with its own version of openssl. I don't know the details of other programs, but it seems they could also get the latest and greatest, without messing up any other part of the system.

(But those are traditional object libraries, aren't they? Not like DLLs? So why would you call it dynamic linking?)
 
Hi phantomflash,

I'm not sure what you mean by "traditional object libraries". Programs can either be statically linked or dynamically linked. If they are statically linked, the libraries the program uses are copied into the executable itself at compile time. Programs that are dynamically linked load the libraries at runtime instead. In the UNIX world, files with the .so suffix (shared object) are usually libraries that are loaded at runtime by programs. The Windows equivalent to shared objects are DLLs.

It wouldn't be a very good idea to have an Apache version of openssl(1) because security/openssl already exists. Apache could be linked to the ports version instead if it turns out that the security updates were not backported.

You are updating the base system with freebsd-update(8), right? You probably are, but it never hurts to be sure.
 
The ELF shared libraries are functionally equivalent to Windows DLLs in every possible way, in fact Windows DLL technology was developed from the dynamic link libraries used in UNIX systems of the day. In case you're thinking of something like COM/COM+, take a look at technology known as CORBA, it is used on UNIX systems but not very widely.

What you are calling "lowest common denominator" strategy is actually what is called "ABI stability". What it means is that once a major version line of FreeBSD is created, let's say FreeBSD 9 you're using, the API/ABI provided by the base system shared libraries is set in stone and can not be changed because that would break the promise that a program that runs on FreeBSD 9.0 must run on (for example) FreeBSD 9.3 without recompilation. This ABI stability allows you to upgrade your system to next minor version (for example 9.2 -> 9.3) without having to recompile a single port after the upgrade and everything will just work.

At the time FreeBSD 9 was created from what was then called 9-CURRENT the newest OpenSSL was 0.9.8 something that was what became the de facto version of it in the base system in FreeBSD 9 and that will not change as long as FreeBSD 9 line exists.
 
atomicbeef said:
Programs can either be statically linked or dynamically linked. If they are statically linked, the libraries the program uses are copied into the executable itself at compile time. Programs that are dynamically linked load the libraries at runtime instead. In the UNIX world, files with the .so suffix (shared object) are usually libraries that are loaded at runtime by programs. The Windows equivalent to shared objects are DLLs.
Click to expand...
OK, that answers my question about terminology (static vs. dynamic linking). I wanted to make sure I was correctly understanding what you meant. Since the answer is "yes," now I can proceed to answer your question.
I'm not sure what you mean by "traditional object libraries".
Click to expand...
I mean object libraries for static linking. "Traditional" since in the (very) olden days, that's all we had.

Actually I wasn't sure *nix had dynamically-linked shared libraries (although it certainly makes complete sense that they were added along the way), except for the Apache loadable modules such as mod_ssl. Thanks for clearing that up for me. Since object files are .o, I somehow got the idea the .so files were static libraries. (Actually those are .a files. Sorry, it's been a while since I looked at such things....)
It wouldn't be a very good idea to have an Apache version of openssl(1) because security/openssl already exists. Apache could be linked to the ports version instead if it turns out that the security updates were not backported.
Click to expand...
Older versions of OpenSSL were so tightly bound to a specific version of Apache that the Apache version was actually part of the OpenSSL version number. And still today, each version of Apache has its own version-synchronized dynamically-loaded mod_ssl containing, ordinarily, the latest and greatest version of OpenSSL that existed at the time this version of Apache was released. Or Apache can be, and often is, statically linked with OpenSSL. So Apache is pretty much "paired" with that version of OpenSSL in practice. That's what I meant. Since it comes along for the ride with Apache, I was pretty shocked to see the very newest version of Apache (binary installed with pkg) having been linked with an 0.9.8 version of OpenSSL, especially since our older version of Apache (source installed with portupgrade) had a 1.0.x version.

I had my doubts about whether this would be workable, and sure enough, it wasn't. The binary is sure easy to install, but if it doesn't pass muster then it's disqualified from installations that need to be security-current (actually, I thought that was everyone). Hence my reverting to "use the source, Luke!"
You are updating the base system with freebsd-update(8), right?
Click to expand...
Right. Although some oddities happened when I updated from 8.4 to 9.3. Maybe the trip to 10.x, when I embark on it, will end up better (I can always hope....).
 
kpa said:
What you are calling "lowest common denominator" strategy is actually what is called "ABI stability". What it means is that once a major version line of FreeBSD is created, let's say FreeBSD 9 you're using, the API/ABI provided by the base system shared libraries is set in stone and can not be changed because that would break the promise that a program that runs on FreeBSD 9.0 must run on (for example) FreeBSD 9.3 without recompilation. This ABI stability allows you to upgrade your system to next minor version (for example 9.2 -> 9.3) without having to recompile a single port after the upgrade and everything will just work.

At the time FreeBSD 9 was created from what was then called 9-CURRENT the newest OpenSSL was 0.9.8 something that was what became the de facto version of it in the base system in FreeBSD 9 and that will not change as long as FreeBSD 9 line exists.
Click to expand...
I can see this is reasonable in most cases. It does rely on the decisions to (1) include OpenSSL in the library, and (2) link Apache against it. I meant that because of what could be considered these two programs' somewhat unique positions, I could visualize an environment where at least number (2) might be done differently, linking Apache statically with that version's "de facto paired" OpenSSL, without affecting any of the other ports. Yes this is leaving out a lot of "devil in the details" but that general concept is what I was referring to.
 
Well, what do you know? I went back and re-ran the command to update the Apache port, and this time it worked. No more complaining about having the wrong version of Perl to install it, or having the Perl port fail to install. After some checking, it appears that along the line, somewhere between my initial failure (about a month ago) and now, the correct version of Perl got installed. I'm thinking I probably managed to do this by using pkg to do a binary install of Perl.

So here we have one hand helping the other. At first, neither Apache nor Perl would install from source. The binary install of Apache didn't have appropriate security, but a new binary install of Perl enabled a source install of Apache to work, which has the up-to-date OpenSSL linked in and all known vulnerabilities patched.

And now all of a sudden my web server is back to a grade of A, instead of the C it had while running the binary-installed Apache.

So thanks everyone for the hints and tips. Definitely a learning experience - binary installs can be very convenient although inflexible. Although my boss is starting to make noises along the lines of, since FreeBSD usually seems to be such a hassle to update, maybe we should switch to Windows for this web server (we already have a couple other web servers running on Windows, doing other things). For now, I'm just glad to get on to the next task.
 
I speak from experience, over the last 3 years or so. I've spent way more time on these forums, trying to find out why something didn't work, and getting at times vague answers like "something must be wrong with your system," than on actually running updates. As I recall, we do have another FreeBSD server, but someone else maintains that one -- and it doesn't have to be kept updated to be annually certified like this one does. Anyway it would be his responsibility to set up something like a repository, and he doesn't have time. But a repository of ports only pushes the problem back a level, and doesn't address updating the base system. Plus as I said earlier, I had to jump out of the ports system, into the packages system, in order to fix the latest issue. (While first having to learn about how packages were different of course. Perhaps the hardest part of all this is filtering out the true information from all the inconsistent and sometimes contradictory sets of instructions on various web sites.)
 
There are multiple advantages for having your own repository. For one, you can build "offline", i.e. you can solve build errors without toughing the production machines. The other benefit is that you can change options (enable or disable) to get the packages you need. And lastly, once everything is built all you have to do is run pkg-upgrade(8) on your production machines.
 
You must log in or register to reply here.
Back
Top