How to deal with the Berkeley DB 4.x cleanup

Hi gang,

Note: In this HOWTO you may notice 2 server names in my examples: smtp2 and chihiro. That's because at the time of writing I'm actually in the process of performing this cleanup myself. Everything I wrote has already been performed on my smtp2 server, as such I'm turning to chihiro whenever I need to show an example of a situation prior to an update.

Introduction

Berkeley DB is a library which provides an easy way to use database storage. It's often used in software which needs to be able and store lists of some sort while also having the option to quickly retrieve those lists again. Unfortunately the program has one "flaw"; although it provides minor upgrades between versions (for example going from 4.1 to 4.2) more than often these aren't minor upgrades at all. Features get added and worse; more than often the newer version isn't (fully) downwards compatible with the previous one.

Partly because of this several software projects have opted to focus their attention on a specific version. And because newer versions don't necessarily supersede older ones you'll often end up in what I tend to call a "Berkeley DB mess". Meaning so much as ending up with several versions which have been installed besides each other:

Code:
chihiro:/home/peter $ pkg_info -Ix db4
db41-4.1.25_4       The Berkeley DB package, revision 4.1
db42-4.2.52_5       The Berkeley DB package, revision 4.2
db47-4.7.25.4       The Berkeley DB package, revision 4.7
Although this isn't a real problem it does tend to make things a little messy. Not to mention providing more hassle when it comes to server administration. After all; having to deal with one package and the ports which depend on it is easier than having to deal with three (or maybe even more).

And that's only talking from the position of the end user; what to think about the maintainer for all this?

The Berkeley DB (4.x) cleanup

Because of this the maintainers have decided to clean up this mess and reduce the amount of Berkeley DB versions from ten to two; databases/db5 and databases/db6. To achieve this all the 4.x versions of Berkeley DB, apart from version 4.8, have been marked deprecated from now on.

Which of course leaves us with a problem; how to go from three versions (in my example) to one?

Now, I think the people behind all this have done an excellent job in documenting the process but in the end it's basically a summary of steps to take. In all honesty I do believe that if you're administrating an environment which is affected by this you should already be aware of how to handle the change. Even so, I think some extra information can't hurt and might even be useful for some of you out there.

Plan of action

The first thing to do is come up with a plan of action. In my situation there are some important services involved such as Postfix, Dovecot and Apache. I can't afford to have these unavailable for too long, and therefore I need to make sure that I'm going to address this as efficiently as possible. These are the steps which I suggest to take:

  1. Identify the involved software.
  2. Make a backup of the system.
  3. Install the new Berkeley DB version (and configure it).
  4. Check that the involved package does indeed use the (new) Berkeley DB version.
  5. Reinstall the involved software package.
  6. Re-create or re-initialize the databases (when you need to).
  7. Clean up.
  8. Start again at step 4 for the next software package.

1 - Identify the involved software
As we could see in my example above I have to deal with 3 versions of the Berkeley DB database and I'll start with version 4.1. First we're going to check which software we need to rebuild:

Code:
chihiro:/home/peter $ pkg_info -Rx db41
Information for db41-4.1.25_4:

Required by:
webalizer-geoip-2.23.8
cyrus-sasl-2.1.26_3
postfix-2.10.2,1
If you're using pkgng then you'd be using something like this: pkg info -rx db41.

In this example I'm going to work on Postfix.

2 - Make a backup of the system
Now, obviously I can't tell you how you should backup your system, but I figured I'd better not skip this step because it is an important one. Always make sure that you have an exit strategy!

3 - Install the new Berkeley DB version

There are officially 3 versions to chose from, but for this example we're going to limit our options to versions 5 and 6. My suggestion would be to install version 5. Not just because of the somewhat different and in my opinion limited licensing scheme in version six, but more so because I consider it more riskfull to skip versions in between. Going from version 4 to 6 is something I consider to be undesirable, even though it may be perfectly doable.

As such we're going to install version 5. We can do so using either of these two commands:

# make -C /usr/ports/databases/db5 install clean

or

# portmaster databases/db5

It doesn't really matter too much which of these two you'll be using.

When you're done make sure to add this line to your /etc/make.conf file in order to make sure that the ports involved will now favour version five:

Code:
WITH_BDB_VER=5

4 - Check that the package uses the (new) Berkeley DB version
First we're going to identify the database files being used:

Code:
chihiro:/usr/local/etc/postfix $ grep hash main.cf
transport_maps = hash:/usr/local/etc/postfix/transport_maps
virtual_alias_maps = hash:/usr/local/etc/postfix/virtual,
smtpd_recipient_restrictions = permit_mynetworks permit_sasl_authenticated reject_unauth_destination check_recipient_access hash:/usr/local/etc/postfix/access_recipients check_policy_service inet:127.0.0.1:10023
relay_recipient_maps = hash:/usr/local/etc/postfix/relay_recipients
chihiro:/usr/local/etc/postfix $ file relay_recipients.db
relay_recipients.db: Berkeley DB (Hash, version 8, native byte-order)
In this particular example we also don't want to overlook /etc/aliases which is also being used. With Postfix we are responsible for creating these databases ourselves, you can do this by using the postmap or postalias commands. Which is good to know; so all we need to do now is get Postfix to use the new Berkeley DB version.

When checking the port itself you'll notice that it has already switched dependencies:

Code:
root@smtp2:/usr/ports/mail/postfix # make run-depends-list
/usr/ports/databases/db5
/usr/ports/databases/sqlite3
/usr/ports/databases/tinycdb
/usr/ports/devel/pcre
/usr/ports/mail/dovecot
/usr/ports/mail/libspf2
/usr/ports/security/cyrus-sasl2
So all that's left to do now is rebuild the port. Note that I'm specifically building and not reinstalling just yet. Keep in mind that this is a mail server we're dealing with, and in my case it's being used by dozens of customers. As such I cannot afford to end up with a "Sorry, it didn't work" kind of situation:

Code:
root@smtp2:/usr/ports/mail/postfix # make build
  [ ... build output deleted ... ]
root@smtp2:/usr/ports/mail/postfix # cd work/postfix-2.10.2/bin
root@smtp2:/usr/ports/mail/postfix/work/postfix-2.10.2/bin # ls postmap postalias
postalias       postmap
root@smtp2:/usr/ports/mail/postfix/work/postfix-2.10.2/bin # ldd postmap
postmap:
        libpcre.so.3 => /usr/local/lib/libpcre.so.3 (0x80084a000)
        libsasl2.so.3 => /usr/local/lib/libsasl2.so.3 (0x800aac000)
        libpam.so.5 => /usr/lib/libpam.so.5 (0x800cc7000)
        libcrypt.so.5 => /lib/libcrypt.so.5 (0x800ed2000)
        libssl.so.6 => /usr/lib/libssl.so.6 (0x8010f2000)
        libcrypto.so.6 => /lib/libcrypto.so.6 (0x801347000)
        libspf2.so.3 => /usr/local/lib/libspf2.so.3 (0x8016ef000)
*>     libdb-5.3.so.0 => /usr/local/lib/libdb-5.3.so.0 (0x801909000)
        libsqlite3.so.8 => /usr/local/lib/libsqlite3.so.8 (0x801c94000)
        libcdb.so.1 => /usr/local/lib/libcdb.so.1 (0x801f3d000)
        libc.so.7 => /lib/libc.so.7 (0x802140000)
        libthr.so.3 => /lib/libthr.so.3 (0x802491000)
Don't mind the little arrow; I added that myself to point out what we're looking for here. As you can see it's using /usr/local/lib/libdb-5.3.so.0 which is obviously part of the new Berkeley DB library:

Code:
root@smtp2:/usr/ports/mail/postfix/work/postfix-2.10.2/bin # pkg_info -W /usr/local/lib/libdb-5.3.so.0
/usr/local/lib/libdb-5.3.so.0 was installed by package db5-5.3.21.0
If you're using pkgng then you can use this command: pkg which /usr/local/lib/libdb-5.3.so.0.

5 - Reinstall the involved software package
The best way to do this is by using portmaster. Not only because it will take care of everything, such as the dependencies, but also because it can automatically create a backup package for us when using the -b parameter. But of course you can also opt to use the more traditional approach by running # make reinstall clean.

Finally; when it comes to server processes such as Postfix, Dovecot and Apache I also prefer shutting the service down before reinstalling. Normally this is done automatically but with services such as these I think it's very important to make sure that everything runs smoothly.

You don't want to end up discovering an existing problem only after you single handedly added a second one yourself.

6 - Re-create or re-initialize the databases
Now that Postfix uses the new Berkeley DB format we can (quickly) rebuild the databases before starting the service again. But as always we're going to try if things work first by testing it on an empty database:

Code:
root@smtp2:/usr/local/etc/postfix # ls -l bcc*
-rw-r--r--  1 root  wheel      0 Jun  7  2013 bcc
-rw-r--r--  1 root  wheel  49152 Dec 16 22:07 bcc.db
root@smtp2:/usr/local/etc/postfix # postmap bcc
root@smtp2:/usr/local/etc/postfix # file bcc.db
bcc.db: Berkeley DB (Hash, version 9, native byte-order)
As you can see it now uses the new version 9 instead of the previous version 8 (see output above).

Now we're ready to convert all databases and then quickly start Postfix again. In a regular situation I'd recommend using something like this:

Code:
root@smtp2:/usr/local/etc/postfix # ls *db
bcc.db                  transport_maps.db
relay_recipients.db     virtual.db
root@smtp2:/usr/local/etc/postfix # foreach a (*db)
foreach? postmap `basename $a .db`
foreach? end
Why start with an ls command? That should be obvious: to make sure that we're not going to use the postmap command on the wrong file and by doing so risk to compromise the systems integrity. Remember: safety comes first!

Now, I'm not going into all the details, but if you're following this HOWTO / tutorial yourself and are actually working on Postfix then don't forget about /etc/aliases!

7 - Clean up
In this case there isn't much to clean up because we have already made sure that the port was cleaned after reinstalling (either by doing this ourselves or by letting portmaster handle this).

But there is more to cleaning up than this (in my opinion). Now it's time to check that everything went according to plan, lets check if there are any files depending on the new Berkeley DB version:

Code:
smtp2:/home/peter $ pkg_info -Rx db5
Information for db5-5.3.21.0:

Required by:
postfix-2.10.2,1
And what do you know? It worked. If you're using pkgng you can use this command: pkg info -rx db5.

8 - Repeat until done
And that's all she wrote.

Keep in mind that some of the steps which I demonstrated above are not always usable, it heavily depends on the port you're working on. Also; if you're dealing with a package where you have little or no control over the database(s) (mail/postgrey and mail/mailman come to mind) then be sure to make an extra backup of the database files just in case. Also remember to use the file command as shown above to check that the database was actually converted.

Another thing to keep in mind is that it seems that some packages seem to be pinpointed on using the previous version. I'm currently looking into ports such as www/gallery2, www/wordpress and textproc/docproj myself.

In those cases its good to remember commands such as # make all-depends-list (see ports(7)), ldd and portmaster.

Sometimes it is possible that everything is in place, the software was actually build against the right version, but the dependency is still pointed to the old Berkeley DB port. If worse comes to worst then keep in mind that you can always try to force things when required. portmaster can be a powerful tool when using its --check-depends feature.

But, as mentioned earlier: safety first. Make very sure that you need to take such drastic steps before taking them!

Good luck!

Erratum

A small update to the final point. I commented on how certain ports seemed to be "pin pointed" to older versions of the Berkeley database even though they had already been build against the newer version. The key word, and that which I overlooked at first, is dependencies.

For example; devel/apr1 is used by many other ports:

Code:
smtp2:/usr/local/etc/postfix $ pkg_info -Rx apr-1
Information for apr-1.4.8.1.5.3:

Required by:
apache22-2.2.26
php5-5.4.23
docproj-2.0_1
gallery2-2.3.2_1
wordpress-3.7.1,1
...
When we look at textproc/docproj you'll notice that it also has a direct dependency on the Berkeley database:

Code:
smtp2:/usr/local/etc/postfix $ pkg_info -rx docproj | grep db
Dependency: db5-5.3.28
Dependency: gdbm-1.10
But merely rebuilding this port does not change it's dependency on devel/apr1 which in its turn also depends on a version of the Berkeley database. So in the event where you rebuild one but not the other you may end up in the undesirable situation that it seems as if one port depends on two version of the Berkeley database.

The solution, and another tip in dealing with this clean-up process, is to check for those dependencies. portmaster is quite capable to rebuild a port and all the ports which depend on it. So in the case of devel/apr1 I simply issued the following command: # portmaster -r apr-1 after which everything got rebuild while automatically depending on the new version of the Berkeley database databases/db5.

So although my previous comments about having an option to enforce certain things maybe true it is most likely not required for this task.

Edit: changed markup, fixed typos, added pkg search command, added erratum, edited the edit notice :stud
 
Thanks for the good info. I've nearly completed the upgrade to databases/db5 on this server. The following commands lead me to think the job is done:
Code:
# pkg_info -Rx db44
# pkg_info -Rx db46
# pkg_info -Rx db5
All ports that had shown up as results for databases/db4 versions are now in the results of the version 5 database. And none remain linked to version 4.

My lingering question is: What about sendmail? On this server it is not installed from ports. And the commands
Code:
# ldd /usr/sbin/sendmail
# ldd /usr/sbin/makemap
do not show these programs linked to any version of bdb. Yet,
Code:
# file /etc/mail/access.db
access.db: Berkeley DB 1.85 (Hash, version 2, native byte-order)
leads me to wonder what state these files and sendmail itself are in.

Any clarifications much appreciated.
 
My lingering question is: What about sendmail? On this server it is not installed from ports. And the commands
Code:
# ldd /usr/sbin/sendmail
# ldd /usr/sbin/makemap
do not show these programs linked to any version of bdb. Yet,
Code:
# file /etc/mail/access.db
access.db: Berkeley DB 1.85 (Hash, version 2, native byte-order)
leads me to wonder what state these files and sendmail itself are in.

Any clarifications much appreciated.

Short answer, the FreeBSD libc contains Berkeley DB 1.85 and many ports are using the libc functionality if no other version is specified. For example Postfix built with "BDB=off" can create Berkeley DB 1.85 btree(3) or hash(3) databases.
 
There are officially 3 versions to chose from, but for this example we're going to limit our options to versions 5 and 6. My suggestion would be to install version 5.

Just for the record. If you use Postfix (current version 2.11.0), you can't select version 6 of BDB because it's not supported - more info in PR 185446. Slightly annoying after I had rebuilt everything but Postfix with BDB version 6 ...

@ShelLuser: Thx Thanks for the write up - the tip about devel/apr1 was very useful.
 
Last edited by a moderator:
Great thread! Thanks to all who contributed. I have just finished migration from db46 to db5. OpenLDAP migration was a bit more work. I finally decided to export the complete user DIT right before rebuilding OpenLDAP. Before I restarted slapd I deleted all files under /var/db/openldap-data. After a restart I imported the user DIT again. This worked fine but I am not sure, wether there is not a more elegant solution.

EDIT: According to the 20140318 entry of /usr/ports/UPDATING I should have better directly migrated to the MDB backend of OpenLDAP :).
 
Back
Top