Bailing out of a self-inflicted issue upgrading between major versions of postgresql

Just an FYI I'm using this as a scratchpad for now while I try to document for others what *not* to do when maintaining your own DIY IT Infrastructure.

Background:
I run my own Nextcloud instance on a FreeBSD server I've had for a while now, and have slacked off at paying attention to keeping the packages and base system updated. Rushing through these has introduced an issue attempting to migrate data from postgresql17-server, to postgresql18-server. On hindsight, this could have been avoided by first upgrading the base system, or the version of Postgresql installed separately, but what's done is done and now you get to read how to avoid headaches introduced by not taking the time to read documentation and do things step by step.

Scenario:
On the system running the Nextcloud instance, the base system was upgraded in the following order, starting at 1:
  1. FreeBSD 14.2
  2. FreeBSD 14.4
  3. FreeBSD 15.1

And the versions of Postgresql were upgraded starting from 1 in that same maintenance period:
  1. 17.9
  2. 17.10
  3. 18.4
The problem I ran into which prompted this post was an issue with missing libraries required to run the pg-upgrade command, which involves starting both an old instance and a new instance of the postgres() executable in order to perform the migration.
ld-elf.so.1: Shared object "libssl.so.30" not found, required by "postgres"

If we compare the shared library dependencies using the ldd() command, a detailed list with two missing libraries becomes apparent:


$ ldd /tmp/pg-upgrade/usr/local/bin/postgres
/tmp/pg-upgrade/usr/local/bin/postgres:
libthr.so.3 => /lib/libthr.so.3 (0x384993d48000)
libintl.so.8 => /usr/local/lib/libintl.so.8 (0x38499522d000)
libzstd.so.1 => /usr/local/lib/libzstd.so.1 (0x384994547000)
liblz4.so.1 => /usr/local/lib/liblz4.so.1 (0x384995e10000)
libxml2.so.16 => /usr/local/lib/libxml2.so.16 (0x3849965e7000)
libssl.so.30 => not found (0)
libcrypto.so.30 => not found (0)
libz.so.6 => /lib/libz.so.6 (0x384997ae8000)
libexecinfo.so.1 => /usr/lib/libexecinfo.so.1 (0x38499892d000)
libm.so.5 => /lib/libm.so.5 (0x384998f2d000)
libicui18n.so.76 => /usr/local/lib/libicui18n.so.76 (0x384999900000)
libicuuc.so.76 => /usr/local/lib/libicuuc.so.76 (0x38499a504000)
libc.so.7 => /lib/libc.so.7 (0x38499ade5000)
libsys.so.7 => /lib/libsys.so.7 (0x38499bdf1000)
libelf.so.2 => /lib/libelf.so.2 (0x38499c129000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x384997951000)
libicudata.so.76 => /usr/local/lib/libicudata.so.76 (0x38499caed000)
libc++.so.1 => /lib/libc++.so.1 (0x38499d002000)
libcxxrt.so.1 => /lib/libcxxrt.so.1 (0x38499d6e4000)
[vdso] (0x38499326f000)
$ ldd /usr/local/bin/postgres
/usr/local/bin/postgres:
libthr.so.3 => /lib/libthr.so.3 (0x1205ef36f000)
libintl.so.8 => /usr/local/lib/libintl.so.8 (0x1205ee86f000)
libzstd.so.1 => /usr/local/lib/libzstd.so.1 (0x1205effa6000)
liblz4.so.1 => /usr/local/lib/liblz4.so.1 (0x1205f1969000)
libxml2.so.16 => /usr/local/lib/libxml2.so.16 (0x1205f25c5000)
libssl.so.35 => /usr/lib/libssl.so.35 (0x1205f0d60000)
libcrypto.so.35 => /lib/libcrypto.so.35 (0x1205f3600000)
libz.so.6 => /lib/libz.so.6 (0x1205f52b7000)
libexecinfo.so.1 => /usr/lib/libexecinfo.so.1 (0x1205f57aa000)
libm.so.5 => /lib/libm.so.5 (0x1205f5ad1000)
libicui18n.so.76 => /usr/local/lib/libicui18n.so.76 (0x1205f6023000)
libicuuc.so.76 => /usr/local/lib/libicuuc.so.76 (0x1205f6ae2000)
libc.so.7 => /lib/libc.so.7 (0x1205f4b43000)
libsys.so.7 => /lib/libsys.so.7 (0x1205f7152000)
libelf.so.2 => /lib/libelf.so.2 (0x1205f84b7000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x1205f7472000)
libicudata.so.76 => /usr/local/lib/libicudata.so.76 (0x1205f8c7e000)
libc++.so.1 => /lib/libc++.so.1 (0x1205f9346000)
libcxxrt.so.1 => /lib/libcxxrt.so.1 (0x1205fa0d7000)
[vdso] (0x1205edea5000)

I'll post more; time permitting. If anyone has a process, solution. or constructive comment, post below.
 
Well I got a lot further but I must reiterate how lucky I was to get to this point. If you try to upgrade and step things forward too fast, you might not have the ability to do what I just did.

My thought was to acquire the missing libraries from previous versions of FreeBSD. The general flow was the following:

1. Download a FreeBSD memstick image
2. mount() the image
3. Look for the missing library
4. Copy the library into the system's current libraries
5. Verify via ldd() that the libraries were detected.

I acquired FreeBSD 14.4 and managed to pull the missing libraries from their respective directories:

# fetch https://download.freebsd.org/releases/amd64/amd64/ISO-IMAGES/14.4/FreeBSD-14.4-RELEASE-amd64-memstick.img
# mdconfig -a -t vnode -f /home/usr/FreeBSD-14.4-RELEASE-amd64-memstick.img


One thing to note is when attempting to mount() the image, you have to specify the partition device, not any of the slices or the memory device itself. Otherwise you get this:

# mount /dev/md0 /tmp/FreeBSD14.4/
mount: /dev/md0: No such file or directory
# mount /dev/md0s1 /tmp/FreeBSD14.4/
mount: /dev/md0s1: No such file or directory


Instead, when you mount the partition, you'll get this:
# mount /dev/md1s2a /tmp/FreeBSD14.4
# ls /tmp/FreeBSD14.4
.cshrc bin etc media proc sbin var
.profile boot lib mnt rescue tmp
COPYRIGHT dev libexec net root usr


So each of the libraries were located in the following paths:

libssl.so.30 -> /usr/lib/libssl.so.30
ibcrypto.so.30 -> /lib/libcrypto.so.30


Now, when copying *any* library into a running system, you *don't* want to overwrite anything. I'm not even sure if this is safe on a running system, but I presume that's why the numbers at the end of the file name are there as sort of a scheme to prevent accidentally loading a different version of the library if a previous version is still present in that directory:

# cp -i /tmp/FreeBSD14.4/lib/libcrypto.so.30 /lib


..and after copying the files over, voila, ldd reports all is well:

# ldd /tmp/pg-upgrade/usr/local/bin/postgres
/tmp/pg-upgrade/usr/local/bin/postgres:
libthr.so.3 => /lib/libthr.so.3 (0x3e89760ef000)
libintl.so.8 => /usr/local/lib/libintl.so.8 (0x3e897681b000)
libzstd.so.1 => /usr/local/lib/libzstd.so.1 (0x3e8977a60000)
liblz4.so.1 => /usr/local/lib/liblz4.so.1 (0x3e89785aa000)
libxml2.so.16 => /usr/local/lib/libxml2.so.16 (0x3e897911e000)
libssl.so.30 => /usr/lib/libssl.so.30 (0x3e897a0fb000)
libcrypto.so.30 => /lib/libcrypto.so.30 (0x3e897a5ee000)
libz.so.6 => /lib/libz.so.6 (0x3e897aadc000)
libexecinfo.so.1 => /usr/lib/libexecinfo.so.1 (0x3e897b012000)
libm.so.5 => /lib/libm.so.5 (0x3e89775d1000)
libicui18n.so.76 => /usr/local/lib/libicui18n.so.76 (0x3e897c120000)
libicuuc.so.76 => /usr/local/lib/libicuuc.so.76 (0x3e897caad000)
libc.so.7 => /lib/libc.so.7 (0x3e897d461000)
libsys.so.7 => /lib/libsys.so.7 (0x3e897e21d000)
libelf.so.2 => /lib/libelf.so.2 (0x3e897ef69000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x3e897f2b5000)
libicudata.so.76 => /usr/local/lib/libicudata.so.76 (0x3e89801a1000)
libc++.so.1 => /lib/libc++.so.1 (0x3e8980d8d000)
libcxxrt.so.1 => /lib/libcxxrt.so.1 (0x3e898127e000)
[vdso] (0x3e8975609000)

Of course, the next hurdle was encountered:


# su - postgres
$ pg_upgrade \
--old-bindir=/tmp/pg-upgrade/usr/local/bin \
--new-bindir=/usr/local/bin \
--old-datadir=/pool/var/databases/pgsql-17/data \
--new-datadir=/var/db/postgres/data18
Performing Consistency Checks
-----------------------------
Checking cluster versions ok

old cluster does not use data checksums but the new one does
Failure, exiting


These can be disabled (see pg_checksums)


$ pg_checksums --pgdata /var/db/postgres/data18 --disable
pg_checksums: syncing data directory
pg_checksums: updating control file
Checksums disabled in cluster


Which, of course, results in some warnings, but it does complete the upgrade!


$ pg_upgrade \
--old-bindir=/tmp/pg-upgrade/usr/local/bin \
--new-bindir=/usr/local/bin \
--old-datadir=/pool/var/databases/pgsql-17/data \
--new-datadir=/var/db/postgres/data18
Performing Consistency Checks
-----------------------------
Checking cluster versions ok
Checking database connection settings ok
WARNING: database "nextcloud" has a collation version mismatch
DETAIL: The database was created using collation version 43.0, but the operating system provides version 48.
HINT: Rebuild all objects in this database that use the default collation and run ALTER DATABASE nextcloud REFRESH COLLATION VERS ION, or build PostgreSQL with the right library version.
WARNING: database "nextcloud" has a collation version mismatch
DETAIL: The database was created using collation version 43.0, but the operating system provides version 48.
HINT: Rebuild all objects in this database that use the default collation and run ALTER DATABASE nextcloud REFRESH COLLATION VERS ION, or build PostgreSQL with the right library version.
Checking database user is the install user ok
Checking for prepared transactions ok
Checking for contrib/isn with bigint-passing mismatch ok
Checking for valid logical replication slots ok
Checking for subscription state WARNING: database "nextcloud" has a collation version mismatch
DETAIL: The database was created using collation version 43.0, but the operating system provides version 48.
HINT: Rebuild all objects in this database that use the default collation and run ALTER DATABASE nextcloud REFRESH COLLATION VERS ION, or build PostgreSQL with the right library version.
ok
Checking data type usage WARNING: database "nextcloud" has a collation version mismatch
DETAIL: The database was created using collation version 43.0, but the operating system provides version 48.
HINT: Rebuild all objects in this database that use the default collation and run ALTER DATABASE nextcloud REFRESH COLLATION VERS ION, or build PostgreSQL with the right library version.
ok
Checking for objects affected by Unicode update WARNING: database "nextcloud" has a collation version mismatch
DETAIL: The database was created using collation version 43.0, but the operating system provides version 48.
HINT: Rebuild all objects in this database that use the default collation and run ALTER DATABASE nextcloud REFRESH COLLATION VERS ION, or build PostgreSQL with the right library version.
ok
Checking for not-null constraint inconsistencies WARNING: database "nextcloud" has a collation version mismatch
DETAIL: The database was created using collation version 43.0, but the operating system provides version 48.
HINT: Rebuild all objects in this database that use the default collation and run ALTER DATABASE nextcloud REFRESH COLLATION VERS ION, or build PostgreSQL with the right library version.
ok
Creating dump of global objects ok
Creating dump of database schemas
ok
Checking for presence of required libraries ok
Checking database user is the install user ok
Checking for prepared transactions ok
Checking for new cluster tablespace directories ok

If pg_upgrade fails after this point, you must re-initdb the
new cluster before continuing.

Performing Upgrade
------------------
Setting locale and encoding for new cluster ok
Analyzing all rows in the new cluster ok
Freezing all rows in the new cluster ok
Deleting files from new pg_xact ok
Copying old pg_xact to new server ok
Setting oldest XID for new cluster ok
Setting next transaction ID and epoch for new cluster ok
Deleting files from new pg_multixact/offsets ok
Copying old pg_multixact/offsets to new server ok
Deleting files from new pg_multixact/members ok
Copying old pg_multixact/members to new server ok
Setting next multixact ID and offset for new cluster ok
Resetting WAL archives ok
Setting frozenxid and minmxid counters in new cluster ok
Restoring global objects in the new cluster ok
Restoring database schemas in the new cluster
ok
WARNING: database "nextcloud" has a collation version mismatch
DETAIL: The database was created using collation version 43.0, but the operating system provides version 48.
HINT: Rebuild all objects in this database that use the default collation and run ALTER DATABASE nextcloud REFRESH COLLATION VERS ION, or build PostgreSQL with the right library version.
Copying user relation files
ok
Setting next OID for new cluster ok
Sync data directory to disk ok
Creating script to delete old cluster ok
Checking for extension updates WARNING: database "nextcloud" has a collation version mismatch
DETAIL: The database was created using collation version 43.0, but the operating system provides version 48.
HINT: Rebuild all objects in this database that use the default collation and run ALTER DATABASE nextcloud REFRESH COLLATION VERS ION, or build PostgreSQL with the right library version.
ok

Upgrade Complete
----------------
Some statistics are not transferred by pg_upgrade.
Once you start the new server, consider running these two commands:
/usr/local/bin/vacuumdb --all --analyze-in-stages --missing-stats-only
/usr/local/bin/vacuumdb --all --analyze-only
Running this script will delete the old cluster's data files:
./delete_old_cluster.sh
 
I love postgresql but its migration tools kind-a suck. For me, to avoid problems its always a complete DB export and then re-import under the newer version. IIRC pg-upgrade requires that you have both versions present and executable on your machine? The only reason I think I remember that is that at the time I found it to be quite an annoying requirement.

For a while I did run two versions but since the older server required the older libraries I mounted the old system root and did some LD_LIBRARY_PATH trickery to get the old version to run on the newer system. You can infer from that, that I never upgrade over top of an existing OS version, but always flip flop between two recent root filesystem partitions.
 
I can see it's to do with going from 14.x (OpenSSL 3.0) to 15.x (OpenSSL 3.5) but do you know where you went wrong?

What was the short-cut that caused this?

Did you do all the OS upgrades first, then try and tidy-up PostgreSQL packages at the end once you were on 15.1?

So you did the move to 15.1 without running any PostgreSQL package update steps? So you had a 14.x PostgreSQL binary looking for the OpenSSL 3.0 libraries that no longer existed on 15.1?

But you should have done each OS upgrade, then the PostgreSQL package upgrade (plus any PostgreSQL migration steps)?

Or something else?

(That's a lot of question marks!)
 
Code:
pkg fetch -o /tmp postgresql-server17
cd /tmp/
mkdir pg-upgrade
cd pg-upgrade
tar -zxvf /tmp/All/postgresql*.pkg

This will fetch the postgresql-server17 package that was built for 15 (and thus doesn't depend on various libraries from 14.x). And can safely use --old-bindir=/tmp/pg-upgrade/usr/local/bin without running into missing libraries.
 
I love postgresql but its migration tools kind-a suck. For me, to avoid problems its always a complete DB export and then re-import under the newer version. IIRC pg-upgrade requires that you have both versions present and executable on your machine?
This. Their default migration path also sucks because you need to install 2 separate versions - which does not work for very good reasons. (Yes, you could fiddle around with unpacking the necessary files to another location and hope there are no broken dependencies or it does not mangle with the wrong path).
So usually I avoid those updates as long as possible; usually this gets you a good 3-4 years. But if I am finally forced to do so, I create a new jail, install the new postgresql version, copy over the postgres configuration and then pg_dump | nc / nc -l | pg_restore from the old to new jail - which is PAINFULLY slow even on quite beefy servers and for moderately sized databases...
Afterwards shutdown the old jail to free its IP (or adjust your DNS records) and everything should now run on the new postgres server. (IIRC homeassistant needs a restart because it will just error out and sit there doing nothing if the database connection was lost.)
 
Back
Top