Performing an unlimited amount of major version upgrades without lifting a finger

Sebulon · Jan 18, 2020

I have always heard how rock solid FreeBSD is as an operating system and if you only have one or two machines, it's alright, you know, no larger issues. freebsd-update isn't great but it's not terrible either, although it's biggest flaw is that it can't really run unmanaged. It stops and asks you stuff, important stuff too, but what if it's your 10th machine doing it? When going over two or three it starts getting really tedious patching, upgrading and keeping everything secure, as it should be. When something is perceived as too time-consuming, chances are that many admins just skip it, or at least tend doing it as infrequently as possible - "don't fix what ain't broken" and all that. No wonder the OS feels solid when nothing rarely ever changes! Larger companies and organizations have specialized ways to upgrade fleets en masse, but the upgrades mostly only works for them, or only works under very narrow conditions.

Since a while back now, we've had the amazing help of pkg and poudriere to create an elegant, simple and unmanaged way of keeping ports updated, and since FreeBSD 12, there is PkgBase! There are some prerequisites for understanding this guide:

And when saying 'without lifting a finger' there's obviously stuff you need to get done, but as I like to think: "Work as hard as you can at doing as little as possible!"

1. Setting up PkgBase
When following the guide on how to set up Poudriere, you get a machine to build the PkgBase packages on. After that's completed and you have a repo set up for it, you also need to serve out /usr/src readonly to all the systems expected to use this. Why comes later!

/etc/rc.conf:

Code:

nfs_server_enable="YES"
nfs_server_flags="-u -t -n 4"
mountd_flags="-r"
mountd_enable="YES"
rpcbind_enable="YES"
rpc_statd_enable="YES"
rpc_lockd_enable="YES"

/etc/exports:

Code:

/usr/src        -maproot=root -ro -network 10.0.0.0/24

service nfsd start

2. The Victim
Install a default FreeBSD-11 system that you then can upgrade to 12 using the packages served from the Poudriere-machine!

Start by mounting the build server's /usr/src filesystem inside the machine getting upgraded:
mount -t nfs URL.TO.PKGBASE.MACHINE:/usr/src /usr/src

Then, set up the repo. I have my ports packages served from port 80 and the base packages from 8080 but you can of course set it up however you like:

Code:

# cat > /usr/local/etc/pkg/repos/FreeBSD-base.conf << EOF
# FreeBSD base system repository
FreeBSD-base: {
  url: "http://URL.TO.PKGBASE.MACHINE:8080/FreeBSD-base/FreeBSD:12:amd64/latest",
  enabled: yes
}
EOF

We update the package database:
env ABI=freebsd:12:amd64 pkg update -r FreeBSD-base
And answer 'y' to change to the newer version. Then it's upgrade time, boys and girls!
pkg install -g 'FreeBSD-*'

We merge all of our config files:
etcupdate
And clean up all of the '.pkgnew' files lying around:
find / -xdev -type f -name '*.pkgnew' -exec rm {} \;

From following the Poudriere tutorial, you should also have a repo file for your ports, like poudriere.conf. After installing the newer base packages for FreeBSD 12, all of your ports also needs to change. So edit the Poudriere repo to point at the new package version, if, for instance, you've had:
/usr/local/etc/pkg/repos/poudriere.conf:

Code:

poudriere: {
    url: "http://URL.TO.PKGBASE.MACHINE/packages/freebsd_11-2amd64/",
    mirror_type: "http",
    signature_type: "pubkey",
    pubkey: "/usr/local/etc/ssl/certs/poudriere.cert",
    enabled: yes
}

We change that to 12.1, like so:
sed -i '' 's/freebsd_11-2amd64/freebsd_12-1amd64/' /usr/local/etc/pkg/repos/poudriere.conf
Update the package database:
pkg update -r poudriere
Then we make sure to only upgrade the ports packages, so that we don't reinstall the base packages again (the OS gets really, really cranky when that happens!):
pkg upgrade -f -r poudriere

And last of all:
reboot

3. Conclusion
This is awesome! Put in a very simple script:

/usr/local/sbin/freebsd-dist-upgrade:

Code:

mount -t nfs URL.TO.PKGBASE.MACHINE:/usr/src /usr/src
cat > /usr/local/etc/pkg/repos/FreeBSD-base.conf << EOF
# FreeBSD base system repository
FreeBSD-base: {
  url: "http://URL.TO.PKGBASE.MACHINE:8080/FreeBSD-base/FreeBSD:12:amd64/latest",
  enabled: yes
}
EOF
IGNORE_OSVERSION=yes env ABI=freebsd:12:amd64 pkg update -r FreeBSD-base
pkg install -y -g 'FreeBSD-*'
etcupdate
find / -xdev -type f -name '*.pkgnew' -exec rm {} \;
sed -i '' 's/freebsd_11-2amd64/freebsd_12-1amd64/' /usr/local/etc/pkg/repos/poudriere.conf
pkg update -r poudriere
pkg upgrade -f -y -r poudriere
reboot

With this script, we can upgrade as many FreeBSD systems from version 11 to 12 as we like; without lifting a finger

Performing an unlimited amount of major version upgrades without lifting a finger

Sebulon