Updated host to 13.0, now how to handle existing jails?

[jimbobmcgee]

I have recently bitten the bullet and taken my old 10.3 home lab server (HP Microserver N54L) up to 13.0. I believe it went well -- certainly less painfully than I'd feared.

My lab runs off a USB boot/root, with some dirs (/usr/local, /usr/share, /usr/home, /var, /var) mounted from a ZFS pool. I installed "fresh-ish" by building a new USB boot/root, using gpart to create a GPT on it, install the pmbr and gptboot bootcode, and bsdinstall distextract to unpack base.txz, lib32.txz and kernel.txz to it. I did some jiggery with my ZFS mountpoints and sysutils/cpdup and have my system back to where it was pre-upgrade, using my old /var and /usr/home. Finally, a freebsd-update to get my system up to 13.0-RELEASE-p3.

I took an awful lot of guidance from https://forums.freebsd.org/threads/...bout-installing-freebsd-on-a-usb-stick.11715/ and https://wiki.freebsd.org/RootOnZFS/GPTZFSBoot -- many thanks to the writers of those; any mistakes are my own.

I haven't done a zpool upgrade yet so, technically, I can still put the old USB back in and boot my old 10.3 system, but I'm hoping not to have do that any more.

However, the meat of my FreeBSD usage is in single-purpose jails. Those jails are built from old 10.1 and 10.3 templates, which I built from a scripted process, stashed in ZFS and made read-only. Jail-specific fstab files mount the read-only template root, a matching read-write data filesystem (/usr/local, /var, /tmp, /etc, some others) and any other directories I need, all via nullfs. Most applications I run from these jails were then installed via pkg-static.

Obviously any new jail I create from here is going to use my newly-minted 13.0-RELEASE jail template, but (appreciating the self-imposed complexity of my system) what is my best course of action regarding maintaining the existing jails, particularly if I wish to resume pkg operations in them (either for updates or new features)?

• Should I alter the existing fstabs to mount my new 13.0-RELEASE template root in place of the old 10.x root? Will I have a bad day(TM) with jails containing a read-only 13.0 root and 10.x writables?
• Should I freebsd-update the existing 10.1 and 10.3 templates up to 13.0? Can I even go up from 10.x to 13.0 via tools? How might one do an update to a "shared" read-only 10.x root?
• Should I do nothing? Is the host running 13.0 enough (for some definition of "enough")
• Should I rebuild all the jails from scratch Hoping not to have to do this, but am open to persuasion
• Some other action?
• Some combination

Have made a poll (since I'm asking a question), but discussion in the thread is very welcome. Any particular direction for approaching the upgrade would be appreciated.

 

[SirDice]

I'd try and migrate the jails and use a jail management tool like sysutils/bastille. That will make it much easier to keep the jails updated in the long run.

 

[jimbobmcgee]

That's an interesting tool, which didn't exist when I first put my lab together, but ultimately has similar aims. Long-term, that might well be an option, but I will need to read up on it first.

In the meantime, however, I'm still stuck as to how to update a series of existing shared/common-root 10.x jails to 13.0.

Can freebsd-update even go from 10.x to 13.0? If I do manage to get one of them updated, would a second one realise it had something to update? If I just mount the 13.0 jail template, how would I drag up the /usr/local and /etc to 13.0?

 

[SirDice]

which didn't exist when I first put my lab together

It's a fairly new tool, so that's entirely possible. On my host I used to use EZJail but that hasn't been updated in years and started to show its age. So now I'm migrating each jail one by one. You can have both EZJail and Bastille running, or in your case have the old 'handmade' jails running next to bastille.

Can freebsd-update even go from 10.x to 13.0?

You may need to make sure that 10.x has the latest patch revision, if I recall correctly there was a bug in freebsd-update(8) that was fixed that prevented this. But yes, you normally should be able to go from 10.x to 13.0 in one step, there's no need to upgrade to any of the intermediate versions.

If I do manage to get one of them updated, would a second one realise it had something to update?

No, for all intents and purposes a jail is similar to a stand-alone server. If you upgrade one machine on your network the other machines don't have those updates either.

If I just mount the 13.0 jail template, how would I drag up the /usr/local and /etc to 13.0?

/usr/local is used by packages/ports, so you'll need to upgrade those too (you always have to reinstall everything after a major version upgrade anyway). As for /etc/, you can use mergemaster(8) or etcupdate(8).

Upgrading might be a rather slow and meticulous process. I would opt to create a new jail with bastille and just migrate the data from the old 'handmade' one to a new bastille jail. That way you can do this one by one and, if the bastille jail fails (because it wasn't set up properly for example) you can always start that old jail again. It's the same when you upgrade to a new system, you build it up next to the old system and make the switch when you're satisfied the new system is set up correctly.

 

[jimbobmcgee]

No, for all intents and purposes a jail is similar to a stand-alone server. If you upgrade one machine on your network the other machines don't have those updates either.

...but if multiple jails share the same read-only base via nullfs mounts, then remounting one base read-write and updating that affects all jails that use the same base. But that means the first jail is fully-updated, and the others are only partially updated. How would a tool like freebsd-update handle that in the second jail I updated.

From the looks of https://github.com/BastilleBSD/bastille/blob/master/usr/local/share/bastille/upgrade.sh, even Bastille won't handle this ( bastille upgrade only appears to work for Bastille-termed "thick" jails which, as far as I can tell, are literal copies (or zfs send | zfs receive of a template release.

/usr/local is used by packages/ports, so you'll need to upgrade those too (you always have to reinstall everything after a major version upgrade anyway). As for /etc/, you can use mergemaster(8) or etcupdate(8).

Appreciating that you haven't said to use freebsd-update anywhere, I vaguely recall being told never to mix freebsd-update and mergemaster. Am I misremembering that, or does that still stand?

You may need to make sure that 10.x has the latest patch revision, if I recall correctly there was a bug in freebsd-update(8) that was fixed that prevented this. But yes, you normally should be able to go from 10.x to 13.0 in one step, there's no need to upgrade to any of the intermediate versions.

Don't suppose you remember the details of that bug, or where I might read up about it? Trying to determine if it's the same one I'm seeing...

As a test, I've taken one of my less important jails and zfs clone-ed the shared template root into its own filesystem, so it is now technically entirely independent. I've run freebsd-update -r 13.0-RELEASE --currently-running 10.3-RELEASE-p16 upgrade install and can see a few thousand patches and files fetched, but the install doesn't seem to complete -- at least $status is 1 and, after restarting the jail, freebsd-version still reports 10.3-RELEASE-p16, so I assume it hasn't completed.

During the Install phase, I see a bunch of Operation not permitted messages:

Installing updates...chflags: ///bin/rcp: Operation not permitted
chflags: ///lib/libc.so.7: Operation not permitted
chflags: ///lib/libcrypt.so.5: Operation not permitted
chflags: ///lib/libthr.so.3: Operation not permitted
chflags: ///libexec/ld-elf.so.1: Operation not permitted
...

...which I guess are due to these being on ZFS, rather than UFS filesystems. If that's not the issue, is there any logging/verbosity I can apply to see exactly why the freebsd-update is not completing? I don't see a -v option.

 

[SirDice]

...but if multiple jails share the same read-only base via nullfs mounts, then remounting one base read-write and updating that affects all jails that use the same base. But that means the first jail is fully-updated, and the others are only partially updated. How would a tool like freebsd-update handle that in the second jail I updated.

In that case you only update that base jail. As all the other jails are using the base jail they're all upgraded at the same time.

From the looks of https://github.com/BastilleBSD/bastille/blob/master/usr/local/share/bastille/upgrade.sh, even Bastille won't handle this ( bastille upgrade only appears to work for Bastille-termed "thick" jails which, as far as I can tell, are literal copies (or zfs send | zfs receive of a template release.

No, the thin jails use a similar construct. You update the base jail and that upgrades all jails built from that base jail.

root@armitage:~ # mount | grep bastille | grep games
zroot/bastille/jails/games on /bastille/jails/games (zfs, local, noatime, nfsv4acls)
zroot/bastille/jails/games/root on /bastille/jails/games/root (zfs, local, noatime, nfsv4acls)
/bastille/releases/13.0-RELEASE on /bastille/jails/games/root/.bastille (nullfs, local, noatime, read-only, nfsv4acls)
devfs on /bastille/jails/games/root/dev (devfs)
root@armitage:~ # ll /bastille/jails/games/root/
total 174
drwxr-xr-x  18 root  wheel    22 Apr 25 19:25 .bastille/
-rw-r--r--   1 root  wheel  1023 Apr  9 08:17 .cshrc
-rw-r--r--   1 root  wheel   507 Apr  9 08:17 .profile
drwxr-xr-x   2 root  wheel     2 May  8 18:00 .template/
-r--r--r--   1 root  wheel  6109 Apr  9 08:26 COPYRIGHT
lrwxr-xr-x   1 root  wheel    14 May  8 18:00 bin@ -> /.bastille/bin
lrwxr-xr-x   1 root  wheel    15 May  8 18:00 boot@ -> /.bastille/boot
dr-xr-xr-x  10 root  wheel   512 Jul 10 18:30 dev/
drwxr-xr-x  27 root  wheel   108 May  9 13:15 etc/
lrwxr-xr-x   1 root  wheel     8 May  8 18:00 home@ -> usr/home
lrwxr-xr-x   1 root  wheel    14 May  8 18:00 lib@ -> /.bastille/lib
lrwxr-xr-x   1 root  wheel    18 May  8 18:00 libexec@ -> /.bastille/libexec
drwxr-xr-x   2 root  wheel     2 Apr  9 08:09 media/
drwxr-xr-x   2 root  wheel     2 Apr  9 08:09 mnt/
drwxr-xr-x   2 root  wheel     2 Apr  9 08:09 net/
dr-xr-xr-x   2 root  wheel     2 Apr  9 08:09 proc/
lrwxr-xr-x   1 root  wheel    17 May  8 18:00 rescue@ -> /.bastille/rescue
drwxr-x---   2 root  wheel     9 Jul 21 21:36 root/
lrwxr-xr-x   1 root  wheel    15 May  8 18:00 sbin@ -> /.bastille/sbin
drwxrwxrwt   7 root  wheel     7 Aug 11 03:01 tmp/
drwxr-xr-x   6 root  wheel    15 May  8 18:05 usr/
drwxr-xr-x  24 root  wheel    24 Jul 10 18:30 var/

Don't suppose you remember the details of that bug, or where I might read up about it? Trying to determine if it's the same one I'm seeing...

It was this one: https://www.freebsd.org/security/advisories/FreeBSD-EN-16:09.freebsd-update.asc

I've run freebsd-update -r 13.0-RELEASE --currently-running 10.3-RELEASE-p16 upgrade install and can see a few thousand patches and files fetched, but the install doesn't seem to complete

You'll need to run freebsd-update install three times. First run will only update the kernel, second run will update the userland and the third run will remove old libraries and other files.

 

[jimbobmcgee]

You'll need to run freebsd-update install three times. First run will only update the kernel, second run will update the userland and the third run will remove old libraries and other files.

I couldn't even get freebsd-update install to complete once. I saw a bunch of Operation not permitted messages. I eventually worked out that this was because I was trying to run it inside my jail, which prevents the chflags operations that freebsd-update does internally. Not sure if that is documented, but I didn't discover it via documentation, so it might be worth updating chflags(1) and freebsd-update(8) to call this out explicitly.

Once I mounted all my jail filesystems in the right order/to the right mountpoints, and issued freebsd-update -b /pathto/jail ... from the host, my updates ran OK.

____

No, the thin jails use a similar construct. You update the base jail and that upgrades all jails built from that base jail.

Yes, on re-read of the Bastille source, I can see that now.

But I still can't see how that then updates the contents of /etc, which is part of the thin jail filesystem and not the release filesystem.

A thick jail is obvious to me -- since it's a straight copy of the release, freebsd-update -b /pathto/jail --currently-running 10.1-RELEASE can update everything in the copy -- and this appears to be what bastile upgrade mythickjail 13.0-RELEASE does. A thin jail looks to have the same problem that I currently have with my hand-cranked thin jail; where, after bastile upgrade 10.1-RELEASE 13.0-RELEASE, the readonly-mounted release (symlinked from /.bastille) is updated to 13.0-RELEASE, but the read-write mounted /etc still has the 10.1-RELEASE versions of the config.

I've briefly looked at mergemaster(8) and etcupdate(8), which look like they apply here, but I can't see how you specify (or how the command otherwise determines) the old version for the three-way merge (i.e. there is no obvious equivalent --currently-running argument, like there is in freebsd-update).

 

[Professor_Fate]

I don't think freebsd-update has options to only update the world or only update /etc. If it did though I think it would make it easier to update nullfs mounted base jails properly. I seem to remember receiving the same errors you were getting, and it only updated the shared world of the jails, not any of the rw files from each jail.

At least by compiling the source you can make installworld to your base folder, and then mergemaster or etcupdate to your rw folders. But I know you're trying to use freebsd-update. Not sure there is a way.