jails Thinjail with nullfs built per the handbook can't be kept fully updated

N

neater_repeater

Running 13.2 and following the FreeBSD Handbook guide for jails, decided to try the thinjail+nullfs route to allow a single freebsd-update to update all jails. However, since many directories get moved out of the base jail template to a skeleton, and particularly this line in the guide:

# mv /usr/local/jails/templates/13.2-RELEASE-base/etc /usr/local/jails/templates/13.2-RELEASE-skeleton/etc
Click to expand...

there is now no /etc in the template. The current update from 13.2-RELEASE-p3 to 13.2-RELEASE-p4 wants to update certs in /etc/ssl/certs/, so doing:

freebsd-update -b /data/jails/template/base fetch install
Click to expand...

fails due to no /etc. Of course the updates would apply fine if only affecting files in /usr, /sbin, or anything else that remains in the read-only base template.

Is there a way around this? The certs in /etc/ssl/certs are all just symlinks off into /usr/share/certs/trusted, but freebsd-update specifically insists on updating them via the /etc/ssl/certs path.

As I don't have many jails built yet, I can of course switch and just do a simple ZFS template that I snapshot and clone to create new jails, but that will mean that instead of a single freebsd-update to get all jails current, it'll be a freebsd-update run separately for every jail. It does also reveal a problem where all thinjail+nullfs setups out there will never get updated system certificates.
 
I've encountered the same issue on trying to upgrade from 14.1-RELEASE-p2 to 14.1-RELEASE-p3 as explained in “17.7.2. Upgrading a Thin Jail Using NullFS”, with many
Code: 
install: /usr/local/jails/templates/14.1-RELEASE-base/etc/ssl/untrusted: realpath: No such file or directory
errors. After the attempted upgrade of the base template,
freebsd-version -j myjail reports 14.1-RELEASE-p3 as the current release in each of my jails, but
freebsd-update -j myjail IDS lists the files in /etc/ssl as different from the release. I'm not sure whether I should take some corrective action. Or perhaps just create a new base template from scratch.
 
On a cursory look, your scripts create symlinks to the base jail from the jails (I think bastille does the same, too). So, your base jail stays pristine. In the approach from the handbook, the symlinks are in the base jail (some directories are moved to a separate “skeleton”), which leads to issue mentioned by the OP when upgrading.​
​
Anyway, as I'm currently building my own scripts for managing jails, I can use yours as an inspiration! Thanks!​
 
lifepillar said:
On a cursory look, your scripts create symlinks to the base jail from the jails (I think bastille does the same, too). So, your base jail stays pristine. In the approach from the handbook, the symlinks are in the base jail (some directories are moved to a separate “skeleton”), which leads to issue mentioned by the OP when upgrading.​
​
Anyway, as I'm currently building my own scripts for managing jails, I can use yours as an inspiration! Thanks!​
Click to expand...
I shamefully copied almost all the concept from sysutils/ezjail. I used this last a longtime, but it's not really updated since ages and the upgrade/update of the base is a bit complex (also no update of the config files during an upgrade).

That's why I wrote these scripts. They do exactly what I want, no more, no less, so there is no unnecessary complication in the code (more complexity = more potential bugs). It was also an opportunity to better understand the basics of jails and no more depend of anyone for the maintenance of my jails.

So when you said, you'll write your own tools for jails, I approve at 100%. Remove unwanted features, add those you want. And create something suited for your needs.
 
To answer the OP, it seems to me that the proper way of upgrading requires mounting the skeleton on the base template and unmounting it after the upgrade. For example:


mount -t nullfs /usr/local/jails/templates/13.4-RELEASE-skeleton /usr/local/jails/templates/13.4-RELEASE-base/skeleton
freebsd-update -b /usr/local/jails/templates/13.4-RELEASE-base fetch install
umount /usr/local/jails/templates/13.4-RELEASE-base/skeleton
service jail restart


If this is correct, it would be nice if the handbook could be updated.
 
lifepillar said:
To answer the OP, it seems to me that the proper way of upgrading requires mounting the skeleton on the base template and unmounting it after the upgrade. For example:


mount -t nullfs /usr/local/jails/templates/13.4-RELEASE-skeleton /usr/local/jails/templates/13.4-RELEASE-base/skeleton
freebsd-update -b /usr/local/jails/templates/13.4-RELEASE-base fetch install
umount /usr/local/jails/templates/13.4-RELEASE-base/skeleton
service jail restart


If this is correct, it would be nice if the handbook could be updated.
Click to expand...

Just wanted to thank you for taking the time to revisit this thread and post your solution. When I hit this back in 2023 I just rebuilt the few jails I had as thick jails so never had the issue again on that host. However I've revisited thinjails again with a new build, and sure enough when following the FreeBSD Handbook for thinjails you do still end up with "broken" jails that can't be updated cleanly with freebsd-update, as I found trying to go from 14.2-p2 to 14.2-p3 today. Some Googling lead me back to my own old forum post that I'd forgotten.

Mounting the skeleton temporarily did indeed work for freebsd-update to complete, however that also means the thinjails based off of the template still don't get updated certs (since the skeleton is cloned per jail, the jails built before this update have a skeleton that doesn't reflect the new cert changes).

Going to look into symlinking things as you and Emrion discussed.
 
I believe you need to run etcupdate inside the jails to keep the contents of /etc updated.

Bastille does this with its own “etcupdate” sub command.

Wondering if this isn’t the issue here.
 
Following the handbook to create thin jails, I come across the same problem.
Now, lifepillar's solution seems to be working fine, but there is still this problem of /etc files in the containers not being updated.
Could someone please shed some light on that topic, how to do it? I don't understand the symlinking solution (seems very complicated to symlink all the contents of /etc, or maybe I did not understand well?). As for etcupdate, it requires a /usr/src, if I am correct. I don't have this on my server. I could install it (maybe into the basejail?) and provide it to etcupdate to perform the update? Would that be a good solution? And then, fetching src.txz anew every time I want to binary patch the jails?
 
Bastille handles this by downloading and extracting src.txz, then creating a tarball from /usr/src.

This is then applied using the -t option along with -D to specify the jail directory.
 
As far as I understand, and I hope someone can correct this if I am wrong, but they're tied to the "base jail", or template, e.g. /usr/local/jails/templates/14.3-RELEASE-base and the matching skeleton. But their /etc is independent from the base jail (and from the host), so you can set them up differently, even if they share the same template. And as a consequence, after freebsd-update has finished its job, the /etc of the jails remain untouched, only the template version is updated.
 
No, freebsd-update cannot be run from inside a thin jail. It will fail, because a number of files or directories are read-only (please anyone correct me if I am wrong).
 
Alain De Vos said:
so you just enter into the jail and do a freebsd-update afterwards ?
Click to expand...
No, because base is read-only from the jails. You use directly freebsd-update from the host specifically directed to the jails base. For instance:
Code: 
freebsd-update -b $Jdir/base \
 -d $Jdir/base/var/db/freebsd-update -f $Jdir/base/etc/freebsd-update.conf \
 fetch install
Where $Jdir is the directory containing the jails base.

For an upgrade, in addition, you need to take care of each jail /etc with etcupdate.

If 15.0-RELEASE uses only pkgbase, I will need to rewrite all that part of my jails management. Idem for all jails management tools. :oops:
 
quamenzullo said:
As far as I understand, and I hope someone can correct this if I am wrong, but they're tied to the "base jail", or template, e.g. /usr/local/jails/templates/14.3-RELEASE-base and the matching skeleton. But their /etc is independent from the base jail (and from the host), so you can set them up differently, even if they share the same template. And as a consequence, after freebsd-update has finished its job, the /etc of the jails remain untouched, only the template version is updated.
Click to expand...
This is correct. /etc is not touched on an upgrade to a thin jail.
 
Emrion said:
If 15.0-RELEASE uses only pkgbase, I will need to rewrite all that part of my jails management. Idem for all jails management tools.
Click to expand...
I've been thinking about that too since PkgBase has been announced.
The releases 15 and 16 will be an interesting time to live, lot of changes for many of us :)
 
You must log in or register to reply here.
Back
Top