Minimal pkgbase jails / chroots (Docker\OCI-like)

[bored2tears]

Hello everyone! After some time around playing with pkgbase, I've found a
way for making minimal OCI\Podman\Docker-like chroot environments where theres
only an app (could be many of them, though) and its dependencies inside a
chroot environment. No need for managing 500+MB bases or having custom minimal
builds of FreeBSD (although with some nullfs magic having a shared base
could actually sometimes be a better choice ).

I believe this is the first real full tutorial how to make this work, and
if someone has more experience and knowledge about stuff in this post -- I
would really welcome the feedback, that would be net positive for everyone.

The real requirements for you would be to have FreeBSD-base repository
enabled for use with pkg(8) and knowing how to mount and unmount filesystems.

As an example, i will create a chroot/jail environment
with editors/neovim.

Short answer to our problems​

In order for pkg -r $directory to work properly, you need to ensure
two things are present in the target directory at the time of running pkg(8):

• /usr/share/keys/pkg/trusted/pkg.freebsd.org.2013102301 -- just copy/nullfs mount the thing
• /var/db/pkg/repos -- temporarily mounting as tmpfs recommended because this directory contains metadata cache

Install FreeBSD-runtime package and packages you need.

Dont forget to mount devfs and
run ldconfig -m /usr/local/lib inside
the new chroot.

Full tutorial​

We need to create the chroot directory, duh...

mkdir -p jail-nvim

In order for pkg to work properly, I chose to copy the repository keys and
use tmpfs for storing package manager metadata cache. You can
probably nullfs-mount the keys for that extra 4K of space savings.

# create the directory layout
mkdir -p jail-nvim/usr/share/keys/pkg/trusted
mkdir -p jail-nvim/var/db/pkg/repos

# copy the repository keys
cp -rn /usr/share/keys/pkg/trusted jail-nvim/usr/share/keys/pkg

# tmpfs mount the pkg metadata cache
mount -t tmpfs tmpfs jail-nvim/var/db/pkg/repos

Now we are free to utilize -r option with pkg. Note that in some
cases (as with vim-tiny port) FreeBSD-runtime package
is not needed (because vim(1)
just needs C libraries bundled with FreeBSD-clibs package, an
automatically installed dependency). FreeBSD-runtime package will usually be
pulled automatically as a dependency and it contains ldconfig utility
for fixing later problems. I'd recommend to explicitly install
'FreeBSD-runtime'.

pkg -r jail-nvim install FreeBSD-runtime neovim
pkg -r jail-nvim clean -a # we dont need cached packages and other stuff inside our little jail :)

If you are feeling extreme, you can delete all manpages from the
chroot environment that got pulled by packages (and yet we dont have
man utility installed).Note that updating packages in our
chroot will pull them back.

I'd write how much space it saved, but at the moment of editing this post, it
looks like FreeBSD-runtime package no longer supplies manpages
(it had some manpages for C libraries). Maybe it was all just a dream .

rm -rf /usr/share/man       # system manpages
rm -rf /usr/local/share/man # ports manpages

After originally testing this stuff on FreeBSD 15-ALPHA3, I noticed that
on FreeBSD 14.3 you have to manually run ldconfig inside the chroot.

chroot jail-nvim ldconfig -m /usr/local/lib

That's it! Now we need to unmount tmpfs stuff, effectively purging the
package manager metadata cache.

umount jail-nvim/var/db/pkg/repos

Maintenance and testing​

To update packages inside this minimal chroot we need to mount tmpfs like
inprevious steps and use the pkg commands we used, but this time
replacing install to upgrade. Don't forget to use
pkg clean .

We now can use this chroot directory with path option in
jail utility or jail.conf. Keep in mind that most software
(editors/neovim, in this case) really need devfs to be present.

Be warned that when you exit neovim, jail(8) utility will not
unmount devfs mounts because of some dark jail magic logic.

doas jail -c \
    path="$(realpath ./jail-nvim)" \
    mount.devfs \
    command=nvim

# now we umount it
umount $(realpath ./jail-nvim/dev)

 

[anguisette]

you may find it interesting to look at the existing OCI container build scripts, like release/Makefile.oci and release/scripts/make-oci-image.sh. dch recently updated these to use pkgbase in 15.0.

If you are feeling extreme, you can delete all manpages from the
chroot environment that got pulled by packages

to prevent installing manpages, you can set the pkg(8) option FILES_IGNORE_GLOB=/usr/share/man/*,/usr/share/openssl/man/*. setting this in a jail-specific pkg.conf should ensure they don't come back on upgrades.