Minimal pkgbase jails / chroots (Docker\OCI-like)

Hello everyone! After some time around playing with pkgbase, I've found a
way for making minimal OCI\Podman\Docker-like chroot environments where theres
only an app (could be many of them, though) and its dependencies inside a
chroot environment. No need for managing 500+MB bases or having custom minimal
builds of FreeBSD (although with some nullfs magic having a shared base
could actually sometimes be a better choice :)).

I believe this is the first real full tutorial how to make this work, and
if someone has more experience and knowledge about stuff in this post -- I
would really welcome the feedback, that would be net positive for everyone.

The real requirements for you would be to have FreeBSD-base repository
enabled for use with pkg(8) and knowing how to mount and unmount filesystems.

As an example, i will create a chroot/jail environment
with editors/neovim.

Short answer to our problems​


In order for pkg -r $directory to work properly, you need to ensure
two things are present in the target directory at the time of running pkg(8):

  • /usr/share/keys/pkg/trusted/pkg.freebsd.org.2013102301 -- just copy/nullfs mount the thing
  • /var/db/pkg/repos -- temporarily mounting as tmpfs recommended because this directory contains metadata cache

Install FreeBSD-runtime package and packages you need.

Dont forget to mount devfs and
run ldconfig -m /usr/local/lib inside
the new chroot.

Full tutorial​


We need to create the chroot directory, duh...

Code:
mkdir -p jail-nvim

In order for pkg to work properly, I chose to copy the repository keys and
use tmpfs for storing package manager metadata cache. You can
probably nullfs-mount the keys for that extra 4K of space savings.

Code:
# create the directory layout
mkdir -p jail-nvim/usr/share/keys/pkg/trusted
mkdir -p jail-nvim/var/db/pkg/repos

# copy the repository keys
cp -rn /usr/share/keys/pkg/trusted jail-nvim/usr/share/keys/pkg

# tmpfs mount the pkg metadata cache
mount -t tmpfs tmpfs jail-nvim/var/db/pkg/repos

Now we are free to utilize -r option with pkg. Note that in some
cases (as with vim-tiny port) FreeBSD-runtime package
is not needed (because vim(1)
just needs C libraries bundled with FreeBSD-clibs package, an
automatically installed dependency). FreeBSD-runtime package will usually be
pulled automatically as a dependency and it contains ldconfig utility
for fixing later problems. I'd recommend to explicitly install
'FreeBSD-runtime'.

Code:
pkg -r jail-nvim install FreeBSD-runtime neovim
pkg -r jail-nvim clean -a # we dont need cached packages and other stuff inside our little jail :)

If you are feeling extreme, you can delete all manpages from the
chroot environment that got pulled by packages (and yet we dont have
man utility installed).Note that updating packages in our
chroot will pull them back.

I'd write how much space it saved, but at the moment of editing this post, it
looks like FreeBSD-runtime package no longer supplies manpages
(it had some manpages for C libraries). Maybe it was all just a dream 🫥.

Code:
rm -rf /usr/share/man       # system manpages
rm -rf /usr/local/share/man # ports manpages

After originally testing this stuff on FreeBSD 15-ALPHA3, I noticed that
on FreeBSD 14.3 you have to manually run ldconfig inside the chroot.

Code:
chroot jail-nvim ldconfig -m /usr/local/lib

That's it! Now we need to unmount tmpfs stuff, effectively purging the
package manager metadata cache.

Code:
umount jail-nvim/var/db/pkg/repos


Maintenance and testing​

To update packages inside this minimal chroot we need to mount tmpfs like
inprevious steps and use the pkg commands we used, but this time
replacing install to upgrade. Don't forget to use
pkg clean :) .

We now can use this chroot directory with path option in
jail utility or jail.conf. Keep in mind that most software
(editors/neovim, in this case) really need devfs to be present.

Be warned that when you exit neovim, jail(8) utility will not
unmount devfs mounts because of some dark jail magic logic.

Code:
doas jail -c \
    path="$(realpath ./jail-nvim)" \
    mount.devfs \
    command=nvim

# now we umount it
umount $(realpath ./jail-nvim/dev)
 
you may find it interesting to look at the existing OCI container build scripts, like release/Makefile.oci and release/scripts/make-oci-image.sh. dch recently updated these to use pkgbase in 15.0.

If you are feeling extreme, you can delete all manpages from the
chroot environment that got pulled by packages
to prevent installing manpages, you can set the pkg(8) option FILES_IGNORE_GLOB=/usr/share/man/*,/usr/share/openssl/man/*. setting this in a jail-specific pkg.conf should ensure they don't come back on upgrades.
 
Thank you for clean instructions.

I was able to create a jail template with FreeBSD-set-minimal-jail. The result is 210M template vs 374M from base.txz (the template name follows the archive or the package set):
Code:
% zfs list -r zroot/jail/template | grep 15.0
zroot/jail/template/15.0-RELEASE        374M  1.75T   374M  /jail/template/15.0-RELEASE
zroot/jail/template/15.0-minimal        251M  1.75T   220M  /jail/template/15.0-minimal
zroot/jail/template/15.0-minimal-jail   210M  1.75T   179M  /jail/template/15.0-minimal-jail
The running jails install packages as needed.

There is one caveat I faced - the installation of FreeBSD-clang-15.0 from the jail for /usr/bin/cc. It triggers FreeBSD-clibs-lib32-15.0 but fails to run chflags on /libexec/ld-elf32.so.1:

I had to install the package from the hosting environment with pkg -r /jail/container/dev ....
 
Hello everyone! After some time around playing with pkgbase, I've found a way for making minimal OCI\Podman\Docker-like chroot environments where theres only an app (could be many of them, though) and its dependencies inside a chroot environment.

Haha nice! I managed to do the same thing today and thought I might post my findings here at the forum. But it seems it has already been done. But I'll add my procedure here since I'm using VNET and bastille as jail manager and It could help someone.

Code:
# bastille create -E empty
# pkg -r /usr/local/bastille/jails/empty/root install FreeBSD-clibs-15.0 (using a local repo that doesnt require keys)
# cp -Rp /usr/share/keys/ /usr/local/bastille/jails/empty/root/usr/share/keys
# pkg -r /usr/local/bastille/jails/empty/root install nginx
# chroot /usr/local/bastille/jails/empty/root/ ldconfig -m /usr/local/lib

Then I need to configure my jail.conf manually:
Code:
empty {
  host.hostname = empty;
  mount.fstab = /usr/local/bastille/jails/empty/fstab;
  mount.devfs;
  path = /usr/local/bastille/jails/empty/root;

  exec.start = "/sbin/ifconfig e0b_empty 172.25.0.77/24 up && route add default 172.25.0.1 && /usr/local/sbin/nginx";
  exec.stop = "";

  vnet;
  vnet.interface = e0b_empty;
  exec.prestart += "epair0=\$(ifconfig epair create) && ifconfig \${epair0} up name e0a_empty && ifconfig \${epair0%a}b up name e0b_empty";
  exec.prestart += "ifconfig vm-trusted addm e0a_empty";
  exec.prestart += "ifconfig e0a_empty description \"vnet0 host interface for Bastille jail empty\"";
  exec.poststop += "ifconfig e0a_empty destroy";
}

Then I start my new jail:
Code:
# bastille start empty

[empty]:
e0a_empty
e0b_empty
empty: created
add net default: gateway 172.25.0.1

and verify that nginx is running inside the jail:
Code:
# telnet 172.25.0.77 80
Trying 172.25.0.77...
Connected to 172.25.0.77.
Escape character is '^]'.
 
Back
Top