jails Understanding how jails for Linux work

I am in the process of learning how to use FreeBSD jails for my homelab. Getting FreeBSD to run is extremely straightforward after a little reading. Though I still have a ton of question for another day.

One that still is tripping me up is how Linux compatibility works with jails. I don't understand how the contents of /compat/linux is made into a several jails.

So lets say I have 2 machines ns01 and fs01 for example. How do they share /compat/linux? In this case would I still install my Linux userland into the ZFS dataset I have created for each jail (e.g. /usr/local/jails/[jail-name]?

I found this article on the FreeBSD wiki but I feel like its not telling me everything I need to know.
https://wiki.freebsd.org/LinuxJails

Can someone help me understand what is going on?
 
/compat/linux is NOT a jail, it's a filesystem overlay used to run Linux binaries directly on your system. This works because for any Linux binary accessing any file, the system first checks for that file below /compat/linux, and only if it doesn't exist there, the actual file (below /) is used instead.

The other way to use Linux binaries is inside a jail, for that, you just install some Linux userland in the jail's root.
 
Ah ok, so I can still treat the jail root as the linux userland root?
The wiki page references fstab mounts that need to be enabled on the host machine. How would this change if I am not using the /compat/linux path? Do I need mounts for each jail?
 
This works because for any Linux binary accessing any file, the system first checks for that file below /compat/linux, and only if it doesn't exist there, the actual file (below /) is used instead.
This used to be more or less hardcoded but nowadays can be changed with the compat.linux.emul_path sysctl(8).

Code:
root@molly:~ # sysctl -d compat.linux.emul_path
compat.linux.emul_path: Linux runtime environment path
 
Ok so let me if I get this right.

To make a linux jail;

make zfs dataset
bootstrap linux userland to jail root
make jail.conf config
add fstab mounts on host for each jail
boot jail

Done?
 
That seems correct at quick glance.

What some people do (and tools like EZJail or Bastille) is to create a so-called "base" jail. New jails are created and have various directories from the "base" jail symlinked to it. That keeps all new jails based on the same "base" install.

For example:
Code:
root@molly:~ # ll /usr/local/bastille/jails/ports/root/
total 246
drwxr-xr-x  18 root  wheel    22 May 21 22:19 .bastille/
-rw-r--r--   1 root  wheel  1023 Apr  9  2021 .cshrc
-rw-r--r--   1 root  wheel   507 Apr  9  2021 .profile
drwxr-xr-x   2 root  wheel     2 Apr 24  2021 .template/
-r--r--r--   1 root  wheel  6109 Apr  9  2021 COPYRIGHT
lrwxr-xr-x   1 root  wheel    14 Apr 24  2021 bin@ -> /.bastille/bin
lrwxr-xr-x   1 root  wheel    15 Apr 24  2021 boot@ -> /.bastille/boot
drwxr-xr-x   7 root  wheel     7 Mar  4  2021 data/
dr-xr-xr-x  11 root  wheel   512 Jun 15 22:46 dev/
drwxr-xr-x  27 root  wheel   108 Jun  1 13:35 etc/
lrwxr-xr-x   1 root  wheel     8 Apr 24  2021 home@ -> usr/home
lrwxr-xr-x   1 root  wheel    14 Apr 24  2021 lib@ -> /.bastille/lib
lrwxr-xr-x   1 root  wheel    18 Apr 24  2021 libexec@ -> /.bastille/libexec
drwxr-xr-x   2 root  wheel     2 Apr  9  2021 media/
drwxr-xr-x   2 root  wheel     2 Apr  9  2021 mnt/
drwxr-xr-x   2 root  wheel     2 Apr  9  2021 net/
dr-xr-xr-x   2 root  wheel     2 Apr  9  2021 proc/
lrwxr-xr-x   1 root  wheel    17 Apr 24  2021 rescue@ -> /.bastille/rescue
drwxr-x---   2 root  wheel     9 Jun  1 13:35 root/
lrwxr-xr-x   1 root  wheel    15 Apr 24  2021 sbin@ -> /.bastille/sbin
drwxrwxrwt   6 root  wheel     6 Jun 28 03:36 tmp/
drwxr-xr-x   6 root  wheel    15 Apr 24  2021 usr/
drwxr-xr-x  24 root  wheel    24 Jun 15 22:46 var/
The directory .bastille is the "base" jail that's nullfs(5) mounted:
Code:
/usr/local/bastille/releases/13.0-RELEASE on /usr/local/bastille/jails/ports/root/.bastille (nullfs, local, noatime, read-only, nfsv4acls)
 
More or less like that. With "booting" a Linux jail, you might run into issues though, e.g. when it tries to setup networking. Simple solution is *not* booting it but just starting whatever should run inside the jail manually (or, write your own scripts).

It's a special case if you want a VNET jail. These have their own network stacks, so the network must be setup inside the jail. It will never work with Linux networking tools, because the networking devices of FreeBSD work quite differently. My workaround for that is to copy FreeBSD tools (like ifconfig) from /rescue inside the Linux jail and use them. But that's probably advanced hackery, right now best start without VNET ;)
 
That seems correct at quick glance.

What some people do (and tools like EZJail or Bastille) is to create a so-called "base" jail. New jails are created and have various directories from the "base" jail symlinked to it. That keeps all new jails based on the same "base" install.

For example:
Code:
root@molly:~ # ll /usr/local/bastille/jails/ports/root/
total 246
drwxr-xr-x  18 root  wheel    22 May 21 22:19 .bastille/
-rw-r--r--   1 root  wheel  1023 Apr  9  2021 .cshrc
-rw-r--r--   1 root  wheel   507 Apr  9  2021 .profile
drwxr-xr-x   2 root  wheel     2 Apr 24  2021 .template/
-r--r--r--   1 root  wheel  6109 Apr  9  2021 COPYRIGHT
lrwxr-xr-x   1 root  wheel    14 Apr 24  2021 bin@ -> /.bastille/bin
lrwxr-xr-x   1 root  wheel    15 Apr 24  2021 boot@ -> /.bastille/boot
drwxr-xr-x   7 root  wheel     7 Mar  4  2021 data/
dr-xr-xr-x  11 root  wheel   512 Jun 15 22:46 dev/
drwxr-xr-x  27 root  wheel   108 Jun  1 13:35 etc/
lrwxr-xr-x   1 root  wheel     8 Apr 24  2021 home@ -> usr/home
lrwxr-xr-x   1 root  wheel    14 Apr 24  2021 lib@ -> /.bastille/lib
lrwxr-xr-x   1 root  wheel    18 Apr 24  2021 libexec@ -> /.bastille/libexec
drwxr-xr-x   2 root  wheel     2 Apr  9  2021 media/
drwxr-xr-x   2 root  wheel     2 Apr  9  2021 mnt/
drwxr-xr-x   2 root  wheel     2 Apr  9  2021 net/
dr-xr-xr-x   2 root  wheel     2 Apr  9  2021 proc/
lrwxr-xr-x   1 root  wheel    17 Apr 24  2021 rescue@ -> /.bastille/rescue
drwxr-x---   2 root  wheel     9 Jun  1 13:35 root/
lrwxr-xr-x   1 root  wheel    15 Apr 24  2021 sbin@ -> /.bastille/sbin
drwxrwxrwt   6 root  wheel     6 Jun 28 03:36 tmp/
drwxr-xr-x   6 root  wheel    15 Apr 24  2021 usr/
drwxr-xr-x  24 root  wheel    24 Jun 15 22:46 var/
The directory .bastille is the "base" jail that's nullfs(5) mounted:
Code:
/usr/local/bastille/releases/13.0-RELEASE on /usr/local/bastille/jails/ports/root/.bastille (nullfs, local, noatime, read-only, nfsv4acls)

Ok so the base jail holds the BSD image that all of the children are clones of? So when you patch the base jail all children are also patched as well?
I assume the same thing would work for a linux jail as well?
 
Ok so the base jail holds the BSD image that all of the children are clones of? So when you patch the base jail all children are also patched as well?
I assume the same thing would work for a linux jail as well?
Yes, correct.
 
Back
Top