jails jail with overlays

symlink · Oct 19, 2020

Hello,

I've recently purchased a virtual private server, with the purpose of fooling around a bit, learning FreeBSD, and hosting some services.

The VPS is very small and cheap, so I'd like to use as little resources as possible.

I'd like to play a bit with jails, and I figured that the size of the base system, uncompressed, is around 800 megabytes, and that the files should basically be duplicated for every jail.

I'm aware (from the Handbook) about ezjail, but as I said I'd like to experiment a bit on my own, to understand the principles, so what I did is the following:

1. Unpacked base.txz in /usr/local/jail/base

2. Added the following lines to /etc/fstab:

Code:

  /usr/local/jail/base    /usr/local/jail/www     nullfs  ro      0       0
  /usr/local/jail/www_rw  /usr/local/jail/www     unionfs rw      0       0

3. Invoked mount -a to construct the overlay of filesystems

4. Configured /etc/jails.conf as follows:

Code:

  www {
          path = /usr/local/jail/www;
          mount.devfs;
          host.hostname = ...;
          ip4 = inherit;
          ip6 = inherit;
          exec.start = "/bin/sh /etc/rc";
          exec.stop = "/bin/sh /etc/rc.shutdown";
  }

5. Invoked service jail start www

The jail seems to work just fine, and as I would expect the modifications to the base image are all stored in the www_rw directory, leaving the base image untouched. So far so good.

The weird thing happens with the /dev filesystem, mounted into /usr/local/jail/www/dev.

The mounted filesystems look like this:

Code:

  # mount
  /dev/ufs/rootfs on / (ufs, local, journaled soft-updates)
  devfs on /dev (devfs, local, multilabel)
  /usr/local/jail/base on /usr/local/jail/www (nullfs, local, read-only)
  <above>:/usr/local/jail/www_rw on /usr/local/jail/www (unionfs, local)
  devfs on /usr/local/jail/www/dev (devfs, local, multilabel)

But this looks weird:

Code:

  # ls -l /usr/local/jail/www/dev
  total 0
  -rw-r--r--  1 root  wheel  0 Oct 19 19:59 null

And this is even more weird:

Code:

  # ls -l /usr/local/jail/www_rw/dev
  total 0
  -rw-r--r--  1 root  wheel  0 Oct 19 19:59 null

If I disable the jail ( service jail stop www) and mount devfs manually ( mount -t devfs devfs /usr/local/jail/www/dev), It seems to work as expected, that is I find all the devices of /dev, and there's no trace of it into the www_rw directory.

Also, if I do a regular jail, without this overlay magic:

Code:

# ls -l dev
total 1
dr-xr-xr-x  2 root  wheel   512 Oct 19 20:28 fd
crw-rw-rw-  1 root  wheel  0x12 Oct 19 20:28 null
dr-xr-xr-x  2 root  wheel   512 Oct 19 20:28 pts
crw-r--r--  1 root  wheel   0x8 Oct 19 20:26 random
lrwxr-xr-x  1 root  wheel     4 Oct 19 20:28 stderr -> fd/2
lrwxr-xr-x  1 root  wheel     4 Oct 19 20:28 stdin -> fd/0
lrwxr-xr-x  1 root  wheel     4 Oct 19 20:28 stdout -> fd/1
lrwxr-xr-x  1 root  wheel     6 Oct 19 20:28 urandom -> random
crw-rw-rw-  1 root  wheel  0x13 Oct 19 20:26 zero

Nothing particularly interesting seems to be reported in /var/log/.

Any idea on what's going on?

Thanks in advance,

- symlink