Using zfs clone with jails

dvl@ · May 3, 2013

I am setting up a new system to run several jails. All the jails will be very similar. In fact, the only differences will be minor (configuration files, user, etc).

With that in mind, I've created one jail with all the apps etc that all other jails will require. I'm looking at using ZFS clone to create the other jails.

FYI: I'm using ezjail-admin(1) and have been for years and would like to continue. But I'm open to suggestions.

My jails are rooted at system/usr/local/jails as you can see below.

Anyone already doing this and care to share? Thank you.

Code:

$ zfs list
NAME                                        USED  AVAIL  REFER  MOUNTPOINT
system                                     5.24G  10.6T   288K  /
system/root                                 336K  10.6T   336K  /root
system/rootfs                               984M  10.6T   983M  legacy
system/tmp                                  392K  10.6T   392K  /tmp
system/usr                                 4.27G  10.6T   288K  /usr
system/usr/home                            64.3M  10.6T   288K  /usr/home
system/usr/home/dan                        64.0M  10.6T  64.0M  /usr/home/dan
system/usr/local                           2.56G  10.6T   242M  /usr/local
system/usr/local/jails                     2.33G  10.6T   288K  /usr/local/jails
system/usr/local/jails/basejail            2.31G  10.6T  2.31G  /usr/local/jails/basejail
system/usr/local/jails/gus.unixathome.org  6.60M  10.6T  6.60M  /usr/local/jails/gus.unixathome.org
system/usr/local/jails/newjail             6.00M  10.6T  6.00M  /usr/local/jails/newjail
system/usr/local/pgsql                      288K  10.6T   288K  /usr/local/pgsql
system/usr/obj                             1.48M  10.6T  1.48M  /usr/obj
system/usr/ports                           1.65G  10.6T  1.55G  /usr/ports
system/usr/ports/distfiles                 96.2M  10.6T  96.2M  /usr/ports/distfiles
system/usr/src                              288K  10.6T   288K  /usr/src
system/var                                 1.27M  10.6T   288K  /var
system/var/audit                            288K  10.6T   288K  /var/audit
system/var/log                              424K  10.6T   424K  /var/log
system/var/tmp                              304K  10.6T   304K  /var/tmp

Beeblebrox · May 3, 2013

ezjail is great, but as you have discovered it has no support for zfs cloning.

I would suggest looking through the poudriere source code - as a concept poudriere creates clones for each build jail.

Not that poudriere would solve your issues, but poudriere+ezjail might and maybe the concept is not that difficult to incorporate into ezjail (with a little insistence on the developer/maintainer).

dvl@ · May 4, 2013

That sounds like a good idea. I cannot investigate that now, as BSDCan looms closer.

In the meantime, I tried something using ezjail and some manual steps. In short, I used ezjail-jail to create a new jail, moved the jail away, put a ZFS clone into its place, and started the jail.

Code:

$ zfs list | grep jails
system/usr/local/jails                            2.33G  10.6T   288K  /usr/local/jails
system/usr/local/jails/basejail                   2.31G  10.6T  2.31G  /usr/local/jails/basejail
system/usr/local/jails/gus.unixathome.org         7.38M  10.6T  6.94M  /usr/local/jails/gus.unixathome.org
system/usr/local/jails/lux.unixathome.org          895K  10.6T  6.92M  /usr/local/jails/lux.unixathome.org
system/usr/local/jails/lux.unixathome.org.EZJAIL  6.24M  10.6T  6.24M  /usr/local/jails/lux.unixathome.org.EZJAIL
system/usr/local/jails/newjail                    6.00M  10.6T  6.00M  /usr/local/jails/newjail

I will try reproducing this soon.

dvl@ · May 4, 2013

Cloning and ezjail

These are my ezjail configuration options:

Code:

ezjail_jaildir=/usr/local/jails
ezjail_use_zfs="YES"
ezjail_use_zfs_for_jails="YES"
ezjail_jailzfs="system/usr/local/jails"
ezjail_zfs_properties="-o compression=lzjb -o atime=off"

The first step is setting up a jail upon which all the other jails will be based. It contains the packages and configurations you want in each jail. Now create a snapshot of that jail.
# zfs snapshot system/usr/local/jails/gus.unixathome.org@BaseForOtherJails
In this case, gus.unixathome.org is the jail I set up.

Now add the IP address to the host system and use ezjail to create the new jail:
# ifconfig em0 alias 10.55.0.89 netmask 255.255.255.255
# ezjail-admin create -f bacula faraday.unixathome.org 10.55.0.89
NOTE: Bacula is the flavour I created when setting up the original jail, gus.

Now I want to replace the existing ZFS filesystem with a clone. First step, rename it. Just in case. Once I know what I'm doing, delete may be a better option here.
# zfs rename system/usr/local/jails/faraday.unixathome.org system/usr/local/jails/faraday.unixathome.org.EZJAIL

This filesystem is still mounted, and in the location we want to put a clone. Here, we unmount it, and give it a new mount point which will not conflict with the clone we want to correct. I do this just to avoid confusion should the filesystem ever get mounted.
# zfs umount system/usr/local/jails/faraday.unixathome.org.EZJAIL
# zfs set mountpoint=/usr/local/jails/faraday.unixathome.org.EZJAIL system/usr/local/jails/faraday.unixathome.org.EZJAIL

Now I clone the snapshot:
# zfs clone system/usr/local/jails/gus.unixathome.org@BaseForOtherJails system/usr/local/jails/faraday.unixathome.org

This is what we now have:

Code:

$ zfs list | grep jails
system/usr/local/jails                                2.34G  10.6T   288K  /usr/local/jails
system/usr/local/jails/basejail                       2.31G  10.6T  2.31G  /usr/local/jails/basejail
system/usr/local/jails/faraday.unixathome.org         16.0K  10.6T  6.67M  /usr/local/jails/faraday.unixathome.org
system/usr/local/jails/faraday.unixathome.org.EZJAIL  6.24M  10.6T  6.24M  /usr/local/jails/faraday.unixathome.org.EZJAIL
system/usr/local/jails/gus.unixathome.org             7.36M  10.6T  6.92M  /usr/local/jails/gus.unixathome.org
system/usr/local/jails/lux.unixathome.org              895K  10.6T  6.92M  /usr/local/jails/lux.unixathome.org
system/usr/local/jails/lux.unixathome.org.EZJAIL      6.24M  10.6T  6.24M  /usr/local/jails/lux.unixathome.org.EZJAIL
system/usr/local/jails/newjail                        6.00M  10.6T  6.00M  /usr/local/jails/newjail

As you can see, the cloned faraday uses only 16 KB. The ezjail faraday is 6 MB. Now let's start up this jail.

Code:

# /usr/local/etc/rc.d/ezjail start faraday.unixatlux.unixathome.org
Configuring jails:.
Starting jails: faraday.unixathome.org.

Here's a test I do, just for my own sanity, to see f I'm in the right jail:
# touch /usr/local/jails/faraday.unixathome.org/usr/home/FARAWAY

Now, ssh to the jail:

Code:

$ ssh -A faraday
The authenticity of host 'faraday.unixathome.org (10.55.0.89)' can't be established.
ECDSA key fingerprint is 70:1e:e8:40:88:e0:c2:96:e1:0a:af:69:7b:9e:76:59.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'faraday.unixathome.org' (ECDSA) to the list of known hosts.
Password:
Last login: Fri May  3 15:50:56 2013 from 10.55.0.16
FreeBSD 9.1-RELEASE (GENERIC) #0 r243825: Tue Dec  4 09:23:10 UTC 2012

Welcome to FreeBSD!

Before seeking technical support, please use the following resources:

o  Security advisories and updated errata information for all releases are
   at http://www.FreeBSD.org/releases/ - always consult the ERRATA section
   for your release first as it's updated frequently.

o  The Handbook and FAQ documents are at http://www.FreeBSD.org/ and,
   along with the mailing lists, can be searched by going to
   http://www.FreeBSD.org/search/.  If the doc package has been installed
   (or fetched via pkg_add -r lang-freebsd-doc, where lang is the
   2-letter language code, e.g. en), they are also available formatted
   in /usr/local/share/doc/freebsd.

If you still have a question or problem, please take the output of
`uname -a', along with any relevant error messages, and email it
as a question to the questions@FreeBSD.org mailing list.  If you are
unfamiliar with FreeBSD's directory layout, please refer to the hier(7)
manual page.  If you are not familiar with manual pages, type `man man'.

Edit /etc/motd to change this login announcement.

$ hostname
faraday.unixathome.org

Now, look in /usr/home:

Code:

$ ls -l /usr/home
total 17
-rw-r--r--  1 root  wheel   0 May  4 22:15 FARAWAY
drwxr-xr-x  3 dan   dan    14 May  3 15:50 dan
$

Yes, no doubt about it, this is the right jail.

You might say: why am I worried about saving 5 MB (or more) on a 16 TB system.

I don't know.

Savagedlight · May 5, 2013

How does this play out when upgrading the jails? Will the similar changes (read: binary updates) match up and not be stored in duplicate?

dvl@ · May 5, 2013

Savagedlight said:
How does this play out when upgrading the jails? Will the similar changes (read: binary updates) match up and not be stored in duplicate?

That's precisely why I think I won't use this in future. Upgrades affect only the base jail, and any jails based upon it. Clones would not be upgraded.

In this case, clones are attempting to do what flavours do. I think I should be using flavours instead of clones. I do not see any advantage in using clones when all I'm saving is a few MB on a multi-terrabtye system at the expense of ease-of-upgrading.

Captain Mooboy · Jun 30, 2023

Since this thread was a top search result for using zfs clone to create an ezjail, I'll note the elements necessary to incorporate the clone into the ezjail configuration (i.e., without using ezjail-admin to create a new jail and moving the jail away).

zfs snapshot, clone
ifconfig an IP for the jail
Create /usr/local/etc/ezjail/jail_config_name (copy from cloned jail)
1. Change all the export statements left of the equal signs
2. Change hostname, IP, rootdir
Create /etc/fstab.jail_config_name (copy from cloned jail and modify)
If necessary, nullfs mount of basejail: mount -t nullfs -o ro /usr/jails/basejail /usr/jails/jail_config_name/basejail
jail /etc/rc.conf: update hostname, maybe stop some services

Using zfs clone with jails

dvl@

Beeblebrox

dvl@

dvl@

Savagedlight

dvl@

Captain Mooboy