jails Which type of jail for production use cases?

omardevonlittle · Aug 10, 2024

Hello everyone,

I am learning the concept of jails and I am just testing things out using "Thick" jails (meaning an isolated complete userland for the jail). I have read the handbook and Absolute FreeBSD yet I am curious to find out which type of jails are you using in your environment, be it either your home network or work environments. In my case, I want to have jails for essential network services (DNS, DHCP, etc), migrate some of my home network services that are on Linux VMs to FreeBSD jails and then try out new things. I might migrate my linux services to Thick jails first and then see how it goes from there.

I see the following use cases and benefits

Thick jails: good for isolation yet heavy on maintenance and manual work.
Thin jails: seize the benefits of ZFS datasets, and set default configuration on the dataset. Every other jail clones the dataset and then adds the jails packages and configuration on its own dataset. Might pose security or dependency risks based on how you configure the base template
Nullfs jails: I read this on Absolute FreeBSD and it's wonderful that the base FreeBSD install is able to do this (which is like magic to me) but I really don't see a use case for this in a home network. Lots of space benefits but might be very hard to manage.

Networking is also hard to decide. Since I am just testing I am using the same network interface as the host. In a more production environment however I want to ideally separate everything, a dedicated interface for the host and another interface and possibly VLANs too for the jails.

In my case, I am interested in having a default configuration for a new jail so it can be configured later. An example is I want to add a public ssh key to the root user so it can be auto configured with Ansible. That means that for the new jail, the ssh service must be configured and also the public key needs to be added to the root user. Creating a template using a thin jail might work for this case, right?

Can anyone share their experiences on using jails and what did you learn on your preferred setup? Thanks!

fraxamo · Aug 11, 2024

You might want to take a look at Dan Langille's presentation at EuroBSDCon 2019. He lists pros and cons for both thick and thin jails. I've linked to his slides, but if you search you can probably find the whole presentation somewhere.

Emrion · Aug 11, 2024

fraxamo said:
You might want to take a look at Dan Langille's presentation at EuroBSDCon 2019. He lists pros and cons for both thick and thin jails. I've linked to his slides, but if you search you can probably find the whole presentation somewhere.

The principal disavantage of thin jail in this presentation is no more true now, using etcupdate. See: https://github.com/Emrion/kjail

Bucky · Aug 11, 2024

My experience. Don't use jail managers - use the standard toolset. Thick jails are for production environments. Thin jails are temporary for experimenting. VNET jails have more overhead (bridge/epair). SharedIP jails are good enough for most cases. Use zfs on the host. Use jail templates. I jail services that are public-internet-facing full-time (eg, email server). Use the pf firewall to control access to most everything.

I know nothing about Ansible.

omardevonlittle · Aug 11, 2024

Bucky said:
My experience. Don't use jail managers - use the standard toolset. Thick jails are for production environments. Thin jails are temporary for experimenting. VNET jails have more overhead (bridge/epair). SharedIP jails are good enough for most cases. Use zfs on the host. Use jail templates. I jail services that are public-internet-facing full-time (eg, email server). Use the pf firewall to control access to most everything.

I know nothing about Ansible.

I agree with using the standard toolset, it is already very strong.
What do you mean by using "SharedIP" jails? Is it just the standard setting of using the same interface as the host (but a dedicated IP address?

Bucky · Aug 11, 2024

degauss4 said:
I agree with using the standard toolset, it is already very strong.
What do you mean by using "SharedIP" jails? Is it just the standard setting of using the same interface as the host (but a dedicated IP address?

Yes. My server has two interfaces - one faces the WAN and the other faces the LAN. I put the sharedIPjail IP address on the LAN interface.

hasbeen · Aug 11, 2024

degauss4 said:
I agree with using the standard toolset, it is already very strong.
What do you mean by using "SharedIP" jails? Is it just the standard setting of using the same interface as the host (but a dedicated IP address?

You can use your LAN IP address space for you jail(s).
ea. 192.168.1.10 in your jail with 192.168.1.1 as the gateway LAN IP for your jail, which is also the GWIP of your host, if that's how your LAN IP spce is set up. You should get familiar with basic concepts of routing and firewall rules, or get dedicated routing/fw appliance.

USerID · Apr 9, 2025

I have just started diving into jails. While my eyes are still getting used to the routine, I will note that:
1. It is advisable to brush up on your English to get the most accurate linguistic nuances. National languages, unfortunately, are not suitable here. To study the complex aspects of jail administration, you only need the original language.
2. Even if it is hard to start, never switch to the Windows environment to "learn" containerization systems.
3. A good start is Chapter 17 + manuals. It is not necessary to memorize all the reference data that concerns the parameters of the jail utility (as an example). Reference information can (for starters) be left out.
4. You need to have perseverance and patience. It is better to take notes. I am fed up with constantly depending on the Internet.
5. A deep understanding of routing, OpenZFS, BE, PF, master the basic tools (zfs, zpool, ...).
6. Basic understanding of security aspects.
7. The 7th point is sacred. Everything should be extremely transparent and, if possible, SIMPLE.

ShelLuser · Apr 9, 2025

Keep in mind that the thread you're responding to is almost 1 year old, something tells me that they probably solved their issues

Still...

USerID said:
2. Even if it is hard to start, never switch to the Windows environment to "learn" containerization systems.

And why not? That argument makes very little sense to me.

USerID said:
5. A deep understanding of routing, OpenZFS, BE, PF, master the basic tools (zfs, zpool, ...).

This also makes very little sense. For starters, there are plenty of servers which use UFS in which cases ZFS is pretty much meaningless. Depending on the setup... you most likely don't need to touch any filesystems at all. You also make it sound as if PF is the only firewall, FreeBSD provides 3 in total. And ... why BE?

You really don't need any "deep" understanding here. Even setting up an alias for your NIC within the same network then having the jail use that would also work. No routing required, see also jail(8). You could even simply set up a crude userland by extracting base.txz and optionally kernel.txz into a specific directory, then use the jail command to spawn a new process right from there.

USerID said:
6. Basic understanding of security aspects.

Why? Many people use jails to be used as a pristine building envirnoment for ports, during which you can just as easily stick with 1 users (root) and leave it at that.

Don't get me wrong here: it's definitely a good thing to keep up on this, but that has more to do with generic Unix systems administration rather than jails in specific.

If anything I'd say that scripting and a basic understanding of config file structures (like /etc/jails.conf) is much more important. And instead of ZFS and/or UFS I'd say nullfs can become more important (also referring to mount_nullfs(8)), especially for ease of use.

vermaden · Apr 9, 2025

degauss4 said:
Can anyone share their experiences on using jails and what did you learn on your preferred setup?

Setup I usually use is described here [1].

There are also 'Single Process Jails' and 'VNET Jails' to complete the view.

Some details here:

[1] https://vermaden.wordpress.com/2023/06/28/freebsd-jails-containers/

omardevonlittle · Apr 9, 2025

ShelLuser said:
Keep in mind that the thread you're responding to is almost 1 year old, something tells me that they probably solved their issues

Haha, although this post is 1 year old, I am still active and open to learning! I have replied to posts that are over 5 years old and got answers back

vermaden said:
Setup I usually use is described here [1].

Thank you vermaden! I always enjoy reading your blogs!

Bucky · Apr 9, 2025

Bucky said:
Yes. My server has two interfaces - one faces the WAN and the other faces the LAN. I put the sharedIPjail IP address on the LAN interface.

I've since changed my setup and now put the sharedIPjails on the lo0 interface but not using the 127/8 address space. For example, 'ifconfig lo0' returns this:

Code:

lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet 127.0.0.1/8
        inet 10.145.xxx.yy/32
        inet6 ::1/128
        inet6 fe80::1%lo0/64 scopeid 0x4
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

where the 10.145... is the address of the jail. The pf firewall points to the jail IP.

Works splendidly.

vermaden · Apr 9, 2025

degauss4 said:
Thank you vermaden! I always enjoy reading your blogs!

Thanks

victort · Apr 10, 2025

vermaden I too enjoy your blog. Every Monday I await your “Valuable News” blog.

As to the jail question, I’m also curious. Thin jails use the “clone” function of zfs correct?
What is the difference though? Doesn’t the initial snapshot of the base release take up as much space as the release itself?

I’ll have to read up on zfs clone looks like…

vermaden · Apr 10, 2025

victort said:
vermaden I too enjoy your blog. Every Monday I await your “Valuable News” blog.

As to the jail question, I’m also curious. Thin jails use the “clone” function of zfs correct?
What is the difference though? Doesn’t the initial snapshot of the base release take up as much space as the release itself?

I’ll have to read up on zfs clone looks like…

Thanks

About 'Thin Jails' ... after You unpack BASE.txz to /jail/template it takes what extracted BASE.txz uses.

Now You create /jail/template@snapshot - that snapshot takes like some BYTES (almost zero space).

Next You create ZFS clone of that snapshot - that clone initially also takes almost zero.

Now every change you make to each clone takes space - but only things You added.

Hope that helps

USerID · Apr 11, 2025

ShelLuser said:
And ... why BE?

Lots of powerful tools. I once tried to learn Asterisk, installed (it was a long time ago) Elastix. I didn't learn anything useful. I just poked the mouse at the web interface and thought for several days about what meaning he put into his terms and phrases. Only after text configs I learned to quickly solve problems, quickly configure telephony and understand the ESSENCE of processes, and not the marketing buzzwords of each inventor.

vermaden said:
Now You create /jail/template@snapshot - that snapshot takes like some BYTES (almost zero space).

Next You create ZFS clone of that snapshot - that clone initially also takes almost zero.

Now every change you make to each clone takes space - but only things You added.

patmaddox · Apr 14, 2025

I use thick jails, because I construct the entire jail from scratch using poudriere. So it's just creating a zsnapshot type, send it to the appropriate place, edit the conf files and go.

I've considered using "thin" jails for experimental stuff, where I create one jail profile that has every package I use across all jails... then I have one dataset to clone, and edit the config files. I wouldn't use this for production because it creates a lot more surface area within a jail, but for my own experimentation it can be useful to have every tool I use inside of every jail. Haven't done it yet though, thick jails are fine by me so far.

DtxdF · Apr 14, 2025

I use thin jails for everything. Although thick jails are not a problem today because of the cheap space we can get, I prefer to stick with thin jails because of their quick creation and destruction. Thin jails are created faster on both fast and slow storage devices. However, what wastes the most space is the service you use inside the jail.

I'm highlighting this because I don't update/upgrade my jails using the classic approach of updating/upgrading each jail using freebsd-update(8). I just run freebsd-update(8) to update the base directory of the jails and re-create my jails. Of course, this cannot succeed unless you correctly separate the data that the applications use within the jail. In the case of the upgrade, I simply fetch the components but for the new version and this creates a new base directory with the new version of FreeBSD. After this, simply recreating the jails with the new version is sufficient.

This fits well for me because I prefer the “easier to recreate than to fix” philosophy. This is most evident when you modify a file inside the jail, but of course you are a human and your memory may fail in the future, so unless you track every file and fix them in case of an upgrade conflict, it may be better to just throw that file inside the jail after you have recreated it. Of course, using an automation tool, not by hand.

For more details on the above approach, see this document:

* appjail-ephemeral(7)

omardevonlittle · Apr 14, 2025

patmaddox said:
I use thick jails, because I construct the entire jail from scratch using poudriere

Aha, I've never heard of creating a jail from scratch using poudriere. Do you mean the packages inside the jail are built using poudriere or the jail itself?

DtxdF said:
I prefer to stick with thin jails because of their quick creation and destruction.

Sure, that's the main benefit from using ZFS and jails, but on "Absolute FreeBSD" from MWL, he said there might be some security issues (say for example, a human error or a malicious actor edits some files on the base directory). Would you say it's worth the tradeoff?

I see that wasting 500MBs for each thick jail for the userland is a bit wasteful (theoretically, ZFS should deduplicate it, but maybe if they are on different datasets they arent), plus the manual work of editing each file on each jail might be annoying.

DtxdF · Apr 14, 2025

degauss4 said:
Sure, that's the main benefit from using ZFS and jails

Even in UFS, the use of thin jails has that benefit.

degauss4 said:
he said there might be some security issues (say for example, a human error or a malicious actor edits some files on the base directory). Would you say it's worth the tradeoff?

Only if the base directory is mounted in read-write mode, but normally it is mounted in read-only mode. With a read-only base directory, processes (malicious or not) cannot modify those files, so they do not affect other jails using that same base directory. The only one who can modify the base directory is the administrator and only from the host with this approach.

degauss4 said:
I see that wasting 500MBs for each thick jail for the userland is a bit wasteful (theoretically, ZFS should deduplicate it, but maybe if they are on different datasets they arent), plus the manual work of editing each file on each jail might be annoying.

Yeah, of course, ~500 MiB is a waste, even more if you have several jails. However, probably many people prefer thick jails because of the feel of a real computer. However, I think the future of jails is thick jails with PkgBase proving to be a good approach for jails that waste little space.

sh:

# appjail quick \
      freebsd14 \
      from=quay.io/dougrabson/freebsd14-minimal \
      start \
      overwrite=force \
      virtualnet=":<random> default" \
      nat \
      template=/usr/local/share/examples/appjail/templates/freebsd-oci.conf
...
# appjail jail list -j freebsd14
STATUS  NAME       TYPE   VERSION      PORTS  NETWORK_IP4
UP      freebsd14  thick  14.1-STABLE  -      10.0.0.2
# appjail cmd local freebsd14 du -sh
 34M
# appjail quick jthin overwrite=force
...
# appjail jail list -j jthin
STATUS  NAME   TYPE  VERSION       PORTS  NETWORK_IP4
DOWN    jthin  thin  14.2-RELEASE  -      -
# appjail cmd local jthin du -sh
 5.9M
# appjail quick jthick overwrite=force type=thick
...
# appjail jail list -j jthick
STATUS  NAME    TYPE   VERSION       PORTS  NETWORK_IP4
DOWN    jthick  thick  14.2-RELEASE  -      -
# appjail cmd local jthick du -sh
454M    .

patmaddox · Apr 14, 2025

degauss4 said:
Do you mean the packages inside the jail are built using poudriere or the jail itself?

Both. poudriere-image(8) will construct an image - including base, kernel, and packages - in a number of different formats. Here are some examples:

DtxdF said:
Yeah, of course, ~500 MiB is a waste, even more if you have several jails.

One thing you can do with poudriere-devel, which I haven't done yet, is use a base snapshot when building other images (it's called an origin image). So you could for example make a snapshot with the utilities you have common to all images. Then custom images use the origin as a base, and create an incremental snapshot. Then for your thin jails you would close the base and receive the incremental. That's how it looks, anyway. Like I said, I haven't tried it yet.

omardevonlittle · Apr 15, 2025

DtxdF said:
Only if the base directory is mounted in read-write mode, but normally it is mounted in read-only mode.

So simple and so secure. For even more security, you could raise the secure levels to where the user can only change/remove flags in a single user boot

Would you recommend using "appjail"? I looked at a few commands for managing jails, but I've found a few drawbacks on each (or the tool being deprecated). I considered the idea of making my own shell scripts for managing jails in a predictable way.

patmaddox said:
Then for your thin jails you would close the base and receive the incremental

FreeBSD impresses me more every day

This is something great to investigate more, I will keep an eye on this option!

victort · Apr 15, 2025

I don’t want to put one jail manager over another, but BastillBSD has undergone some recent changes and improvements.

Appjail is great too.

Both are being actively maintained right now.

Appjail is good for automation and “docker-like” deployments of jails.

BastilleBSD is more just a wrapper around the standard jail tools included in BSD.

I suggest trying both and then choose which you prefer.

DtxdF · Apr 15, 2025

degauss4 said:
Would you recommend using "appjail"? I looked at a few commands for managing jails, but I've found a few drawbacks on each (or the tool being deprecated). I considered the idea of making my own shell scripts for managing jails in a predictable way.

AppJail will not become obsolete [:surprised face:], I update it from time to time and I usually work on other projects related to it. I use AppJail every day.

Of course, if you find any problem or want something to be improved, let me know.

Emrion · Apr 15, 2025

I use only thin jail (nullfs based) for one and only one reason: update and upgrade are way faster.
I left behind me the days where an upgrade of a handful of jails takes hours.