jails Which type of jail for production use cases?

[JohnK]

The concept of 'ephemeral data' is an axiom. *thumbs up*.

I too agree with the concept of 'easier to recreate than fix'. *thumbs up*

I keep my jail configs in directories instead of using:
<program> -s up -w down -i left -t right -c 098 -h 123

I prefer to have my "wants" kept in a key/value type config file for each jail instead of calling a <program> with specific -s -w -i -t -c -h es which may or may not change depending on future requirements/needs. Not to mention, having configuration records ('wants') separate from actual implementation allows me to remember what was and wasn't set in each jail.

name=up
key=down
ip=123
foo=bar

The <program> should read that config file and be expected to do what's necessary. This is called Separation of Concerns (SoC); -i.e. why you typically create a library in programming. If the "how" creating a jail changes, I shouldn't have to change all my 'wants' to account for that. My configurations list what I 'want' in my jails not 'how' because the 'want' is irrelevant to the 'how'. You do not write a function to "calculate GCD" for every program you write.

I create thick jails, but I want to dive into trying out thin when I find time.

Upon a new release.
1. I download the latest userland.
2. Create a "base userland" -i.e., add in all the stuff I want in all jails like: dot.nexrc, dot.cshrc, sshd_config, whatever, etc..
3. Zip that up.
4. (re)create my jails from that 'base userland'.

 

[omardevonlittle]

4. (re)create my jails from that 'base userland'.

Another great idea. If we will be making thick jails, why not start at a base template with everything we will need and zip it. Make things reusable

Which jail manager do you use to read your key-value config files? Or did you create a script yourself?

 

[JohnK]

Which jail manager do you use to read your key-value config files? Or did you create a script yourself?

I created myself a tool to keep my jail configurations in key/value files. Currently, my tool reads the key/values and builds a jail using the handbook method but it can be a wrapper for almost any "jail manager" (my tool isn't about being a jail manager but as it stands there are zero dependencies to try it).

The examples I gave are a bit more complicated than you'd actually need in real life but they are for demonstration purposes (my actual configs I use are not as complicated as those) and should work with little configuration and/or give you something to build a test upon. Also, the git example is an emulation of my git server and may or may not make total sense.

NOTE:
1. I built my key/value to emulate the UCL config language jail.conf uses.
2. If you want to test my scripts, clone the repo and do a "doas make install". A man page will be installed so you can "man jcreate". A "doas make uninstall" will remove the tool.
3. If you do actually test my script, let me know if it works for you.

GitHub - JohnKaul/jcreate: Script to create a FreeBSD jail.conf style jail with setup script, mounts, copy-in abilities

Script to create a FreeBSD jail.conf style jail with setup script, mounts, copy-in abilities - JohnKaul/jcreate

github.com

To create a custom userland I do something like the following (replace with default commands/methods):

# jcreate /path/to/templates/base/base.conf
# doas service jail start base
# doas jexec base
... configure 'base' jail ...
# doas service jail stop base
# cd /usr/local/jails/media
# doas chflags -R 0 /usr/local/jails/containers/base
# doas tar -cJf userland.txz -C /usr/local/jail/containers/base/ .
# jdestroy base

 

[omardevonlittle]

However, I think the future of jails is thick jails with PkgBase proving to be a good approach for jails that waste little space.

Could you expand on this? I keep seeing that "PkgBase" will be a very important thing for FreeBSD. I haven't read much about it, but as far as I know, it unifies having updates to both software (ports) and the base system on one area, instead of having to use "pkg upgrade" for the third party packages and "freebsd-update" for the base system.

How would "PkgBase" improve thick jails?

 

[DtxdF]

Could you expand on this? I keep seeing that "PkgBase" will be a very important thing for FreeBSD. I haven't read much about it, but as far as I know, it unifies having updates to both software (ports) and the base system on one area, instead of having to use "pkg upgrade" for the third party packages and "freebsd-update" for the base system.

How would "PkgBase" improve thick jails?

For thick jails, upgrading/updating will be simple and much easier to debug than freebsd-update when things go wrong. The integration with the pkg-provides plugin will be a bonus, but what a bonus, as it can help create an image with the exact packages your jail (or VM?) will need, reducing the size, and reducing the storage and bandwidth overhead, although in most cases the application (and dependencies) intended to run inside the jail is the one that has the most weight on the size of the image.

See also:

* https://github.com/AppJail-makejails/pkgbase
* https://github.com/DtxdF/overlord/wiki/appconfig

 

[ollopa]

The idea of thin jails + recreation rather than fixing sounds good, but does it force you to upgrade everything at the same time and use common versions of dependencies?

When you have several services that have been running for a long time there are many things that can happen that make updating via thin jails, or even recreation, challenging. Sometimes they have a major version update that would break a recreation recipe, or they need to be built with custom options. Sometimes they stop being maintained and require different versions of web and database dependencies. The config file syntax or database structure may have changed and require manual migration.

Lots of things can go wrong and I don't like it when I have to update the base jail for one jail, but doing so breaks other jails reusing the base and requires me to fix them when I don't have time for it.

I think the isolation of thick jails can help with management of jails that are not easily recreated via automation, but I'm curious if there are counter-arguments or other perspectives.

 

[DtxdF]

The idea of thin jails + recreation rather than fixing sounds good, but does it force you to upgrade everything at the same time and use common versions of dependencies?

When you have several services that have been running for a long time there are many things that can happen that make updating via thin jails, or even recreation, challenging. Sometimes they have a major version update that would break a recreation recipe, or they need to be built with custom options. Sometimes they stop being maintained and require different versions of web and database dependencies. The config file syntax or database structure may have changed and require manual migration.

Lots of things can go wrong and I don't like it when I have to update the base jail for one jail, but doing so breaks other jails reusing the base and requires me to fix them when I don't have time for it.

I think the isolation of thick jails can help with management of jails that are not easily recreated via automation, but I'm curious if there are counter-arguments or other perspectives.

The described problem can still happen no matter if you use thin or thick jails and treat them as a cattle or a pet. If you upgrade software that has breaking changes like the ones you mention, i.e. its configuration file or even something more complex like updating the database schema, this still requires you to run some instructions after updating the software. Other projects may automate this step, although even when this is not the case, many projects tell users in the release notes what instructions they need to perform to complete the upgrade.

In addition to reading the release notes for the software we are upgrading, we can back up the data and deploy a new jail with that data to see if there are any further steps we need to perform. This is better than upgrading the jail used in production first as we can gain some confidence in this process. ZFS shines here due to snapshots, but for the types of jails I mentioned above (ephemeral + thin jails) this can be done much better than thick jails as only volumes (the data that needs to persist) are backed up.