jails infrastructure as code and orchestration tools : ansible vs iocage vs (appjail + director + overlord)

toddg

toddg

I am currently developing a commercial application. For a variety of reasons,
I've decided to not use languages/runtimes/hosting services that I've used in
the past, e.g. pulumi + terraform deploying python/java/go microservices to
ubuntu server instances running on aws (docker/fargate, etc.).


This time, I am using languages/runtimes/hosting services that I want to
use, e.g. gleam + erlang deploying to jails on FreeBSD host instances running on hetzner
or similar providers.


For context, I'm relatively new to FreeBSD. I have read MW Lucas' books such
as: Absolute OpenBSD, Absolute FreeBSD, and I am currently reading FreeBSD
Mastery : Jails. After that, I have his books on network administration and ZFS
queued up...


My questions revolve around how to deploy and manage my systems. I plan
to start with a single server, and use jails to partition my
environments (dev, stage, prod). Each environment will have these services
running: a webserver, a database (postgres), and a backend. I plan to put each
service in it's own jail, and only allow specific traffic between the jails,
e.g. the webserver can access the backend, the backend can access the
database. I'll only allow external traffic over https to the webserver.

From the start, I'll need to be able to easily and quickly deploy new code and
database schemas, run tests, etc.

As my customer base grows, I'll need to be able to easily grow my tech stack,
migrate to a larger hosts, add warm backup hosts in different datacenters, etc.

Clearly I don't want to manually configure any of this. I want to use standard dev/ops
practices and use infrastructure as code. I know how to do this in other realms,
but not on FreeBSD...

I've heard of folks using the following for iac and deployment orchestration in
FreeBSD:

* bsddeploy : https://github.com/ployground/bsdploy
(mentioned in MW Luca's Jails book)
* ansible jexec plugin : https://blog.hofstede.it/managing-freebsd-jails-with-ansible-the-jailexec-connection-plugin/
* ansible tutorial : https://eoli3n.github.io/2021/06/08/jails-part-1.html
* iocage module for Ansible : https://github.com/vbotka/ansible-iocage
* ansible tutorial : https://tudorr.ro/blog/ansible-service-in-jail/
* install ansible on freebsd : https://www.adminbyaccident.com/freebsd/how-to-freebsd/how-to-install-ansible-on-freebsd/
* gread reddit comment on this topic: https://www.reddit.com/r/freebsd/comments/ic02up/help_me_better_understand_jail_networking/
* iocage : https://iocage.readthedocs.io/en/latest/
* vnet presentation slides : https://freebsdfoundation.org/wp-content/uploads/2020/03/Jail-vnet-by-Examples.pdf

So I was thinking, sure Ansible is the "way to go"... but then I stumbled
accross DtxdF 's AppJail:

* https://github.com/DtxdF/AppJail

And then there's more potentially cool stuff that takes this from docker to
docker compose or docker swarm like functionality:

* https://github.com/DtxdF/director
* https://github.com/DtxdF/overlord/wiki

So my questions are:

* What are you using for IAC and deployment orchestration?
* What have your experiences been with Ansible|AppJail|something else?
* What would you recommend in my situation?
(I'm giving myself a month to try and fully understand this space and get my first live FreeBSD host up and running)
* Would you run all 3 environments on the same host, or is that going to give me heartache in weird ways
(e.g. database's somehow conflicting?)

Thank you in advance!
 
You might want to get yourself a copy of the book Practical Internet Server Configuration by Robert La Lau. It's tag line is 'Learn to Build a Fully Functional and Well-Secured Enterprise Class Internet Server'. It does this with examples in FreeBSD and Linux. This means that you can compare the Linux instructions that you're already familiar with and the FreeBSD instructions that you're not familiar with to achieve what you want. I've read the book and can highly recommend it. As I recall, it doesn't do all of what you want, but it will bootstrap you into getting web and database servers up and running using jails which you can then use to further explore the bits that you want to automate. Best of luck with the project!
 
fraxamo thank you! That's a great recommendation. I'll give that a read.

My primary questions are more about devops tooling... ansible vs puppet vs chef vs appjail + director + overlord vs custom scripts etc.
 
Hi toddg,

I don't consider Overlord and Ansible to be competing tools. In fact, I use both.

Before creating Overlord, I only had an array of machines distributed in different locations around my country. I used SSH and then deploy with Director, but as I had more and more machines, this didn't scales so well. Furthermore, the problem with an orchestration tool is not only deployment, but also extracting information about your jails. That's the question.

I had thought about Ansible before creating Overlord, and even bsdploy, since it uses ezjail under the hood, it shouldn't be difficult to adapt to use AppJail, but the problem is that I wanted to use Director to deploy jails because it doesn't just deploy a single jail, it also deploys a group of jails (yes, this is inspired by docker-compose), and it also automates some interesting things like AppJail volumes, etc. In addition, I wanted to deploy virtual machines...

So I created Overlord and now I use it to deploy jails on all my machines. And, as I mentioned, this does not replace Ansible: I use Ansible to create the base directory of thinjails, update them, configure AppJail and configure Overlord to adapt them to my environment, and everything related to the host.

Currently, I deploy both jails and virtual machines with GitOps. For virtual machines, I have implemented the ephemeral concept as described in this article.

My recommendation is that you first familiarize yourself with FreeBSD, then with vanilla jails, and finally with the jail management of your choice. Follow that order and don't forget to learn about networking, especially related to jails.

If you choose AppJail, I recommend evaluating whether you really need something like Overlord. Perhaps Director (and therefore AppJail) alone will suffice for your use case.

toddg said:
Would you run all 3 environments on the same host, or is that going to give me heartache in weird ways
(e.g. database's somehow conflicting?)
Click to expand...

If you use VNET(9), there can be no conflicts, since each jail uses its own network stack.
 
And, as I mentioned, this does not replace Ansible: I use Ansible to create the base directory of thinjails, update them, configure AppJail and configure Overlord to adapt them to my environment, and everything related to the host.

DtxdF do you have any examples of how you use Ansible? I've spent a few days studying Ansible, and I am at a point where I can install packages on remote servers. However, when I look at the FreeBSD ansible plugins, to my untrained eye, they seem complicated. For each of my FreeBSD servers (jail hosts) I'd like to:

* create a firewall with pf
* install prometheus and node-exporter (see https://cloudspinx.com/install-prometheus-with-node-exporter-and-grafana-on-freebsd/)
* install whatever I need to get my jails up and running
* per MWLucas' recommendations, I'll try not to install anything in the host os, and push stuff into the jails as much as possible

Q: do you use vbotka's ansible freebsd plugins? (e.g. https://github.com/vbotka/ansible-freebsd-pf)
Q: if you don't use vbotka's ansible plugins, what do you use?
Q: do you have links to examples that you can share of ansible + freebsd?

I looked on the forum (and online) and I don't see many (any) examples of managing FreeBSD via ansible... only mentions that this is a "good idea".

Here's my example of progress to-date. I'll keep adding to this as I learn how to configure FreeBSD from ansible:

https://github.com/ToddG/freebsd-sa...ble-01-basic-install/ansible/01-basic-install
 
toddg said:
Clearly I don't want to manually configure any of this
Click to expand...
Meh, I would, and already did it a bit :p (wildest set-up had 4 metal servers, distcc on 1 with my workstation, shared DB on 1 with webserver and game servers on separate machines, and the DB machine was also a NAS; even had an arcade machine same-room and rack copying scores to the NAS and prior it was a HTPC/media room streaming from the NAS :p)


I'm not sure how standard dev-ops does it, but I work off templates I made, and expand em (new tweaks/etc learned over-time). Like WordPress; my template is how the notes and command boxes are laid out. Copy/paste sets it up, and the template works to Linux. If I want to host a new site on similar nginx/MySQL stack, I can copy the WP page, change some words, and have it up in maybe 10 mins (MyBB forums was that easy :p)

My template style can easily convert to Ansible/etc, but the time to understand how other apps work to automate my stuff confidently could be better spent making more generic templates I can understandably throw-down easy any OS :p Jails sound easy to template :D
 
toddg

toddg said:
DtxdF do you have any examples of how you use Ansible? I've spent a few days studying Ansible, and I am at a point where I can install packages on remote servers. However, when I look at the FreeBSD ansible plugins, to my untrained eye, they seem complicated. For each of my FreeBSD servers (jail hosts) I'd like to:

* create a firewall with pf
* install prometheus and node-exporter (see https://cloudspinx.com/install-prometheus-with-node-exporter-and-grafana-on-freebsd/)
* install whatever I need to get my jails up and running
* per MWLucas' recommendations, I'll try not to install anything in the host os, and push stuff into the jails as much as possible
Click to expand...

Unfortunately, no. All my Ansible-related files are private. I recommend that you consult the documentation for the specific module that solves your problem, but I also think you should read up on the basics, such as variables, templates, files, playbooks, tasks, plays, etc. The good news is that Ansible offers excellent support for FreeBSD:

* https://docs.ansible.com/projects/ansible/latest/collections/community/general/sysrc_module.html
* https://docs.ansible.com/projects/ansible/latest/collections/community/general/pkgng_module.html

toddg said:
Q: do you use vbotka's ansible freebsd plugins? (e.g. https://github.com/vbotka/ansible-freebsd-pf)
Q: if you don't use vbotka's ansible plugins, what do you use?
Click to expand...

No, I don't use modules other than the official ones, since, at least in FreeBSD, most things can be configured in simple files, and most Ansible roles implement many things that I'm not going to use. For example, to configure pf, you just need to copy a pf.conf(5) file to /etc/pf.conf, enable the rc script with pf_enable="YES", and start or reload it. See this snippet from I playbook of mine:

Code: 
    - name: Configure PF
      ansible.builtin.template:
        src: '{{ pf_conf }}'
        dest: /etc/pf.conf
        owner: root
        group: wheel
    - name: Enable PF
      community.general.sysrc:
        name: pf_enable
        value: 'YES'
    - name: Start PF
      register: pf_status
      ansible.builtin.service:
        name: pf
        state: started
      retries: 2
      delay: 5
    - name: Reload PF
      when: not pf_status.changed
      ansible.builtin.service:
        name: pf
        state: reloaded
      retries: 2
      delay: 5

toddg said:
Q: do you have links to examples that you can share of ansible + freebsd?
Click to expand...
No, but maybe the following examples might help you a bit:

Installing backrest

playbooks/backrest.yml:

Code: 
---
- name: Backrest
  hosts: all
  tasks:
    - name: Install
      community.general.pkgng:
        name: backrest
        state: latest
    - name: Set prefix
      ansible.builtin.replace:
        path: /usr/local/etc/backrest/config.env
        regexp: /var/db/backrest
        replace: '{{ backrest_prefix }}'
    - name: Set Port
      ansible.builtin.lineinfile:
        path: /usr/local/etc/backrest/config.env
        line: 'BACKREST_PORT={{ backrest_port }}'
    - name: Set User
      community.general.sysrc:
        name: backrest_runas
        value: root
    - name: Enable
      community.general.sysrc:
        name: backrest_enable
        value: 'YES'
    - name: Start
      ansible.builtin.service:
        name: backrest
        state: started

Console:

Code: 
$ rg --pcre2 '^backrest_(prefix|port): '
group_vars/all.yml
31:backrest_prefix: /var/db/backrest
32:backrest_port: '127.0.0.1:9898'

host_vars/cicd.yml
9:backrest_prefix: /zdata/backrest
14:backrest_port: '0.0.0.0:9898'

---

Deploy a jail using Ansible and AppJail Director
playbooks/hello.yml:
Code: 
---
- name: Deploy "hello" jail using AppJail Director
  hosts: controller
  tasks:
    - name: Create a temporary directory
      ansible.builtin.tempfile:
        state: directory
        suffix: appjail-director.
      register: temp_dir
    - name: Create Director file
      ansible.builtin.template:
        src: 'director/hello/appjail-director.yml'
        dest: '{{ temp_dir.path }}/appjail-director.yml'
    - name: Create Makejail file
      ansible.builtin.template:
        src: 'director/hello/Makejail'
        dest: '{{ temp_dir.path }}/Makejail'
    - ansible.builtin.shell:
        cmd: 'cd {{ temp_dir.path }} && env DIRECTOR_PROJECT=hello appjail-director up'
playbooks/templates/director/hello/Makejail:
Code: 
ARG hello_port=8888
ARG hello_text=Hello, world!

OPTION start
OPTION overwrite=force

PKG darkhttpd

CMD echo "${hello_text}" > /usr/local/www/darkhttpd/index.html

SYSRC "darkhttpd_flags=--uid darkhttpd --gid darkhttpd --port ${hello_port}"
SYSRC darkhttpd_enable=YES

SERVICE darkhttpd start
playbooks/templates/director/hello/appjail-director.yml:
Code: 
services:
  hello:
    options:
      - alias:
      - ip4_inherit:
    arguments:
      - hello_port: '4884'

Console:

Code: 
$ ansible-playbook -vv --inventory inventory.yml playbooks/hello.yml
...
$ curl http://localhost:4884
Hello, world!

Keep in mind that Director is idempotent, but certain conditions must be met. One of them is that the Makejail's modification time does not change from the last one, and since we are using a temporary directory, it will be detected as new every time we perform a deployment. To avoid this, use a normal directory instead of a temporary directory.
 
Espionage724 said:
Meh, I would, and already did it a bit :p (wildest set-up had 4 metal servers, distcc on 1 with my workstation, shared DB on 1 with webserver and game servers on separate machines, and the DB machine was also a NAS; even had an arcade machine same-room and rack copying scores to the NAS and prior it was a HTPC/media room streaming from the NAS :p)


I'm not sure how standard dev-ops does it, but I work off templates I made, and expand em (new tweaks/etc learned over-time). Like WordPress; my template is how the notes and command boxes are laid out. Copy/paste sets it up, and the template works to Linux. If I want to host a new site on similar nginx/MySQL stack, I can copy the WP page, change some words, and have it up in maybe 10 mins (MyBB forums was that easy :p)

My template style can easily convert to Ansible/etc, but the time to understand how other apps work to automate my stuff confidently could be better spent making more generic templates I can understandably throw-down easy any OS :p Jails sound easy to template :D
Click to expand...
Espionage724 Thank you. I looked at your wiki templates. That's well documented. If my foray into Ansible doesn't work out, I'll try your strategy.
 
DtxdF said:
No, I don't use modules other than the official ones, since, at least in FreeBSD, most things can be configured in simple files, and most Ansible roles implement many things that I'm not going to use. For example, to configure pf, you just need to copy a pf.conf(5) file to /etc/pf.conf, enable the rc script with pf_enable="YES", and start or reload it.
Click to expand...
DtxdF that is a gold nugget and exactly what I was wondering. Ok, cool. I'll follow your lead and stick to official modules and use your simpler strategy.

DtxdF said:
Keep in mind that Director is idempotent, but certain conditions must be met. One of them is that the Makejail's modification time does not change from the last one, and since we are using a temporary directory, it will be detected as new every time we perform a deployment. To avoid this, use a normal directory instead of a temporary directory.
Click to expand...
DtxdF your examples all make sense to me, thank you!
 
You must log in or register to reply here.
Back
Top