jails infrastructure as code and orchestration tools : ansible vs iocage vs (appjail + director + overlord)

toddg said:
I thought I'd recap the past few months. I'm beyond embarrassed to say that I'm still working on deployments. So far I have written, oh about 4, different deployment systems...one in Ansible, one using JohnK's drist tool, and several more written from scratch in gleam. All of them worked until I fiddled with them until they broke, or I decided I didn't like them, or I accidentally deleted the code (yup). Anyway, I have now spent 100x the time on this problem than is reasonable. I know a bit more about jails and networking, but I'm not an expert in anything. I'm thinking of coming back, full circle, and am looking at bastille.

Here's my path. Feel free to comment at any level. I'm a one person startup, trying to build a SaaS app. My thinking goes like this:

1. spend as little money as possible. don't risk overspend by using AWS/GCP/Azure. instead, rent bare metal (gthost) and use freebsd + jails
2. only rent one server, and have it dns point to it. this server then uses caddy to spread traffic for dev/stage/prod to dev/stage/prod jails
3. define a cidr schema to segregate traffic by environments, e.g. 10.0 -> global things like caddy, 10.1 -> dev, 10.2 -> stage, 10.3 -> prod
4. further refine the cidr schema to separate apps, such that 10.0.0 -> Caddy, 10.0.1 -> nginx, 10.1.0 -> dev webserver, 10.1.1 -> dev database
5. finally, use the last cidr set for blue/green deployments, such that 10.1.0.1/25 -> blue dev webserver, 10.1.0.128/25 -> green dev webserver

Note: I have other reasons for choosing freebsd than price. I'm a supporter of FOSS, EFF, and freedom in general. I think large corps are dysfunctional and a problematic.

Anyway, I actually had all of this working several times. Sigh.

One of the issues with the above setup is that I had it all working in an 10./8 space. Just one large network/broadcast segment. So I had templates to manage pf.conf for managing allowed traffic. With macros and whatnot, it's manageable, but pf.conf get's pretty big. However, the crucial failing of my approach is that every change to pf.conf has the potential to bring the network down, and the blast radius is global/dev/stage and prod environments.

So then I thought, separate bridges! That's the way to separate traffic. Somehow I didn't get that working.
So then I thought, VLANs! That's the way to separate traffic, and again, somehow I didn't get that working either.
So then I thought, vlanfilter! That's the way to separate traffic, and again, somehow I didn't get that working either.
So then I thought, hierarchical jails! That's the way to limit the blast radius and make the pf.conf files easier to deal with.

Currently, I don't have anything fully working. And I need to pivot from this madness and just build the damn apps, get some customers, and then revisit all of this stuff when its actually relevant b/c I need to scale.

* Do I need my custom fancy jail deployment system that is like docker for freebsd? No.
* Do I need dev/stage/prod right now? No.
* Do I need hierarchical jails (for dev/stage/prod)? No.
* Do I need bridges, vlans, vlanfilter? I dunno, ...Probably a bridge...
* Do I need blue/green deployments...clearly No.
* Do I need bastille or any other system? I dunno, I think it's easier to use a tool than the bare jails tools, and getting ifconfig stuff right can be tricky sometimes. For instance, I never got thinjails working, so I defaulted back to thickjails.

One of the reasons I wrote my own system was so that I could generate templated pf.conf files that allowed traffic like this:
1. Allow traffic from the caddy loadbalancers to the jail web apps. This statement affects these files:
* pf.conf
* Caddyfile

2. Allow traffic from (dev|stage|prod) web jails over port X to database (dev|stage|prod) jails. This statement affects these files:
* pf.conf
* Caddyfile
* postgres conf file (iirc hba.conf)
* database ip address and password are injected into an environment file that the webapps use to auth to the database

I think that switching to vlans and/or vlanfilter would mean pf could be configured with vlan ids rather than individual jail ip addresses. That would definitely be much more manageable than what I've been doing.

So my questions for anyone reading this thread are:

Questions
1. What do you use to manage your jails and application deployments?
2. In the case of something like bastille, how manage it for repeatability? Shell scripts? Ansible? Nothing and write commands in the terminal each time?

Thanks in advance for any input/advice y'all might have.
Click to expand...
I'd begin by pointing out that the the comments you make about your failed experiments are too vague for anyone to offer much help. You briefly mention some approaches you attempted and technologies you tried, but then don't offer any details about what exactly you did in each case, what outcomes you expected, and what occurred instead, and without that information is pretty hard to even attempt to fix anything.

I am successfully:
  • Using vm-bhyve to manage my VMs.
  • Which I then configure via non-trivial Ansible playbooks that exercise quite a few roles (node bootstrapping, system tuning, networking, monitoring, services, etc.).
  • Deploying jailed services (web & database servers, GitLab & GitLab runners, Uptime Kuma, Grafana dashboards, QA & prod environments for each of those, etc.) to those VMs via Bastille templates and some minor scripting glue (which will probably become its own Ansible role as well in due time).
  • Distributing all of those workloads across a number of bridges that live on the host(s) of those VMs, each bridge assigned to a purpose-specific VLAN (e.g. QA, prod, DMZ, management, etc).
  • Routing and filtering traffic to/across those VLANs via a pfSense router that sits in front of it all (pfSense also takes care of, of course, DHCP, DNS, ACME, etc.).
Moreover, I'm currently working on a massive Ansible playbook and some new roles to migrate a far from trivial TrueNAS CORE setup and all the services I run on it to vanilla FreeBSD, though admittedly that's progressing at a rather slow pace because it's *quite* a bit of work and also, at times, complex.

So, in any case, I'm pretty confident in saying that what you're attempting is indeed possible, but without much info on what went wrong in your experiments its not easy finding the errors and suggesting fixes.

Hope that helps!
 
toddg *nod* Thanks.

I use my own jail making script which has the ability to launch a script in the jail and on the host (which no jail manager allows you to do as far as I know); I pushed a copy to github if you'd want to try but it won't solve all of your problems. It was a different approach but no one really understood (or cared about) my concept--and I stopped pushing to github because of Ai, lack of participation/understaning, and/or more likely "I aint smart 'nuff"-. I couldn't even get anyone to talk to me let alone "install my tool". -e.g. AppJail and CBSD didn't want to talk. Someone from Bastile told me they didn't have make(1). etc.

I know your problem though--a "setup" needs to be dynamic and almost "JIT" so there is no "one setup" that seems to work (templates only take you sofar). If I have two servers, and wanted a "plex media server" on both of them, one single template will work if the servers are in the same location but if one has different external mount points, for example, this is when you need two templates.

My personal jail making script was essentially intended to be a wrapper to a jail making tool (pot, cbsd, bastile, appjail, base system, etc). -i.e. The "big idea" behind my script was the ability to keep the configs in a directory and a config file. However, I too ran into road blocks so I made other tools like "sysconf" (jail/host/config mgmt) and my fork of "drist" (deployment) but I'm now rebuilding my jail making script in C in a more flexible "hook system" methodology for myself but unfortunately I'm not trying to solve the 'JIT problem' with my redesign because I've sort of resigned to having two templates, so to speak.

My new system is designed much like a bottom-up design you'd do in Lisp (see: structure and intreptation of computer programs) but in C.

You may also want to check out BSD POT as a jail manager. Their templates are setup more docker like. I believe AppJail also has something along these lines too.
 
jmpalacios said:
I'd begin by pointing out that the the comments you make about your failed experiments are too vague for anyone to offer much help. You briefly mention some approaches you attempted and technologies you tried, but then don't offer any details about what exactly you did in each case, what outcomes you expected, and what occurred instead, and without that information is pretty hard to even attempt to fix anything.

I am successfully:
  • Using vm-bhyve to manage my VMs.
  • Which I then configure via non-trivial Ansible playbooks that exercise quite a few roles (node bootstrapping, system tuning, networking, monitoring, services, etc.).
  • Deploying jailed services (web & database servers, GitLab & GitLab runners, Uptime Kuma, Grafana dashboards, QA & prod environments for each of those, etc.) to those VMs via Bastille templates and some minor scripting glue (which will probably become its own Ansible role as well in due time).
  • Distributing all of those workloads across a number of bridges that live on the host(s) of those VMs, each bridge assigned to a purpose-specific VLAN (e.g. QA, prod, DMZ, management, etc).
  • Routing and filtering traffic to/across those VLANs via a pfSense router that sits in front of it all (pfSense also takes care of, of course, DHCP, DNS, ACME, etc.).
Moreover, I'm currently working on a massive Ansible playbook and some new roles to migrate a far from trivial TrueNAS CORE setup and all the services I run on it to vanilla FreeBSD, though admittedly that's progressing at a rather slow pace because it's *quite* a bit of work and also, at times, complex.

So, in any case, I'm pretty confident in saying that what you're attempting is indeed possible, but without much info on what went wrong in your experiments its not easy finding the errors and suggesting fixes.

Hope that helps!
Click to expand...
Thanks jmpalacios. It does help. And I agree, my post was pretty vague. I think I was more trying to get emotional support than to ask anyone to help solve a particular problem for me. Your note helps :-).

BTW - do you have any books on networking that you'd recommend? I have the TC/IP Guide, Absolute Freebsd, and The design and Implementation of the FreeBsd Operating System on my desk. But I really need something that explains things like network aliases, vlans, trunks, etc.
 
JohnK said:
toddg *nod* Thanks.

I use my own jail making script which has the ability to launch a script in the jail and on the host (which no jail manager allows you to do as far as I know); I pushed a copy to github if you'd want to try but it won't solve all of your problems. It was a different approach but no one really understood (or cared about) my concept--and I stopped pushing to github because of Ai, lack of participation/understaning, and/or more likely "I aint smart 'nuff"-. I couldn't even get anyone to talk to me let alone "install my tool". -e.g. AppJail and CBSD didn't want to talk. Someone from Bastile told me they didn't have make(1). etc.

I know your problem though--a "setup" needs to be dynamic and almost "JIT" so there is no "one setup" that seems to work (templates only take you sofar). If I have two servers, and wanted a "plex media server" on both of them, one single template will work if the servers are in the same location but if one has different external mount points, for example, this is when you need two templates.

My personal jail making script was essentially intended to be a wrapper to a jail making tool (pot, cbsd, bastile, appjail, base system, etc). -i.e. The "big idea" behind my script was the ability to keep the configs in a directory and a config file. However, I too ran into road blocks so I made other tools like "sysconf" (jail/host/config mgmt) and my fork of "drist" (deployment) but I'm now rebuilding my jail making script in C in a more flexible "hook system" methodology for myself but unfortunately I'm not trying to solve the 'JIT problem' with my redesign because I've sort of resigned to having two templates, so to speak.

My new system is designed much like a bottom-up design you'd do in Lisp (see: structure and intreptation of computer programs) but in C.

You may also want to check out BSD POT as a jail manager. Their templates are setup more docker like. I believe AppJail also has something along these lines too.
Click to expand...
JohnK Thx for the suggestion. I'll check out BSD POT.

The problem I'm trying to solve is this:

Say I have a simple webapp, a service, and a database. In this example, I want to expose the webapp and the service to the internet, but not the database (postgres). I also want to enforce ip, port, and username, password restrictions within the postgres configuration file...which means the webapp also needs to know the ip address and port of the database as well as the username and password.
Now suppose I want to deploy N instances of the webapp and M instances of the service but only 1 of the database (no replication for now). So I need to dynamically allocate ip addresses to the webapp and the service from a subnet.
Add to that the ability to deploy to multiple environments, e.g. global/dev/stage/prod.

No problem. This is all pretty easy and I've done it. I have a system that reads a config file, generates a model of all the resources, then uses that model to generate all of the files (/etc/rc.conf, /etc/pf.conf, /etc/jail.conf, /etc/jail.conf.d/*, etc.).
I've even added the abililty to deploy to subnets for blue/green deployments by dividing /24 network segments into /25 or whatever.

However, I decided to make 2 improvements that have been problematic.

1. Hardcoding
I hardcoded the loadbalancer and the database configuration into my deployment system, b/c they needed so much data to configure. However, this was just me being lazy. So I rebuilt the system to be generic, and make the load balancer and the database just a normal app, like any other app.

2. Networking
I had stuff working when I had everything in a single 10/8 network segment. I generated a pretty involved pf.conf file to handle all the allowed connectivity. But really, it'd be safer to break this down into network segments, and ideally use vlans and hierarchical jails to fully isolate things. But this is also my weak spot.

Thanks for your comments. It helped me to both read about your approach and to explain what I'm doing and why.
 
toddg said:
Thanks jmpalacios. It does help. And I agree, my post was pretty vague. I think I was more trying to get emotional support than to ask anyone to help solve a particular problem for me. Your note helps :-).

BTW - do you have any books on networking that you'd recommend? I have the TC/IP Guide, Absolute Freebsd, and The design and Implementation of the FreeBsd Operating System on my desk. But I really need something that explains things like network aliases, vlans, trunks, etc.
Click to expand...
Other than Michael W. Lucas', I don't have any FreeBSD-centric books that come to mind right now, and it's been very long since I last read anything network related in proper book form. For the last several years most of my reading on these topics, other than Lucas' books (Absolute Freebsd, his series on ZFS, and his book on jails), has been in the form of articles and/or tutorials for something very specific that I want to accomplish, man pages and/or other forms of vendor documentation, and even source code (the ultimate source of truth that has provided definitive answers multiple times when higher-level documentation has either been lacking or conflicting).

So apologies for being mostly empty-handed on this one, but those books that you mentioned should most definitely put you on the right track.
 
toddg said:
JohnK Thx for the suggestion. I'll check out BSD POT.

The problem I'm trying to solve is this:
<snip>
Click to expand...
I want to say that I understand but you are far, far above my skill level so please keep that in mind when I share the link below. This sounds very much in Pot's wheel house; many of "potluck's" readme instructions are saying things like this.

Below is the potluck readme for a posgres-single. There are other flavours (check out the master potluck list of flavors).


I apologize if this system (pot/potluck) is way off what you need; I really don't do this stuff professionally (I have a three file servers in multiple locations that have jails that do simple things).

If I understand the situation though (I would do something like): Instead of hardcoding, I would add some rc.d script(s) for updating the configs. For example, create a rc.d script that runs on cron and/or startup that takes a file at the root location and updates the config in /var/db/. This file could be updated on the fly by the host system. The "database jail" could use this system to only accept a connection from the jail's that are authorized (sort of how ssh's known_hosts works).

I may not understand what you're doing but it does sound cool/interesting so, please plan to add my ssh key to your server(s) later so I can learn.
 
JohnK Thank you for the link. Potluck is similar to what I'm doing.

victort Thank you for the suggestion.

Once I have my system working, I'll post it here so y'all can kick the tires. More soon.
 
victort What you have built, bastille and rocinante, look amazing. I'm not much of a shell script reader/writer, but your code looks really clean to me. I have looked at Bastille, pot, appjail, and cbsd and may eventually use one of them. So far, Bastille is my favorite b/c of the clean code and readable documentation.

* But first I wanted to learn all the inner details of networking and jails and such. On that note, I'm still working on understanding how to create thin jails, and how to partition networks and get dns working, etc.
* I also wanted to learn gleam and erlang... and I think I have a working understanding of those now.
* The last reason is that I'm not sure that Bastille can do quite what I want...and maybe what I want isn't what I need, so I thought I'd check in with you on this.

I want to build secure systems, such that I use defense-in-depth techniques. So that means clearly defining allowed traffic and enforcing these constraints at multiple levels, e.g. pf, caddy, public and private bridges, and allowed ip addresses, usernames and passwords in postgres. I also want to be able to spin groups of jails up and down, resize the number of instances, as well as re-define them on (not necessarily migrate) other servers.

So I have created glm_freebsd_cloud. This is an unlisted repo, but you and anyone else here is welcome to browse it. I'll publish it if and when it's ever ready. The idea is that I define my network and the apps in that network via a single toml file. Then, instead of using DHCP for the jail instances, I calculate the entire network (subnets, ip addresses for jails, etc) and put that in a model. I then use this model to generate all the configuration files and run the commands to start the services and jails and whatnot. The readme describes all of this.

A typical use case using hierarchical jails (hierarchical jails aren't necessary for this tool, I just think they might work well for my use case):

Host
* pf redirects 80/443 to caddy 8080/8443, nats the parent jails
* Top Level Jails
* caddy : for load balancing to dev|stg|prd jail hosts
* dev jail host (parent jail host)
* pf redirects parent jail host 8080/8443 to child web jail
* child web jail
* child db jail
* other child service jail

* stage jail host
...
* prod jail host
...

The hierarchical jails _should_ make things simpler, and make the host more stable (no changes to pf.conf at the host level, etc.)

Questions:
* Is my approach to defining the network overkill? Is there a simpler way?
* Is my approach to defense-in-depth even useful? I could be over-constraining the problem. Is there another more recommended way to enforce network security that simplifies the problem?
* How does Bastille handle creating isolated subnets and bridges?
* Do you have any examples of complex scripting of Bastille templates? The examples I could find were all very simple...

In the above repo, under ./example/outputs/server_a, you can see what this tool creates. This shows what a typical deployment output looks like (so you don't have to install anything or read my code to see what I'm talking about)
 
You must log in or register to reply here.
Back
Top