Pot and nomad

E

einsibjani

We run around 100 vm's (staging and production) and most of those are running FreeBSD.
The last two years we've made some progress in shifting to ansible and single function vm's.
Deploying vm's is done with ansible, where the playbook creates a custom install iso, which gets deployed to Proxmox cluster and after boot, freebsd-update and ssh startsup the playbook continues to configure services.

I've been looking at ways to have Gitlab CI build images for us, so every deployment of ephemeral servers would replace the whole image.

I've been aware of pot and seen presentations about using pot + nomad for container orchestration and I'm very interested, but at the same time I'm vary of adding this much complexity to our infrastructure.

Is there anybody here that is using pot + nomad (or other orchestration tool) with freebsd and is it worth it to go down that road?
 
This response is probably utterly useless from your position/perspective but: If you're running 100 VMs where most of them are FreeBSD guests, have you considered using FreeBSD host(s) instead and using jails?
I'm brining this up because we too once maintained a small project-specific cluster based on 5 Proxmox nodes with ~70 FreeBSD VMs and eventually transitioned to FreeBSD hosts with jails. It's much more clean, lean, easier to maintain, more flexible and less resource intensive.

Sorry for not being able to comment on your actual question(s).
 
jbodenmann said:
This response is probably utterly useless from your position/perspective but: If you're running 100 VMs where most of them are FreeBSD guests, have you considered using FreeBSD host(s) instead and using jails?
Click to expand...
Not at all useless! My journey actually started looking at ways of migrating to jails, and how to deploy them from ansible. This started down the rabbit hole of maybe building the jails in the CI pipeline and deploying from there.

Just migrating to jails would solve some problems for us (less resources, easier management) but the holy grail is full image deploys.
 
You should definitely be able to do that with jails (note: I'm far from an expert - might be worth getting someone like SirDice in here). Please take the following with a grain of salt - especially terminology wise:

With jails you can either spawn a jail as a copy (doesn´t even need to be an actual copy) of the host OS or you can build a custom jail. For example, on our build server we're running stable/13 on the host but we're spawning different jails for different scenarios (eg. jails for 13.0-RELEASE etc).
These jails can be pre-configured so you don´t need to do that manually.

There are various utilities available to manage & maintain jails with a bit more comfort if you don´t feel like writing all the scripts yourself. I´d say notable examples include: sysutils/bastille, sysutils/iocage and sysutils/cbsd.
Personally, I've been using sysutils/cbsd for many years now both in staging and production (as well as on my personal machines). The nice thing about CBSD is that it supports stuff like pre-configured profiles, migrations to different nodes and so on. Furthermore, you can use the same utility to manage both jails and VMs (bhyve & Xen).
Huge shout-out to Ole.
 
einsibjani said:
Deploying vm's is done with ansible, where the playbook creates a custom install iso, which gets deployed to Proxmox cluster and after boot, freebsd-update and ssh startsup the playbook continues to configure services.
Click to expand...
Do I understand you correct, that you are looking for a way to automatically create FreeBSD based virtual machines, which are then deployed to a Linux based Proxmox cluster, so using KVM and other technologies?

So your host OS is Linux, and guest OS mostly FreeBSD?
 
hardworkingnewbie said:
Do I understand you correct, that you are looking for a way to automatically create FreeBSD based virtual machines, which are then deployed to a Linux based Proxmox cluster, so using KVM and other technologies?

So your host OS is Linux, and guest OS mostly FreeBSD?
Click to expand...
Mostly I'm asking if anybody is using pot+nomad to do container/jail orchestration from a CI pipeline. How we would deploy the pots is a different issue, probably FreeBSD VM running on Proxmox, or we might deploy dedicated FreeBSD servers.
 
I have had to do this on a much smaller scale. There the approach was to spin up new host FreeBSD VMs. Then archive the jail to be cloned, and then send it over to the new host. On the new host, create a new jail from this archive. Most of it was scripted. Jails were managed with ezjail. I don't know how this will play with CI pipelines.

I am right now looking at a more structured way of doing this. So considering Bastille and Pot/Nomad. I also have the wild idea of using the previous scripts and extending them to play with the load balancer api. It is also possible to simply zfs clone the jail directory and send that over. It is also possible to simply take a snapshot of the whole VM and create a new instance from that. Most cloud api's allow all this.

What did you end up doing einsibjani ?
 
You must log in or register to reply here.
Back
Top