Best practice advices for automation of administration of multiple servers and VM

I've been using FreeBSD on two isolated servers for some time and intend to increase the numbers up to 7-8 physical servers and multiple VM (Windows, Linux) for various tasks (regular and on demand).

I'm rather fine with the system administration on each computer, connecting with SSH for doing some maintenance but would like to get the systems more integrated with each others and increasing automation.

What I gathered after some search for my needs:
  • Monitoring : Monit
  • Centralized Authentication : LDAP
Now one area where I'm not fully sure is how to automate upgrades. I wouldn't want to have to connect on each computers and VM to upgrade them when there is a new version of a package or base system. I've got three ideas that might need to be mixed:
  • Remote administration tool like Puppet for giving orders (is there a better tool?)
  • Using a master computer that I set up properly and upgrade then replicate on all my FreeBSD machines (likely using ZFS replication)
  • Using jails (bastille) on a central development computer and then copying them on the production servers

I'd like to keep it simple and lower my maintenance cost and care, but still feeling in control of what is happening and being able to see at a glance that all is fine on my servers so that's why I ask people here that likely have more experience if my conclusions are fine.

Example of scenarios that I'd like to check at a glance or receive notifications:
  • That all the servers/VM are running properly
  • That a program in use/installed has a security vulnerability found and needs action
  • List of services are running fine
  • That disk health/space is fine
  • Backup is up to date
  • Warning/Error from logs of all the machines
Could you share with me what would be your solution for a similar goal and if you see better tools for doing what I want. Also if you know a good guide for setting up all this to take inspiration from.
I'm a programmer with rather correct knowledge of Unix so I'm not afraid of more complex solutions if they are better in the long run. Thanks!
 
Remote administration tool like Puppet for giving orders (is there a better tool?)
You could definitely use Puppet to manage many systems from a centralized point. Ansible is more popular at this time though. Puppet has quite a steep learning curve to be able to use it effectively, something to keep in mind. Ansible is a little easier to get started with.

That all the servers/VM are running properly
Buy a license for M/Monit and set up a M/Monit server for the monit nodes to report to. Their FreeBSD version runs excellent. Good support too, ran into a bug a few years ago, reported and fixed within a day. They're license is relatively cheap too (compared to other payed products).

That a program in use/installed has a security vulnerability found and needs action
Configure the system's mail to get sent to a central mail address. Actually look at the periodic(8) mail that's generated in security, daily, weekly and monthly reports. This is already in place.

  • List of services are running fine
  • That disk health/space is fine
M/Monit + monit.
 
pkg search ansible gives you an idea whats available.

I almost recommended it but this is more of a personal choice versus 'better tool'.
 
You could definitely use Puppet to manage many systems from a centralized point. Ansible is more popular at this time though. Puppet has quite a steep learning curve to be able to use it effectively, something to keep in mind. Ansible is a little easier to get started with.
I almost recommended it but this is more of a personal choice versus 'better tool'.
I had discarded Ansible earlier because I had improperly read that Ansible was requiring a python interpreter on the node. After reading yours messages, I checked the documentation and it seems to be a good match for what I want to do.
I've seen that a GUI exist for it (AWX) but doesn't seem to be compatible with FreeBSD (docker/kubernete dependency?). Is there an alternative one that you advice or do you use CLI only?
Buy a license for M/Monit and set up a M/Monit server for the monit nodes to report to.
Thanks for the advice, I'll start with the trial and see. I'd like to isolate most of my services into separate jails, and I hope that this would not inflate the number of hosts required for the licence. I'll have to check this point.
sysutils/syslog-ng is a nice stand alone log server. You would need to check monit for its capabilities.
This is not a good idea. Each machine has thier own SSH keys, Hostname ect....
Thanks for the advices. On the replication part, I was thinking there should be a way of isolating the machine dependent files (which should be mostly located into /etc, /usr/local/etc and /boot) from the rest. I was thinking if I go through this solution of setting the core system as read-only apart from specific folders and only getting it modified through replication. This is not a hard requirement for those local servers, but I'd like to try this for a remote one that serves a website on a machine I don't own.
 
I had discarded Ansible earlier because I had improperly read that Ansible was requiring a python interpreter on the node. After reading yours messages, I checked the documentation and it seems to be a good match for what I want to do.
Puppet requires a puppet agent (written in Ruby) on all the hosts you want to manage and a Puppet server. Ansible only requires SSH access and a Python interpreter, no agents and no server required to use it.
I've seen that a GUI exist for it (AWX) but doesn't seem to be compatible with FreeBSD (docker/kubernete dependency?). Is there an alternative one that you advice or do you use CLI only?
Just use the playbooks from the CLI. I've had a look at porting AWX to FreeBSD but that's going to be a major undertaking. Not impossible to do, just a lot of work.
 
I agree that Ansible can help with many of the tasks you’re looking to do. It’s what I use to provision new VMs quickly and ensure consistency in state between machines in the fleet. It’s super light weight. You can spin up a tiny VM to be the master/controller and go from there.

You could write playbooks for tasks that are common to all machines, like updating packages, outputting disk usage reports, etc., then run that playbook on your fleet.

If the machines have different purposes (web server, DB server, etc.) you could make specific playbooks for those machine classes. There’s even a way to group machines by type I believe, so you can say something like “run this playbook on all DB machines.”

I used to use Salt but didn’t like that there was a root level agent always running on each machine, listening for commands to execute. Seemed like a vulnerability vector and I never really felt comfortable leaving that running all the time.
 
Back
Top