Sysadmin advise request

fred974

fred974

Hi,

I am looking for some advises from the experienced system admin here :)

We are a small web development business and we offer free hosting for all our clients..

When we started, we decided that it was best to keep each client (domain) inside their own jail so if a web application get compromised, then it is contained and cannot affect our other clients..

So far, we have 21 jail all running their own webserver and version of php but all sharing 1 common database (also in a jail).

This setup sounded like a good idea at the time but as we are starting to grow, we found a few issue with it. For example, managing and updating the jails and their software is prouving challenging. We started looking at poudriere but could get it to work for us as some jail has different version of php.

Anyway.. the reason for this post is this...Everyday, my inbox is flooded with daily/weekly run output email report for all the jails and I find it difficult to keep up to look at all of them snf often endup discarding them without reading it all.

How do you guys deal with this problem?
Do you think my setup is overkill for my needs?
I will welcome any sugestion and tools that you guys use

Thank you all
Fred
 
fred974 said:
I am looking for some advises from the experienced system admin here :)
Click to expand...
Ha! You are setting the hurdle high for answering. Obviously the experienced ones are too busy for an answer. So as an completely unexperienced one I jump in, if you don't mind. ;)
fred974 said:
Anyway.. the reason for this post is this...Everyday, my inbox is flooded with daily/weekly run output email report for all the jails and I find it difficult to keep up to look at all of them snf often endup discarding them without reading it all.
Click to expand...
Get an idea first of what you want to be informed on. Usually this are some specific strings in a logfile's line, such as "error", "warning", "failed", "not"... etc.

You can easily run tailored cronjobs scanning for the words you are interested in using grep(1) mailing you the filtered output.

I have disabled redundant logfiles in jails which would be related to specific functions of the jail-host. Make up your mind on this. And there is always the possibility to look at a jail's logfiles from the the jail-host so mailing might be avoidable here and then.

General rule: Use daily or shorter frequencies for emails only for the absolute high grade alerts you have to act on immediately. For periodic reviews weekly or monthly work might be sufficient. Tailor it for the priorities in your daily workflow, i.e at what hours do you want which infos.
 
Hi,

I'm doing something similar (different target audience, but also using jails in a similar way).
If you're looking to scale this to dozens and hundreds of jails there are a few bits of advice:
  • Setup the jail using nullfs for the base system. It allows updating the base in one go
  • Minimise what runs in your jails. They don't need to run email, reduce cron jobs. Limit what periodic does
  • Use automation like Ansible, Salt, Puppet or Chef to setup and maintain the jails
As for the logging. I'm looking at something like ELK to handle the log files in a central location.

I'm on the fence about nullfs, ZFS clones seem sufficient. I just make templates for the kinds of services I run, for example:
  • Jails with OpenJDK 8
  • Jails with PHP70
  • Jails with PHP56
  • Jails with MySQL
  • Jails with NGINX
I clone these and create a configuration in /etc/jail.conf for quickly spinning up and use Ansible to set each jail up with the right settings.
It's possible on a jail by jail basis to deviate from this. For example to add more PHP modules for specific projects.

Hope this helps.
 
hi bvansomeren
Thank you for sharing your advise, they are very valuable.
I will look at ELK tomorrow. I hear about ansible automation but don't really know what I need to automate.. Could you please give example?
What jail utility do you use?
I curently use sysutils/iocage and sysutils/bbcd
 
Hi,

I like the example by sko it's very complete and shows you just what you can do with automation and jails.
Not sure If it's frowned on to list your own work, but I just got done committing my own first role to handle jail setup on Github.
To be honest it's pretty simple and if you managed to do this by hand than automating it should be pretty simple.

Good luck!
 
fred974 said:
This setup sounded like a good idea at the time but as we are starting to grow, we found a few issue with it. For example, managing and updating the jails and their software is prouving challenging. We started looking at poudriere but could get it to work for us as some jail has different version of php.
Click to expand...

We use iocage update jail-name for more involved custom jails like the one I use for Gogs I iocage clone the original jail preform an update and then when everything works as expected I promote it to the master.

fred974 said:
Anyway.. the reason for this post is this...Everyday, my inbox is flooded with daily/weekly run output email report for all the jails and I find it difficult to keep up to look at all of them snf often endup discarding them without reading it all.
Click to expand...
Sure vanilla FreeBSD has awful defaults. I was able to deal with e-mail noise by studying FreeNAS configuration files https://github.com/freenas which indeed have a very sane defaults for e-mail notification.
 
Oko In which folder did you find the mail config files?
Oko said:
iocage clone the original jail preform an update and then when everything works as expected I promote it to the master
Click to expand...
How do you promote it back to the master?
 
Hi guys,

After a lot of reading I've decided to installed Ansible..
Could you please advise if I should install it on the FreeBSD host or jail? Or a completely different machine?
In FreeBSD how do I create 'Dynamic Inventory' of the jails?

Thank you
 
You must log in or register to reply here.
Back
Top