Best way to automatically keep system, packages, and daemons up to date

[japoc]

Hi,

I'd like to know if you have a good way of keeping a system completely up to date in a completely automated way.

If my understanding is correct, I can add

42 8 * * * freebsd-update cron

in root's crontab to have the base system (I guess that means the kernel and the programs in /bin) patched with the latest security updates everyday at 8:42.
Now, does that restart services that need to be restarted? For example, if sshd depends on a library that's been patched, does freebsd-update cron restarts the sshd daemon?
If not is there a way to deal with that automatically?

In order to upgrade the other programs (those installed in /usr/local/bin), I guess the way to do that would be to add an

55 9 * * * env ASSUME_ALWAYS_YES=YES pkg upgrade

entry in cron.
I have the same question about that: are services that depend on the updated packages automatically restarted?
I'm thinking about services like postgresql, apache or nginx for example.

I haven't looked into ports yet (I'm quite new to FreeBSD) but I guess this is something that has to be dealt with too in term of security updates.

So I guess my question is: how do you keep your system secure with the least human intervention possible?

Thanks for your help.

 

[SirDice]

Now, does that restart services that need to be restarted? For example, if sshd depends on a library that's been patched, does freebsd-update cron restarts the sshd daemon?

No, it doesn't.

I have the same question about that: are services that depend on the updated packages automatically restarted?
I'm thinking about services like postgresql, apache or nginx for example.

Again, no.

To save yourself a lot of headaches, don't update your system automatically. Sooner or later it's going to bite you and you wake up to find a non-functioning server.

 

[storvi_net]

#2.
Set up cronjobs / periodic-tasks to check:

1. existent updates for your system.
2. existent security flaws for your packages.

For less than 5 systems I think emails every day are enough - for more than 5 setup kind of monitoring the result of this tasks. Then manually update the systems!

If you want to lower the effort, try to implement some automation-software like sysutils/puppet, SaltStack or anything else where you can easily bulk update many systems at once (of course after you tested a single system).

Regards
Markus

 

[Peacekeeper2000]

Hmm, when - like here it is suggested multiple times - only manual maintainance is longterm working, how the hack are people telling me, that Amazon needs only one (human) admin per 10,000 server ?

 

[protocelt]

That one admin is still doing manual/supervised maintenance. The admin is using software such as Puppet and SaltStack as already suggested by storvi_net above to manage the systems removing a whole lot of effort and time. It doesn't remove the supervised maintenance requirement.

 

[ANOKNUSA]

..how the hack are people telling me, that Amazon needs only one (human) admin per 10,000 server ?

We can restate protocelt's point in the form of a question: if everything's automated, why have any sysadmin on the payroll at all? To add to protocelt's point, that Amazon system administrator not only manually updates their systems, they almost certainly only do so when there's a compelling reason to--once per year, perhaps? Maybe less often than that. Automating a potentially dangerous operation you rarely perform is pointless.

 

[ondra_knezour]

...how the hack are people telling me, that Amazon needs only one (human) admin per 10,000 server ?...

Take one small team of the highly qualified admins which prepare and test all scenarios which can arise when you upgrade your pool. When everything is tested many times, put resulting image on the network storage from which all your machines boots and click rolling pool upgrade. Let the machines do the rest. This way you can keep your admins/machines ratio in ranges you mentioned, however this would not work in highly heterogeneous environments (read we collected dozens, maybe hundreds machines of different types, OSes, utilization, hardware etc. during our way to this day and we want to automate almost everything).

 

[Peacekeeper2000]

That one admin is still doing manual/supervised maintenance. The admin is using software such as Puppet and SaltStack as already suggested by storvi_net above to manage the systems removing a whole lot of effort and time. It doesn't remove the supervised maintenance requirement.

Thanks for pointing me to Saltstack and Puppet - was not aware and learned something ! ( Not sure if I will use these tools in my private datacenter, but you never know )

We can restate protocelt's point in the form of a question: if everything's automated, why have any sysadmin on the payroll at all? To add to protocelt's point, that Amazon system administrator not only manually updates their systems, they almost certainly only do so when there's a compelling reason to--once per year, perhaps? Maybe less often than that. Automating a potentially dangerous operation you rarely perform is pointless.

Actually that was a discussion about industry 4.0 and if a sysadmin is still needed. The forecast was "No" - where I have my doubt: that sounds like using a car and retire all garages

Take one small team of the highly qualified admins which prepare and test all scenarios which can arise when you upgrade your pool. When everything is tested many times, put resulting image on the network storage from which all your machines boots and click rolling pool upgrade. Let the machines do the rest. This way you can keep your admins/machines ratio in ranges you mentioned, however this would not work in highly heterogeneous environments (read we collected dozens, maybe hundreds machines of different types, OSes, utilization, hardware etc. during our way to this day and we want to automate almost everything).

I think your scenario sounds correct when you think about VM's : these VM's could be identical on different hardware server. So you would maintain the VM's and keep the real hardware "in a way up an running - no/low touch"

 

[kpa]

Automation can be done right and wrong. The wrong kind of automation is installing updates blindly without paying attention to what t

Hmm, when - like here it is suggested multiple times - only manual maintainance is longterm working, how the hack are people telling me, that Amazon needs only one (human) admin per 10,000 server ?

They have tons of customized automation tools that allow them to do automated maintainance and updates in a safe way. You can do automated updates safely but it requires some real work done on the tools and that work is not cheap in terms of manpower and money. FreeBSD doesn't have anything compareable yet in its base and that's why you should never use unattended updates on it.

 

[storvi_net]

For example see the Satellite-Server from RedHat or Spacewalk for money saving. There you can say "Patch every 2 Weeks and restart anything".

We have this for some services (not sooo critical) and this works. But for the very critical services we do it manually and test it before in testing environment, then reference / staging and finally production (nightshift).

Regards
Markus