Solved How to ensure security updates in a fleet of servers?

[tux2bsd]

debian has 'cron-apt', and that's the tool to mimic for what you're wanting to do.

 

[Jose]

On the other hand: Would you really trust a team to backport a lot of fixes in a lot of foreign codebases where upstream does not support this? It seems a bit risky to me, for something to "slip through", or accidental breakages of edge cases…

This has gone wrong in spectacular fashion relatively recently:

debian-ssh/README.md at master · g0tmi1k/debian-ssh

Debian OpenSSL Predictable PRNG (CVE-2008-0166). Contribute to g0tmi1k/debian-ssh development by creating an account on GitHub.

github.com

The proper way for a herd of servers is to automate updates via orchestration tools (ansible, chef, puppet etc) - this way you still control WHEN, HOW and IF packages are updated...

Ansible (and perhaps some of the others) allows you to break up your servers into groups. You can then update them in stages. That's what I would do. Split the world into 2-3 groups, update one group at a time and let it bake overnight before updating any others.

And I will probably never understand why people provide advice that somebody has explicitly asked for not to recieve...I will try to close this thread-.

This is not your personal support forum. This is a public discussions forum. You've started a topic that's of interest to a lot of server admins, and we're going to discuss it whether you like it or not. We're not necessarily talking to you.

 

[msplsh]

I posted the "answer" because you seemed like the kind of person who thinks perfect is the enemy of good. While it's possible that the suggested automatic update will completely hose something (hence the "not a good idea"), you seem to be willing to take that chance, and if you get cron's email, it will show you if it blows up and what it did to blow it up so you know how to fix it.

A slight upgrade (but major programming change) to this program might email you the packages needed to be upgraded and an "unlock key" you would post on a master server somewhere to authorize the program to actually perform the upgrade which it would check on the server for next time the program is run, making the install automatic but not the decision to do the install.

A "more correct" answer would be some kind of system that determines the update flow, detects when additional packages get roped in, maybe checks semver on the packages, and detects a potential for major disruption, but that's way more complicated than Debian's solution. I figured if you were good with how they updated things, then you were good with whatever packages are offered as an "upgrade" in FreeBSD.

Sometimes, though, upgrades will bring in dependencies and cause breakages. That's why people think you should use a CI/CD (more infrastructure intensive) to detect this or just do it manually (more labor intensive). I sometimes YOLO pkg upgrade on things I'm not sure about, so why can't you?

 

[zirias@]

This has gone wrong in spectacular fashion relatively recently:

debian-ssh/README.md at master · g0tmi1k/debian-ssh

Debian OpenSSL Predictable PRNG (CVE-2008-0166). Contribute to g0tmi1k/debian-ssh development by creating an account on GitHub.

github.com

Wouldn't call that "recently" though. Of course, you couldn't miss this huge fail back then. I didn't look into the details until now…

Troll mode: So, using uninitialized memory is a nice idea to collect entropy, right?

But seriously, this is beyond fail, and not the result of backporting security fixes. More like "oh this code generates warnings? Let's remove it, problem solved!"

I'd expect problems with the backporting approach to be more subtle. It's well possible that even someone very experienced, doing the job very carefully, might fail. Fixing a security issue requires complete understanding of the problem, and this often also means intimate knowledge of the affected code base. So IMHO, there's always a risk attached to the idea of a distributor doing this kind of backporting.

 

[SirDice]

Totally offtopic but I've spent 2 years chasing RedHat support on getting a bug fixed in Samba. A bug that was already squashed upstream. Yeah, back-porting works really awesome

 

[hbauer]

This is a public discussions forum. You've started a topic that's of interest to a lot of server admins, and we're going to discuss it whether you like it or not. We're not necessarily talking to you.

since there seems to be no desire to stay on the topic of the thread I take the chance to express some surprise. I would have expected this to be a forum of technical oriented and focused people. If I look at this thread I get the impression i joined a club of retired seniors at a sunday afternoon tea time.

 

[zirias@]

Translation: "Anyone mentioning how what I want to have could cause problems is insulting me, so I'll just insult anyone engaging in such disrespectful lese majesty."

Of course, risks from fully automated updates are perfectly on-topic, and I won't start to talk about my professional background here.

 

[msplsh]

I would have expected this to be a forum of technical oriented and focused people

The technical analysis of your problem was that it necessitates either a flawed solution or an unacceptable amount of excess work for a correct one. That's why I said they pay people the big bucks to solve it.