Solved How to ensure security updates in a fleet of servers?

[hbauer]

I am coming from the Linux side and I am running a handful of servers. Within Debian there is a mechanism called unattended-upgraddes. The purpose of unattended-upgrades is to keep the computer current with the latest security (and other) updates automatically.

So everytime one of my packages on the servers has a security update my system will update this at a certain time of the day. Years ago I was hesitating but this process has been extremely stable in that last years.

I was looking for a similar feature in the FreeBSD world but I was not successful. (Yes I know the proper commandlines to update a system) Did I just chose the wrong search pattern or is such a mechanism just not there?

What a best practises to maintain the updates on a swarm of server and jails? Just individual cron / batch solutions? I cant imaging that everybody does this manually.

Any hints are appreciated
Hagen

 

[msplsh]

They pay people the big bucks to answer this IT question.

The way you describe linux working doesn't sound much different than the cron solution.

 

[hbauer]

They pay people the big bucks to answer this IT question.

Ah. In Debian this is for free

The way you describe linux working doesn't sound much different than the cron solution.

To my knowledge a cron solution would update all packages. Or is there a list that pkg could use to only update updates for security relevant packages?

And just to be sure, this question is not about Linux vs FreeBSD. I am just looking for something similar.

 

[reddy]

You can use pkg audit -q and pipe the output in a cron job. This command will list all the packages having known vulnerabilities.

Edit: However, there is then a need to restart the service/application and I am not sure this can be easily automated since the proper approach will be situation specific. For example, if your database has a vulnerability, you may not want to update it straight away to avoid downtime.

 

[hbauer]

You can use pkg audit -q and pipe the output in a cron job. This command will list all the packages having known vulnerabilities.

Edit: However, there is then a need to restart the service/application and I am not sure this can be easily automated since the proper approach will be situation specific. For example, if your database has a vulnerability, you may not want to update it straight away to avoid downtime.

I would be surprised if a database update would need a reboot. And that linux tooling is telling me if a kernel upgrade would require a reboot that needs to be done manually.

I will try to follow this pkg audit path.

 

[SirDice]

You can use pkg audit -q and pipe the output in a cron job. This command will list all the packages having known vulnerabilities.

Read the daily periodic(8) security emails.

Checking for packages with security vulnerabilities:
Database fetched: Sun Apr 18 03:59:47 CEST 2021
curl-7.75.0

 

[zirias@]

Restarting a service is not a reboot. Still, it produces downtime (of that service).

 

[hbauer]

Restarting a service is not a reboot. Still, it produces downtime (of that service).

you are right. my mistake. But I my case a database restart in the middle of the night is acceptabel in my situation and worked without problems multiple times in the last 8 years

 

[SirDice]

I absolutely despise automatic updates, restarts and reboots. I much rather keep control to myself, so I can determine exactly when, how and what to take offline to update. What if both my HAProxy machines decide to update themselves at the same time. Or they go in order but the update breaks something that needs a configuration change. Then they'll go offline, one after the other. No thanks, not taking any chances here.

 

[hbauer]

I absolutely despise automatic updates, restarts and reboots. I much rather keep control to myself, so I can determine exactly when, how and what to take offline to update.

Good to know but my question was not if people like the idea I am just trying to move my servers to FreeBSD and I am looking for ways to re implement functionality that has been working nicely in the past years

 

[msplsh]

Ah. In Debian this is for free

I think I'm joking about something that you're not doing, which is a CI/CD setup.


This is not a good idea, but I coded it up for you

import json
import sys
import subprocess

input_json = sys.stdin.read().encode('utf-8')
if input_json:
    try:
        audit_info = json.loads(input_json)
        if 'packages' in audit_info:
            for package in audit_info['packages']:
                update_results = subprocess.run(
                    ['pkg','upgrade','-y',package],
                    stderr=subprocess.STDOUT,stdout=subprocess.PIPE
                )
                print(update_results.stdout.decode('utf-8'))
    except Exception as e:
        sys.exit(1)

and then in cron

pkg audit --raw=json | /usr/local/bin/python3.7 security_pkg_updates.py

 

[rootbert]

thats also a feature I miss. The thing with ports and pkgs in FreeBSD is that bugs and security fixes are not backported to a version here. Without such an infrastructure (most importantly the manpower behind this) it is difficult to setup such a thing ... suppose we have a security issue in nginx-1.18, but upstream is already at 1.20, so 1.18 does not get any updates. In debian this is backported at some point, however in FreeBSD it is up to the port maintainer if she backports a fix. Usually this does not happen, just with some ports, so it is just advised to upgrade to nginx-1.20. The config in nginx-1.20 might have changed, so we might have problem here with automatic updates.

so in your case I would choose the quarterly packages and track the security related stuff for a certain amount of time, then you can estimate when/how fast a security update is rolled out for your packages. Also, I would use a semi-automated way to test those updates, maybe in conjunction with ansible and jails or other tools.

 

[SirDice]

suppose we have a security issue in nginx-1.18, but upstream is already at 1.20, so 1.18 does not get any updates. In debian this is backported at some point, however in FreeBSD it is up to the port maintainer if she backports a fix. Usually this does not happen, just with some ports, so it is just advised to upgrade to nginx-1.20.

Fixes like that are almost never backported. The port will simply get updated to the new upstream version. I know of very few cases (over a span of 20 years) where a security issue was patched in the port before it was fixed upstream. And that only happened because upstream kept dragging their feet and not fixing the issue in a timely manner.

 

[Jose]

Upgrading the whole system (including packages)

Hello FreeBSD forum! I just installed FreeBSD for the first time and wanted to upgrade the whole system. (I'm planning on only using the pre-compiled packages, if that matters. And I have ran the # /usr/sbin/pkg command) But did I understand correctly that if I run # freebsd-update fetch #...

forums.FreeBSD.org

 

[richardtoohey2]

I think the short answer is no, FreeBSD doesn't support this. I think the general principle is as SirDice said - you are anticipated to want a lot more control over when things (services/servers) are updated and restarted/rebooted.

It's a bit more conservative in approach.

So although it works well elsewhere having automated updates and restarts, it's not something that FreeBSD does.

I use pkg audit to look for vulnerabilities and other tools to look for updates - all that works well. But as to when the updates are done and when a service/server restarts after an update is left to me.

After some updates there are instructions for configuration changes or steps you might need to take to migrate to a new version or things you might want to check.

Having used some variants of Linux I know what you mean - with updates it's a command or two, and it goes off, takes care of everything, migrates, restarts, suggests a reboot etc. I don't think there's anything like that in FreeBSD and I think that's a design choice.

 

[hbauer]

I don't think there's anything like that in FreeBSD and I think that's a design choice.

Ok. At least I have not missed something in my search. I understand that unattended upgrades are not good for every situation but there are certainly situations where they would be good.

I also dont believe that this is a design decission. Debian for example doesnt force you either. Both systems give you the same control if you want to manage all by yourself. The Debian community provides you with the additional tooling to go for automated patching if your system architecture allows for it.

To bad but I am sure I will live with it and build my own custom cron tooling with the input for msplsh.

 

[zirias@]

suppose we have a security issue in nginx-1.18, but upstream is already at 1.20, so 1.18 does not get any updates. In debian this is backported at some point,

Having used Debian for a long time – on the one hand, this seems nice for the admin to provide a system that requires close to zero work unless doing a major upgrade. On the other hand: Would you really trust a team to backport a lot of fixes in a lot of foreign codebases where upstream does not support this? It seems a bit risky to me, for something to "slip through", or accidental breakages of edge cases…

The perfect solution is LTS-releases supported upstream, but only a few large projects do this. In absence of these, well, you have to do your maintenance work as the admin. Maybe it could be interesting to discuss tooling how to deploy changes to a whole farm after upgrading and testing a single machine (including configuration changes that might be necessary)…

 

[hbauer]

I have a feeling that I may have used the wrong wording. My itend is automate security patching. So its not about updating major or minor versions. Lets take one current example:

curl-7.74.0 is vulnerable:
  curl -- Automatic referer leaks credentials
  CVE: CVE-2021-22876
  WWW: https://vuxml.FreeBSD.org/freebsd/b1194286-958e-11eb-9c34-080027f515ea.html

  curl -- TLS 1.3 session ticket proxy host mixup
  CVE: CVE-2021-22890
  WWW: https://vuxml.FreeBSD.org/freebsd/d10fc771-958f-11eb-9c34-080027f515ea.html

nettle-3.6 is vulnerable:
  nettle 3.7.2 -- fix serious ECDSA signature verify bug
  WWW: https://vuxml.FreeBSD.org/freebsd/80f9dbd3-8eec-11eb-b9e8-3525f51429a0.html

May be have have not yet experienced such a case but my impression is that in this type of "patches" there is extremly seldom a configuration change. Most people will apply this blindly.

Also if I am running a mariadb for a wordpress site a security patch for Mariadb in most personal blog situations can be updated/restarted unattended. (yes I know this does ot apply to mission critical multi server HA systems)

 

[Morris]

Isn’t this something you could sort of build using Poudriere?

You have your machines check the local Poudriere repository at a useful frequency depending on the situation (hourly? six-hourly? nightly?) and then use Poudriere as your gateway for update rollout.

Anything that you are not worried about receiving unattended updates and restarts goes into your Poudriere to be picked up by your machines. Things you are a bit more concerned about (database servers?) you test before you flick the switch on those packages in Poudriere.

This way you have a semi-automated rollout of updates while still maintaining control for stability's sake.

 

[SirDice]

I also dont believe that this is a design decission.

Actually it is. On Debian for example, if MySQL gets updated apt will automatically restart the service. I'm not talking about automatically installed updates here, it happens when you run apt upgrade. Same on RHEL, yum will restart the service after it installed the updated package. Doing this kind of automated restarts of services when packages are updated is specifically not allowed.

 

[sko]

Automatically updating and restarting services is a really bad practice - but it seems bad practices are the "new hip thing" on linux ("a container crashed, just start 2 new ones"), so no wonder they also promote unsupervised updates...

The proper way for a herd of servers is to automate updates via orchestration tools (ansible, chef, puppet etc) - this way you still control WHEN, HOW and IF packages are updated (even on a per-host/jail basis) and services are restarted AND you get immediate feedback. Because you triggered the process manually, you can also investigate immediately.
For jails you can either treat every jail individually as a separate host, or (better) just instrument the jailhost to update packages in its jails. This way not every jail needs to run its own sshd and/or agent for the orchestration/configuration management tool and you can completely confine them i.e. on loopback interfaces with very limited or no access from/to the external world.

Everything disruptive that has been completely automated WILL fail at the most inconvenient time and usually could have waited until you're back in the office to trigger updates manually. Highly critical securityupdates are usually announced a few days in advance, so they can be scheduled and performed manually, including verification if the problem has been mitigated.


Automation is good - but not everything SHOULD be fully automated, especially if it is/can be disruptive and ruin your weekend.



edit:
RE the mariadb example: Databases very rarely have to be accessible from the internet (read: they shold NEVER be accessible by the world until you absolutely know what you are doing). Especially for website stuff you usually completely confine them within the host where they are needed (i.e. a jail connected to a loopback interface).

 

[hbauer]

To be honest your advice

Automation is good - but not everything SHOULD be fully automated, especially if it is/can be disruptive and ruin your weekend.

is useless because you have successfully demonstrated that your are unable to read/understand or respect the question

Good to know but my question was not if people like the idea I am just trying to move my servers to FreeBSD and I am looking for ways to re implement functionality that has been working nicely in the past years

I really appreciate comments that dont tell me my idea ist stupid. There are a lot of people out there that have systems and organisational requirements that do follow such an approach. If your situation is different then be happy and move on .

 

[zirias@]

So, you used something on a specific Linux dist and people explain to you how this could break and ruin your day. It would make sense to take some time thinking about that when looking for a good way to manage your servers – much more than scolding people trying to offer advice.

If your conclusion is still "I want to automate everything", that's fine if it is an informed decision. You should be aware of a few things (kind of summarizing this thread):

• Scripting with cron et al will be your job, there's no pre-made tooling for this
• FreeBSD ports never attempt to "back-port" security fixes. If there isn't an upstream LTS, applying a security fix equals upgrading the package
• Services aren't restarted automatically, so if you want/need this, keep it in mind

 

[hbauer]

It would make sense to take some time thinking about that when looking for a good way to manage your servers – much more than scolding people trying to offer advice.

Why do you assume I didnt do the thinking the before I asked the question. And I will probably never understand why people provide advice that somebody has explicitly asked for not to recieve. Only because somebody is new to one operating system does not mean he has no idea how to run a server. There are other OS out there too.

And yes, as I already have written. I will move on with a cron based approach and pkg audit...
I will try to close this thread-.

 

[tux2bsd]

I think with the pkg audit method you will find this scenario:
vulnerability exists
fix/new is not ready yet
so then you loop on above until fix/new is ready (could be days+)

Just something to keep in mind. (I'll be corrected if I'm wrong, as is tradition.)