Backdoor in upstream xz/liblzma leading to SSH server compromise

Jose said:
Thoughts on the infamous Debian Openssl patch?
Click to expand...
First of all I think there are several issues with Debian. Their stable release is IMO excessively conservative, often several years behind current releases. That leads to Debian being in a bubble disconnected from upstream and Debian doing their own thing. So they miss out on upstream fixes and apply their own, of dubious quality.

Secondly l don’t know of any Debian developers contributing to upstream Valgrind. Red Hat, on the other hand, is a major contributor. Putting that in fairly strong terms, I don’t think that anyone at Debian really has a clue how Valgrind works.

In general the problem is “quality by numbers”. Making errors go away without really understanding the root cause is a recipe for disaster.
 
Paul Floyd said:
That leads to Debian being in a bubble disconnected from upstream and Debian doing their own thing. So they miss out on upstream fixes and apply their own, of dubious quality.
Click to expand...
I do tend to agree. Actually, many people think of Debian as being one of the standard Linuxes but in many ways Debian does tend to have the most non-standard adapter scripts compared to others. If you look through many of the postinst scripts in the .deb files it is quite horrific. For just one example, vconsole.conf is replaced with their own console-setup alternative that often fails (I think this bug still pops up or is ignored on faster platforms).

But I do kind of get why. They want to make an OS that, like Windows and macOS moves at a snails pace. This is a *feature* of a good operating system so that users and developers can focus on *their* software. Linux as a technology just moves too fast, creating that ugly separation. Linux is ultimately the slightly wrong tool for the job here.
Paul Floyd said:
Secondly l don’t know of any Debian developers contributing to upstream Valgrind. Red Hat, on the other hand, is a major contributor. Putting that in fairly strong terms, I don’t think that anyone at Debian really has a clue how Valgrind works.
Click to expand...
I don't think this matters too much to be an upstream developer. Like FreeBSD (the Valgrind maintainer is sometimes on these forums), it is largely a best effort approach. Interestingly Debian has much less work here compared to FreeBSD.
 
kpedersen said:
I don't think this matters too much to be an upstream developer.
Click to expand...

Not essential but it helps.

kpedersen said:
Like FreeBSD (the Valgrind maintainer is sometimes on these forums), it is largely a best effort approach.
Click to expand...

The maintainer, that's me, but not just the port maintainer. I also do most of the development of Valgrind these days.

kpedersen said:
Interestingly Debian has much less work here compared to FreeBSD.
Click to expand...

Not sure what you mean there. Debian has less work to do on Valgrind maintenance? Or contributes less?
 
Paul Floyd said:
The maintainer, that's me, but not just the port maintainer. I also do most of the development of Valgrind these days.
Click to expand...
Ah, it is you. In that case, thank you for your work!

Paul Floyd said:
Not sure what you mean there. Debian has less work to do on Valgrind maintenance? Or contributes less?
Click to expand...
Typically packagers for Linux distros have less work to do compared to FreeBSD packagers. Less patches required to make it portable, etc.

So less for them to mess up basically.
 
kpedersen said:
Typically packagers for Linux distros have less work to do compared to FreeBSD packagers. Less patches required to make it portable, etc.

So less for them to mess up basically.
Click to expand...

Dealing with BSD make is a bit of a hassle, but the port committers seem to keep an eye on the quality of the ports Makefiles.

Also Valgrind is definitely an exception to that rule. Mostly I don't bother with patches. I just push fixes upstream and bump 'valgrind-devel'.

Debian Sid https://packages.debian.org/sid/valgrind (2 and soon 3 versions out of date) has 10 patches. Looking at the patches they don't inspire much confidence.

Debian does this

# On armhf, stack-clash-protector is implemented by writing out of stack
# bounds. https://bugzilla.redhat.com/show_bug.cgi?id=1522678
{
stack-clash-protection-armhf
Memcheck:Addr4
obj:*
}

which hides ALL 4 byte address errors.

In the RedHat link they say

Per comment 1, there is simply not enough upstream support to fix this. We will have to build armhfp without -fstack-clash-protection.

No prizes for guessing which one I consider to be the better solution.
 
Paul Floyd said:
In general the problem is “quality by numbers”. Making errors go away without really understanding the root cause is a recipe for disaster.
Click to expand...
But-but-but we need quality metrics! All joking aside, this is why I'm not a fan of "if you can't measure it, you can't improve it." There are things that are hard to measure, like quality and happiness.

kpedersen said:
Ah, it is you. In that case, thank you for your work!
Click to expand...
I figured he would have an opinion ;)
 
Jose said:
Thoughts on the infamous Debian Openssl patch?
Click to expand...
Interesting read, thanks.

It's quite revealing that, while the Debian maintainer patched the code, after taking advice from an OpenSSL dev, it almost seems like those questionable lines of code in OpenSSL were more to blame for this than anything else. In spite of this, the Debian project have been taking the blame for this ever since, along with insinuations of back doors, etc. This takes me back to OpenSSL bug https://cve.mitre.org/cgi-bin/cvename.cgi?name=cve-2014-0160 and the loss of confidence which came from that, followed by forks.
 
So... reading through the thread, anger seems to be a common topic. People seem to get angry enough to:
  • Find a scapegoat. Hey, I didn't do it, here's who did what, and this is why they are to blame. Yeah, that sows distrust.
  • Fork the code. Hey, it's Open Source, what's wrong with just one more option that is incompatible with all others, and creates chaos? (Hint: too many options, too little documentation/standards)
  • Quit the project (after burnout)
  • Do something that amounts to smoke and mirrors in the code itself, and maybe gets attention.
  • Make quick, off-the-cuff, slanderous commentary pretending to be informed on what actually happened.
Maybe ignorance is bliss?
 
I would say there is only one significant problem with xz-utils/liblzma: a malicious actor infiltrated the project at the top level and implanted a backdoor.

The first, related problem is that some Linux people had implemented a systemd specific kludge to OpenSSH for systemd interoperability.

The next possible related problem is that systemd, or to be more correct libsystemd, link to liblzma for whatever reason.

The first one could have happened to any project.

The second and third are quite arguably issues surrounding the quality of code in larger Linux centric projects.
 
blackhole said:
The next possible related problem is that systemd, or to be more correct libsystemd, link to liblzma for whatever reason.
Click to expand...
That looks like the compressed binary logs, and the lib is in this case the "everything and the kitchen sink" place. The lib is used in all the components, and needs to support all components. That is an architectural problem, and not many would see or acknowledge this. Did I mention that pid1 ought to be linked statically?
 
gladiola said:
Thanks to whomever built this xz right for FreeBSD.


https://snyk.io/blog/the-xz-backdoor-cve-2024-3094/
Click to expand...
This has got nothing to do with "built it right for FreeBSD." The exploit inserted a blob (key) that made use of the fact that sshd on Linux systems using systemd, needed some way for sshd to be managed by systemd. OpenSSH needed to be linked with systemd libraries, one of which was liblzma (where the exploit was put). Any system whether BSD or Linux that did not use systemd was not affected.

The reason they chose this was simple. Obfuscation. Who looks at binary blobs? And a binary blob in a compression library would not appear to a casual observer to affect sshd. Quite ingenious. It almost worked.

Unless FreeBSD is used to host a high value service, FreeBSD is probably below the radar. OTOH, given that there are many millions more Linux servers out there, chances that a Linux machine would be hosting a high value opporunity would be greater, not to mention compromise almost every Linux server on the planet would put the attackers in a considerable position of power.

Most FreeBSD systems are below the radar only because there are so few of them in comparison.
 
patmaddox said:
It's worth noting that FreeBSD has some blobs checked directly into the source tree, under these drivers (the *.o files):
I haven't looked closely at them (and 99% most likely won't). Just something that I saw the other day that caught me by surprise, particularly in light of the xz attack.
Click to expand...
I looked at the first one. It's a driver for a RAID controller provided by the manufacturer:
github.com

Import new version of the HPTMV driver from Highpoint. The major change · freebsd/freebsd-src@d2bd3ab

here is the support for amd64, as well as possible support for PAE. Many thanks to Highpoint for continuing to support FreeBSD. Obtained from: Steve Chang @ Highpoint MFC After: 3 days.
github.com github.com

It was originally provided as an object file uuencoded (wtf?) into a text file that had to be decoded at build time. It was uudecoded and checked in as a binary file by Warner Losh before the 14.0 release:

⚙ D29635 hptmv: use .o files directly

reviews.freebsd.org reviews.freebsd.org

None of those drivers appear to be in the GENERIC config, so you'd have to add them manually before being exposed to any shenanigans in those object files.
 
Jose said:
None of those drivers appear to be in the GENERIC config, so you'd have to add them manually before being exposed to any shenanigans in those object files
Click to expand...

Thanks for looking into that: That’s what I was wondering, what you’d need to do to exclude them - and as you say, they’re opt-in.

As for the provenance… can’t say I care. I saw those same details, provided by vendor and committed by a trusted member. But that’s the difference between blobs and source code - source code makes it more practical to “trust, but verify.”
 
You must log in or register to reply here.
Back
Top