Linux

Kinda tired of the constant flow of endless "analysis" of xz at this point.
There's no real good solution to "upstream gets owned by evil nation state maintainer" - especially when they run it in multi-year op.

It simply doesn't matter what downstream does if the upstream build systems get owned without anyone noticing. We're fucked.

Debian's build chroots were running Sid - so they stopped it all. They analyzed and there was some work done with reproducible builds (which is a good idea for distro maintainers). Pushing out security updates when you don't trust your build system is silly. Yeah, fast security updates are nice, but it took multiple days to reverse the exploit, this wasn't easy.

Bottom line, don't run bleeding edge distros in prod.

We got very lucky with xz. We might not be as lucky with the next one (or the ones in the past).

Were systems in the stable branch at risk of compromise? Were there delays in releasing security fixes in the stable branch.

This blog post misses entirely that this has nothing to do with the unstable channel. It just happened to only affect unstable this time because it gets updates first. If we had found out about the xz backdoor two months later (totally possible; we were really lucky this time), this would have affected a stable channel in exactly the same way. (It'd be slightly worse actually because that'd be a potentially breaking change too but I digress.)

I see two way to "fix" this:

• Throw a shitton of money at builders. I could see this getting staging-next rebuild times down to just 1-2 days which I'd say is almost acceptable. This could even be a temporary thing to reduce cost; quickly renting an extremely large on-demand fleet from some cloud provider for a day whenever a critical world rebuild needs to be done which shouldn't be too often.
• Implement pure grafting for important security patches through a second overlay-like mechanism.

Isn't that the risk of running an unstable build of anything?

You could hold off on the latest updates

This means users such as myself who use the unstable branch for all of their packages will still be pulling the (potentially) infected xz tarballs onto their machines!

Yeah dont do that. On any OS that's asking for problems.

Nix & Hydra’s scheduling is super basic. There is room to optimize the builds in many ways. In this case, the fact that xz is in libarchive as well as in input for Nix makes the rebuilds particularly bad.

As of today, NixOS (like most distros) has reverted to a version slightly prior to the release with the Debian-or-Redhat-specific sshd backdoor which was inserted into xz just two months ago. However, the saboteur had hundreds of commits prior to the insertion of that backdoor, and it is very likely that some of those contain subtle intentional vulnerabilities (aka "bugdoors") which have not yet been discovered.

As (retired) Debian developer Joey Hess explains here, the safest course is probably to switch to something based on the last version (5.3.1) released prior to Jia Tan getting push access.

Unfortunately, as explained in this debian issue, that is not entirely trivial because dependents of many recent pre-backdoor potentially-sabotaged versions require symbol(s) which are not present in older versions and also because those older versions contain at least two known vulnerabilities which were fixed during the multi-year period where the saboteur was contributing.

After reading Xz format inadequate for long-term archiving (first published eight years ago...) I'm convinced that migrating the many projects which use XZ today (including DPKG, RPM, and Linux itself) to an entirely different compression format is probably the best long-term plan. (Though we'll always still need tools to read XZ archives for historical purposes...)

Could SUSE's open build system be used an alt to hydra if its a bottle neck for updates?