An introduction to supply chain attacks
Published on 2 May 2024 by Andrew Owen (4 minutes)
On March 28 Andres Freund discovered malicious code in the XZ Utils package that could have compromised the security of around half the servers on the internet. The attack was audacious in its scope, planning and timescale, leading many to speculate that it was conducted by a state agency. What’s really terrifying is that it was discovered by accident by a database developer. Security researchers failed to spot it.
“This might be the best executed supply chain attack we’ve seen described in the open, and it’s a nightmare scenario: malicious, competent, authorized upstream in a widely used library."—Filippo Valsorda
So what is a supply chain attack? It’s when a trusted outside provider or partner is compromised for the purpose of accessing your systems and data. Attackers target both commercial and open-source software. There is also a concern that software originating from certain countries may contain government-mandated malicious code.
The XZ Utils attack targeted x86-64 Linux variants derived from Red Hat and Debian (including Ubuntu). Because it was added to the released binaries and not present in the source code, 32-bit and non-Intel architectures were not affected. Crucially, versions of Linux that did not escalate the privileges of XZ Utils by binding it to OpenSSH were also unaffected. In practice, only a few distributions were affected before it was spotted. The code would remain dormant until a specific encryption key was used, at which point it could completely compromise the system. For the curious, Ars Technica has a more detailed description.
But the reason it got as far as it did is because XZ had a single unpaid maintainer who was losing interest in the project. A coordinated campaign of social engineering over a period of years gained the maintainer’s trust and ultimately resulted in a fake account being added as a second maintainer and eventually effectively taking over from the original maintainer. After the exploit was added, more fake accounts were used to push for the compromised code to be included in the next release of various Linux distributions.
This should underline what security researchers have always known: people are the weakest link. As Thom Holwerda wrote: “While we have quite a few ways to prevent, discover, mitigate, and fix unintentional security holes, we seem to have pretty much nothing in place to prevent intentional backdoors placed by trusted maintainers. And this is a real problem.” His proposed solution is to set up a foundation dedicated to providing maintainers who are dealing with non-code related issues with help, advice, and possibly even financial and health assistance.
It’s now over a decade since the Heartbleed security bug in OpenSSL was discovered. The industry reaction was to set up the Core Infrastructure Initiative as part of the Linux Foundation to fund at-risk projects. In August 2020 it was replaced by the Open Source Security Foundation. But the focus is on security components like OpenSSH. Since 2018, we’ve seen supply chain attacks such as Event-stream, ASUS, SolarWinds and Mimecast. Had it been successful, XZ would have dwarfed them all.
During my time in the security industry, I saw a pivot from intrusion prevention to intrusion detection. If XZ had gone undetected, monitoring systems would have been crucial in mitigating against it. Fortinet, one provider of such software, recommends these preventative measures:
To that, I would add: adopt a zero trust security model and set up redundant secure backups (and do regular test restores).
Image: Original by Claudio Schwarz.