On Friday, 29 March 2024, a developer posted to the oss-security openwall mailing list that they had identified a backdoor in the liblzma data compression library. The backdoor can compromise openssh on systems where openssh has been patched to support systemd notifications (Debian and others).
Upon report of the backdoor, we checked the few places in our fleet where we allow OpenSSH to run at all and we confirmed that all installed versions of liblzma are 5.2.5 or lower. Additionally, we do not expose openssh anywhere on the public internet so it would not have been possible for an external attacker to exploit a vulnerable instance.
With all possible respect to the Herculean efforts of the OpenSSH maintainers, who are some of the hardest working people in show business, and acknowledging the (excepting today) otherworldly security track record of OpenSSH as a codebase, we do not trust it because it’s written in a memory-unsafe language. We believe we’ve built our infrastructure not to depend on OpenSSH’s security.
For internal SSH access to resources, we use Teleport, which is a certificate-based implementation of SSH in Golang. The big win for us with Teleport, besides granular access control that keeps us from devolving into a state of “every developer can pop a shell on anything”, is transcript-level logging of sessions. But a secondary win for Teleport is that you can’t backdoor it by infiltrating one of OpenSSH’s dependencies.
For customer SSH access to Fly Machines, we use “Hall Pass”, a Golang implementation of SSH that we ourselves built. If you didn’t go way out of your way to stick OpenSSH on your Fly Machine and expose it as a service on our Anycast network, you don’t have to think about today’s backdoor news (at least with respect to your Fly Machines). Hall Pass isn’t smart enough to get backdoored.
We’ll continue to keep an eye on things but, for now, we’re confident that Fly.io’s systems are not affected by this backdoor and there has been no impact to any users or user data.