Should we be concerned about all these exploits going on? The Fly team hasn't said anything on Twitter regarding this

Should we stop all our services and wait for it to blow over? Will something like the railway exploit happen on fly as well? What are the dependencies fly has updated so far?

Hi, I’m Mike. I run engineering at Fly. Fair of you for asking, so I’ll tell you how it’s gone.

First, I want to acknowledge it’s been a heck of a week from a security perspective, no matter what company you’re at or what work you’re doing. Phew.

Last Friday, we learned of a KVM vulnerability, and we rebooted our fleet to protect us against an escape where you could potentially own the entire system by breaking through that KVM boundary.

On Tuesday, we patched CA certificates for everything because of the eMudhra CA weak-private-key fiasco.

Wednesday, we vetted the fleet to ensure that we were not running any PHP internally. (Thankfully, we were not, nor do we ship PHP on our Sprite images for customers). Machines load in the image you tell them to, so that doesn’t apply quite the same way.

We also updated Node everywhere, and that includes our applications and our images for sprites.

Finally, since learning about the compromise and attacks at other platform companies, we’ve been diligently going through our systems looking for anomalies to ensure that we’re not seeing anything suspicious. It certainly doesn’t mean we’re 100% secure, but we’re watching.

It’s awesome that you guys are proactively patching and being on the ball with these things.

Perhaps now is a good time to revisit the KMS project you guys were thinking of doing? Currently for secrets management / injection, it’s using the hashicorp vault as mentioned in the post?

Given what has happened in railway, do you recommend against setting crypto wallet private keys as secrets in deployment?

Per the last comment in that topic, it seems the KMS feature moved into the Machines API. The only thing that wasn’t shipped was the filesystem APIs (i.e. encr and decr being available via /.fly/kms.)

These are the relevant APIs that deal with keys: