Just got this a few minutes ago:
```
Some of your apps in gru region are on a host which has suffered a hardware failure and will be down for an extended period. You are not being charged for this resources on this host. You may await the host’s return to service, or you may spin up new Machines in the region and restore from backups.
```
Glad to hear it, @ohnoimdead! Yeah, multi-hour, global outages like this are rare lately, but, typically, they don’t jump into the fray in the forum while one is in progress…
(Fundamentally, the same people who are fixing the problem are also responsible for communicating about it. That’s usually an advantage, since you get the unfiltered version. During emergencies, though, it has downsides.)
The Infra Log should eventually have an explanation of why a sudden update was needed, once it rolls around 7–10 days from now.
I understand small teams and getting by with what you have. The issue for us, in this instance, is two-fold: 1. the first three outage notifications said expected host downtime was roughly 15-20 minutes, 2. no instructions were given for what to do if that time window was exceeded. “Just hang in there” only works for the first 20 minutes or so. After that, we need to get our service back up. I could have restored our host and volume into a new region if I had some assurance it wouldn’t get rebooted as well. We were left in a limbo of information, which isn’t cool.
I wanted to share an update on the fleetwide reboot that we kicked off earlier today.
By now there’s a good chance you’ve heard about PwnEscape but, if not, it’s a KVM guest-to-host escape vulnerability that was disclosed this morning. As soon as we learned about the vulnerability and understood the risk it posed to our customers we had to make the call: unannounced downtime or risk the speed with which an LLM-augmented attacker could attempt to exploit the platform. We chose the path of safety, but that came with an unavoidable cost.
We’re sorry: this wasn’t a problem with our platform but downtime is downtime, regardless of the reason, and we want to make it right. We’ll have more to say about how we rolled a 1200-server fleet but, for now, I’ll say I’m proud of how our engineers sprang into action.
I’m sorry for the impact this caused and I wish that we had more time to have rolled out this fix without disruption. But we had to choose: safety over availability. We chose safety first.
Thanks for the update and nice work springing into action! The decision to patch and reboot hosts immediately makes sense. If you are open to feedback: I think the through-line on the customer experience (at least from this thread) had less to do with the downtime and was more about proactive comms. What should we expect? What do we do? If you say hosts will be down 15-20 minutes and it’s more than double that, then what? For us, a “if your hosts aren’t back up you can redeploy, we aren’t scheduling VMs to vulnerable hosts” message would have been very useful. That alone would have saved us ~30 minutes of overall downtime.
Also, a heads-up notification would have also saved us scrambling. We have external monitoring so we knew the site was down immediately, however, if that had alarmed with an accompanying email from Fly, we would have had more reassurance, if nothing else.
Update from our side: we received the hardware-failure notice for this host and have already restored service elsewhere from an earlier backup, so we’re no longer down.
However, volume vol_4y5ml592895wlojr (app pzzb-api-db, GRU, zone 3575) holds ~8.5 hours of production data that exist nowhere else.
Two requests for the team handling this host:
Thanks, and thanks for the explanation about PwnEscape.
Thanks Scott, we appreciate it.
It’s understandable the team was under a lot of pressure; what didn’t feel quite right in my case was that the expected down time per host was in the order of 20 minutes. So after a couple of hours of the host that had our machines and volumes being down, I did not have a way to know if this was to be expected as a “slower” rollout or if it was a different issue. And if it was a different issue, were you aware of it?
In any case, I appreciate the response and the work you’ve done to keep things safe. It was the right decision.
hey, I’m on the infrastructure team at Fly and I can try to explain a bit about what happened last night at a process/technical level, and what we’re doing about it.
firstly, for anyone who wanted an advance notification: we published a notification on our status page around 40 minutes before the first hosts started getting rebooted. you can subscribe to get emails, SMS or slack messages when we post updates there - there’s a button on the top right.
this event was a first for us. we’ve done planning and tabletops on a fleet-wide reboot, and had some idea of what could go wrong, but the only way to know for sure was to do it. some things went much better than we expected, some things much worse… such as the 4-node chassis in gru whose power supplies coincidentally died during this incident.
we came up with a lot of processes and tooling in the moment that will definitely be standardised and improved in case this ever happens again (though, of course, we hope it won’t). a big part of this is communication, for sure: we could’ve been clearer that the 15-20 minutes was an estimate, and we could’ve notified customers when a host didn’t come up right and was going to take longer to get fixed.
the ideal state is one where we can migrate Machines to patched hosts before rebooting the old ones, and customers would see no more downtime than regular maintenance.
unfortunately this was/is not possible today for a few reasons - like most other infrastructure vendors, we’ve been struggling to keep adequate spare capacity available in every region, which I’m sure some of you have noticed; and machine migrations are still fairly expensive - but our teams have been working hard at getting us to a much better place.
like Scott said, this is not the kind of service we want to provide, even - and especially - for urgent security maintenances. we hope that in the future, keeping your apps safe will be as transparent to you as any other part of operating the platform.
Hi folks! At this point the rolling restarts are complete. Given that Scott and Lillian chimed in with an explanation of why this process was necessary on short notice, and some infrastructure-side details about the process and timeline, I’m going to close this thread to further comments - if you’re still experiencing issues, it’s best if you create a separate post. We’re better able to help that way, as opposed to having a mega-thread with possibly (but possibly not) issues, related (or maybe not anymore) to the rollout and that may be different things at this point.
The intent here is to keep things topical and organized, never to curtail any conversation about the PwnEscape situation and its handling, so please feel free to open new posts about that as well - but as mentioned above, it’s best if a thread that was specifically about technical issues in a single region during the rolling reboots doesn’t veer into unrelated territory.
Cheers!