Since approximately 18:00 UTC on October 16, two regular Machines with attached volumes in GRU have been unreachable, causing a production outage:
Machine
Volume
287e356f0de718
vol_vje06q75566nmqp4
48e7453c371318
vol_vly5w8d0x25dzmm4
Both Machines report state: started and host_status: unreachable. Both volumes report zone 3575 and an unreachable host.
Volume listings show attached_machine_id: null, although the Machine configurations still reference their respective volumes mounted at /data. We haven’t recreated or modified these resources.
A third Machine, 287e353cd20138, also reports started with an unreachable host.
As of 19:09 UTC, these resources remain unavailable—well beyond the initially announced 15–20-minute per-host interruption.
Could you confirm whether their hosts are still undergoing maintenance, provide an ETA, and advise whether the existing Machines and volumes can be recovered or migrated while preserving their latest contents?
Is the missing volume attachment information expected while the host is unreachable?
App: corinthio-crm
Machine: 7817965fd4e398 — “host could not be reached” (HostStatus: unreachable)
Volume: vol_re15j62wo5z089o4 (zone f6de)
Proxy: [PU03] unreachable worker host. Restart returns 408 (Request ID: 01M53068Y696ZQXJ3YSXZZYCJQ-gru)
Down since ~18:05 UTC. Other hosts in our org came back after the reboot; this one did not.
This volume holds ~14 hours of production data newer than our last snapshot, and it exists nowhere else.
We are temporarily running on a replacement machine (683d151efd4348) from that snapshot.
Could you please prioritize bringing host f6de back, or share an ETA?
Please do NOT migrate or wipe vol_re15j62wo5z089o4. We need its data back intact.
As a small reminder… People who have a Support plan would get much better results using its dedicated portal, rather than posting here in the community forum.
Questions in the forum only get answered when someone has a bit of free time, which could be quite a while from now (given everything that’s happening today)…
Additional details for Prepped in JNB (Johannesburg):
Prepped production has been unavailable since approximately 18:17 UTC on 16 October. Database app prepped-db, machine d891364b034738 in JNB, reports host_status: unreachable; volume vol_r1ldgk01jxwzngp4 is also unreachable.
Paying users are affected, and downtime has exceeded the incident’s initial per-host estimate. Please confirm the host’s recovery status and ETA, preserving the original volume and latest data.
Also did I miss some communication? Is fly.io hosting for hobbyists or something this sort of operation should have tons of notices going out to customers to prepare for production outages
For us it’s been 2 hours. The updates in the status page do not describe longer rollout/unavailability times so it is impossible to know whether Fly.io’s team are unaware of these issues.
It’s important to post and make it visible. And of course if someone does have a support plan and can point them at this thread, that would be extremely helpful.
My machines have rebooted, and I’ve confirmed they are running and serving traffic locally. However the load balancer fails to serve. From the outage description I would expect serving to recover after reboot, is there another issue?
```
[PR03] could not find a good candidate within 40 attempts at load balancing. last error: [PR01] no known healthy instances found for route tcp/443. (hint: is your app shut down? is there an ongoing deployment with a volume or are you using the ‘immediate’ strategy? have your app’s instances all reached their hard limit?)
```
Down since ~18:01 UTC. Our app machines in GRU already came back on rebooted hosts; only this one didn’t.
Production is fully down for our customers. Our last backup is ~14h old, so we are not restoring from it. Please do not migrate or wipe vol_4y5ml592895wlojr — we need the data on it.
Any ETA for zone 3575 / this host would help a lot. Thanks.
On yyz here and have been down for > 50 minutes. I’m debating redeploying, but worried we’ll end up on a host not yet rebooted (and then be in the same boat). We are also a little disappointed at the comms here.
most of our machines have encountered outages of roughly 14 minutes, but this has been (is) a horrible experience. even multi-region deployments are at the whims of the rebooting sequence, and we have apps down from simultaneous reboots in different regions. redeploying failed machines or pushing new deployments is not an option either because we’ve already experienced deploying on a host that’s not yet rebooted causing yet more disruption.
If you’re seeing a lot longer outage than the promised 15–20 minutes, then it’s probably worth redeploying (in my opinion). Running servers don’t always reboot cleanly, and fixing the rare cases that didn’t come back to life on their own may not be the highest priority, given that the Fly.io Machines platform treats single-host outages as more or less normal.
(Indeed, data that exists only on a single volume can be lost permanently, which is why flyctl always nags people when they’re in that situation, etc.)
Unfortunately I’m unable to destroy our volume and restore it from our litestream backup because the control plane is reporting that our volume is still attached to the instance, even though that instance has been destroyed.
❯ fly -c fly.prod.toml machines list No machines are available on this app cosmic-production
❯ fly -c fly.prod.toml volumes destroy vol_rn80lpxyo8e23ker
Warning! Every volume is pinned to a specific physical host. You should create two or more volumes per application. Deleting this volume will leave you with 0 volume(s) for this application, and it is not reversible. Learn more at https://fly.io/docs/volumes/overview/
? Are you sure you want to destroy this volume? Yes
Error: failed destroying volume: failed to destroy volume vol_rn80lpxyo8e23ker: failed_precondition: volume is currently bound to machine: 807de9ae545d28
Update! I was finally able to destroy the volume and redeploy a new volume and instance, so we are back up for the moment. No idea if Fly is still scheduling instances to non-rebooted hosts, but we’ll see…
Just updated my message above. I was finally able to delete the volume. Switching region was my next option if that continued to fail though. Here’s to runbooks and litestream generally working as advertised. I’m still disappointed in the comms though. And if our uptime alarm goes off again, I’m going to be further disappointed. Our Fly bill isn’t huge but it ain’t zero either.
I’m still disappointed in the comms though. And if our uptime alarm goes off again, I’m going to be further disappointed. Our Fly bill isn’t huge but it ain’t zero either.