Following up from the (now closed) rolling-reboots thread. Per Lillian’s note, our host appears to be on the 4-node chassis in GRU whose power supplies failed. We received the hardware-failure email for it.
App: pzzb-api-db (Legacy Fly Postgres, single node)
Machine: d8946d9c4d97d8 / host_status: unreachable since 2026-10-16 ~18:01 UTC
Volume: vol_4y5ml592895wlojr (zone 3575)
Also on the same failed hardware: pzzb-api machines e28654e12cd686 and 781796dc577278, and pzzb-demo machine 7847923f360948
We’ve already restored service elsewhere, so this is not urgent for availability. But the volume holds ~8.5h of production data that exist nowhere else.
Requests:
Please do not wipe or decommissionvol_4y5ml592895wlojr during the repair.
When power is restored, could you take a snapshot of the volume (or let us know so we can do it)? That’s all we need to recover the missing data.
An ETA for the chassis repair would also be appreciated.
@hallenmaia, what do you see currently at https://fly.io/status ?
(They usually explicitly say “irreperable” or similar when they know a physical host can’t be recovered. See the screenshot in the following older thread for an example:)
Thanks! Our dashboard status still shows only “hardware failure… down for an extended period” (18h ago), no mention of irreparable. Given Lillian’s note about the power supplies on the 4-node chassis, we’re assuming the disks are fine and the host is awaiting repair. Would appreciate any confirmation/ETA from the Fly team.
Yes we are working with our data centre provider to replace the power supply units and get these hosts back online. We do not have a firm ETA to recovery at this time, we are waiting for PSUs to be delivered to the DC.
This is our current expectation as well, though with the PSU down we can’t 100% confirm the current disk state. Assuming they’re fine, once the PSUs are replaced, we’ll bring the hosts up and volumes + machines on them will then become available as normal.
Any further updates for those hosts will be communicated to users with impacted machines/volumes via their personal status page at https://fly.io/status
Our app emissornac was affected by the ongoing GRU hardware incident 6038gl0rp3ml147k, reported since 2026-10-16 22:03:21 UTC.
We restored service on a healthy host from the latest available snapshot (2026-10-16 16:56:47 UTC). The recovered app is healthy, but the original volume is still unreachable and we need to compare any records written after that snapshot.
Original volume: vol_rkg8j9mlwdwgo8w4
Region / zone: gru / 2cea
Volume name: emissornac_recovery_260811
Former Machine: 7811d932b96568
The former Machine was deliberately destroyed after the replacement was validated, to avoid simultaneous processing when the failed host returns. The original volume was NOT deleted and remains listed in our app. No restore or overwrite was applied to it.
Could you please confirm:
The current recovery status and ETA for this host and volume.
That the original volume and all recoverable data/snapshots will be preserved without deletion, reinitialization, or replacement with an older snapshot.
Whether access to the original contents, or a recovered copy on another healthy host, can be provided for an isolated read-only comparison.
Please do not modify our currently running replacement Machine or its volume. We are requesting recovery of the original data, not a rollback of the recovered production system.
The general support@fly.io address returned an automated message directing accounts without paid support here. No credentials, certificates, database files, or customer records are attached.
Fly team, our dashboard reports hardware failures affecting vibo-staging-artifacts in GRU. At 2026-10-19 15:19:59 UTC, the native full Machines inventory returned 34 Machines; these five required image-retention Machines still had host_status=unreachable and no full config:
They are intentionally stopped image-retention holders, including an image from an existing trial. Four of the original nine affected holders have recovered. All nine required holders remain present. The incomplete inventory prevents our normal hosting-cost worker from producing current authoritative spend evidence, so our invite-only beta trial remains stopped.
Can Fly restore these hosts, or perform provider-managed migration while preserving the existing Machine IDs, image digests, full configuration and metadata? Please preserve the intended stopped state and do not delete, recreate, scale, redeploy, change image pins or run application workloads as a workaround. If same-ID recovery is not possible, please tell us the available preservation procedure and recovery estimate before taking a destructive action.
Our beta deadline is October 20. This request approves no paid support subscription, additional resources or billable recovery work. No credentials or user data are included.
Update (2026-10-19 15:46 UTC): authenticated, read-only registry queries successfully returned the exact immutable image manifests associated with all five holders, including the original trial image. Image layers were not downloaded or independently verified, so this is not a claim of a complete image backup. No holder, pin, image or workload has been changed.
this request has a lot of extraneous LLM text and I do not understand what your actual need is (it appears to be a XY problem).
if what you are looking for are for docker images uploaded to registry.fly.io to not be deleted, there’s no need to worry as we only delete images when an app is deleted; having or not having machines in said app makes no difference.
Hi @Sam-Fly, quick check-in now that it’s a business day: have the replacement PSUs arrived at the DC yet, or is there a rough ETA for bringing the chassis back up? We’re stable on our side and just waiting to recover the data on vol_4y5ml592895wlojr. Thanks!
Thanks—that clarifies image retention. Our images are available. The remaining problem is that the Machines API lists these five stopped Machines as unreachable with empty configs, so our inventory/cost process cannot complete. Can Fly restore their API configs, or advise how to clear the failed Machine records while preserving the app and registry images? Are they waiting for the GRU hardware repair, and is there an ETA? Please advise before changing anything; no deletion or paid recovery is authorized.
quick update on this: we should be able to perform the required maintenance on these tomorrow and hopefully bring the servers back up. we’ll post in this thread once we have any further information.
Thanks for the update. We have now force-deleted only our five failed Machines through the Machines API; each returned HTTP 200. The default list omits them, but include_deleted=true still shows all five as destroying. Their image manifests remain available and the app is unchanged. Is there a supported way to finish this cleanup before the host repair, or must these destroying records wait for the hardware to return?
Update: the chassis is back and our host is reachable again, thanks! The volume looks intact (fsck: clean).
However, our Legacy Postgres machine d8946d9c4d97d8 (app pzzb-api-db) can’t boot. Since ~22:03 UTC it’s been crash-looping, every start panics with:
failed to verify cluster state Get "https://consul-iad-13.fly-shared.net/v1/kv/pzzb-api-db-g3zmqx5204z79dlp/INITIALIZED": dial tcp: lookup consul-iad-13.fly-shared.net ...
Looks like internal DNS isn’t resolving from this host. Still happening as of 22:35 UTC. Could you check networking/DNS on it? We only need Postgres up long enough to take a dump, and please don’t modify or wipe vol_4y5ml592895wlojr. Thanks!
Final update: the Consul DNS issue cleared on its own around 22:54 UTC, Postgres came up, we took a dump and have fully recovered the missing data. For anyone else on this chassis seeing the same failed to verify cluster state … consul panic after the host returned: in our case it resolved by itself after ~50 minutes, no changes needed.
Thanks @lillian and @Sam-Fly for the updates and for preserving the volume, much appreciated!