Emergency Maintenance Affecting My App - Can I Migrate or Do I Need to Wait?

[valehelle]

Hi,

One of my apps (in LHR region), which hosts my database, is currently undergoing maintenance. When I visit the app page, I see the message: “We are performing emergency maintenance on a host that some of your app instances are running on. Apps may be unavailable until the maintenance is completed.”

Is there a way for me to clone the app to restore functionality, or do I need to wait until the issue is resolved?

[mayailurus]

Hi… For Managed Postgres, you would just need to file a support ticket.

(Look in “Managed Postgres” versus “Legacy Postgres” in the dashboard, to see which you have.)

Legacy Postgres, in contrast, was always intended to be managed by the user, although not everyone realized that when they were signing up. Basically, you are your own DB sysadmin. If this is multi-node Legacy Postgres then you should, with that hat on, be able to recover immediately, since the surviving nodes still have all the data.

For single-node Legacy Postgres, you might be able to recover from the streaming backups, if you had those enabled. If not, then there’s unfortunately a choice between (a) waiting in the hopes of successful hardware repair or (b) getting back up and running now with some data loss. The automatic volume snapshots occur every ~24 hours, but if you have a more recent pg_dump, or similar, then you should use that instead.

These maintenance periods can last a very long time—and may end with the entire physical host machine being lost in the end. (The Fly.io infrastructure staff can’t always tell the timing or the probable outcome in advance.)

In general, the single-node Legacy Postgres databases shouldn’t be used for production on the Fly.io platform, although obviously the temptation is understandable.

@john-fly

But many users use single-machine Fly PG for production! Please don’t do this – it’s not a supported configuration! I’ve had to break the news to several users recently that their data was lost/may be lost when a host goes offline for a (suspected) hardware failure, taking the data with it. Always run a Fly PG cluster with 3 or more nodes in production.

Single-node databases other than Fly Postgres would typically have similar caveats.

Hope this helps!

[valehelle]

So what I finally did was to create a new postgres app using yesterday backup. This link help a lot Troubleshoot apps when a host is unavailable · Fly Docs