Database Suspended, No Leader Found, Volume missing

My app’s database is suspended currently. When I try to restart it, it fails with the following error:
fly pg restart: Error: no active leader found

When I run status, it reports no machines being avail
fly status: No machines are available on this app xxxxx

As a last resort, I went to check if I could make a new pg and move the volume over, but it seems the volume is missing.

What is my path to getting this database back up and running?

Hi… You might be able to use the restore a deleted volume instructions from the fine print.

This shouldn’t have happened without you getting a notice about a failed underlying physical host machine, though, so it might be prudent to double-check that you really were looking in the right place, etc., :thought_balloon:

I don’t know the volume_id to be able to use this method

What was the exact volumes list command that you tried?

fly volumes list -a app_name

Try --all, like in the instructions.

(And also take care that app_name is the Postgres app and not the web app; I make that mistake all the time!)

No dice with --all either!
And yeah, I used to make that mistake a bunch but I triple checked it was right, it was suffix’d with “-postgres” so I’m positive it was the right app

Hm… That’s pretty strange. Personally, I would try also checking from the Web UI dashboard, just in case this is another metadata synchronization glitch. That’s kind of grasping at straws, though.

If you have one of the Support plans, then they may have an internal actions record somewhere that they can trace things through…

Nothing in web UI
After making a new database, my web app couldn’t connect to the new db (even after using the attach command)

I really can’t believe this data was lost. Kind of wild.

Since I needed to get my app back up and running, I tried render.io. Worked the first try, just as cheap/free, much faster deploy process that doesn’t leave you in bad states.