Opening this question in case anyone
else is in the same boat (having their
Postgres instance killed …)
Fly.io App Disabled
We received the following email from Fly.io:
Hi. We’re in the process of removing Nomad from Fly.io. This means any apps that haven’t been upgraded to Apps V2 will no longer function. Your app authprodb is one of those apps. Here’s what this means for you: 1. authprodb is currently disabled. - That means the Nomad instances (virtual machines) that ran your app have been shut down, but that the app’s configuration still exists. 2. Your app is technically now on the “machines” platform, rather than “nomad.” - That means that any subsequent fly deploy on your app will attempt to launch Fly Machines. - In many cases, a deploy will immediately work. In some cases, there may be some additional configuration required to get your app running on Machines. The community forum is a great place to get help troubleshooting your deployment if you get stuck. 3. Nothing has been deleted. - Your app’s Docker image still exists, and any Volumes that were attached to your app still exist. Thanks for bearing with us throughout this whole migration saga. It’s not been easy, but the results should soon speak for themselves. If you have any questions or comments, we’ve posted a similar announcement on the community forum. Thanks, Your Friends at Fly.io
They key point of the email:
they shut down our
and according to point 3 “Nothing has been deleted”.
So I dived into trying to recover the database.
Tried re-deploying the
authprod App that was attached to the
authprodb Postgres instance:
fly deploy --verbose
It errors because the
Elixir App cannot run the
mix release command:
Error: release command failed - aborting deployment. error release_command machine 48ed037a343368 exited with non-zero status of 1
Full log output in: Internal Server Error 😢 🔥 · Issue #325 · dwyl/auth · GitHub
After a couple of hours searching and trying everything I could think of with zero progress.
I’m opening this forum topic.
Viewing the volumes for the
authprodb, they have
0 MB i.e. no data
Luckily, we have a working “staging” version of the
auth App also hosted on Fly.io: https://authdemo.fly.dev/
and can easily compare the Volume:
https://fly.io/apps/authdemo-db/volumes we see that the
authdemo-db is using
182 MB the available volume:
This suggests that our data was in fact deleted when Fly.io disabled our App.
If anyone has seen this issue and can help us recover our
Postgres data, please comment!