database suspended but why?

1

I have the problem that our staging database is marked as suspended in the fly.io dashboard interface. Which I think is the reason why I get strange “ERROR: cannot execute SELECT FOR UPDATE in a read-only transaction” errors.

But I cannot get it out of it, since running “fly pg restart” results in an error:

Error: unknown platform version

And if I hit the buttons “scale” or “machines” I get a 404 in the fly.io dashboard interface. How do I get the server back.

And also: How did it end up there? If that happens to our production database, that would be bad.

And, I would have send this via E-Mail (I actually did), but I am getting this Email reponse now:

Message not delivered

Your message couldn’t be delivered to redacted@support.fly.io because the remote server is misconfigured. See technical details below for more information.

Can anyone help, or point me in the right direction?

2

Hi calamari,
Try running fly apps set-platform-version nomad, and try again. It looks like your app got into a weird state.

3

That results in:

$ fly apps set-platform-version nomad
Error: postgres apps should use fly migrate-to-v2 instead

4

I just tried, what was suggested, which didn’t do anything except printing out help infos, where I see a troubleshoot command.
I used that one fly migrate-to-v2 troubleshoot, which resulted in:

Oops! We ran into issues migrating your app.
We're constantly working to improve the migration and squash bugs, but for
now please let this troubleshooting wizard guide you down a yellow brick road
of potential solutions...
               ,,,,,
       ,,.,,,,,,,,, .
   .,,,,,,,
  ,,,,,,,,,.,,
     ,,,,,,,,,,,,,,,,,,,
         ,,,,,,,,,,,,,,,,,,,,
            ,,,,,,,,,,,,,,,,,,,,,
           ,,,,,,,,,,,,,,,,,,,,,,,
        ,,,,,,,,,,,,,,,,,,,,,,,,,,,,.
   , ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,

The app's platform version is 'detached'
This means that the app is stuck in a half-migrated state, and wasn't able to
be fully recovered during the migration error rollback process.

Fixing this depends on how far the app got in the migration process.
Please use these tools to troubleshoot and attempt to repair the app.
No Apps v2 machines found. Setting platform version to Nomad.

? What would you like to do? List Nomad VMs
Nomad VMs:
Nomad VMs
ID              PROCESS VERSION REGION  DESIRED STATUS                  HEALTH CHECKS           RESTARTS        CREATED              
xxx8d257        app     6       fra     run     running (leader)        3 total, 3 passing      0               2023-06-24T20:03:21Z


? What would you like to do? List Machines
No machines are available on this app xxx-staging-db

? What would you like to do? Autodiagnose issues
Process group issues:
 * 'app' has no machines

VM count issues:
 * 1 more Nomad VMs than machines

To fix this, you can try:
 * running `fly deploy` to create missing process groups,
   then removing existing Nomad VMs and switching to V2

Not quite sure if that helps. I get the option " Destroy existing machines and use Nomad", but honestly I am afraid I loose the Data in the DB if I try that :slight_smile:

5

Hey @calamari, I have just sent you an email regarding that “Message not delivered” error you had.

1 Like
6

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.