Lost access to postgres: Error no active leader found

Fly failed to charge my credit card, and it took a couple of days until I fixed it up. I’m not sure if this is related, but after this I can no longer connect to the attached database. I can SSH into the apps, but if I try to connect to the database, I get:

FATAL:  role "XXXXXX" is not permitted to log in

When I look at my dashboard overview, I can see the database (openmanifest-staging-db) running on a machine with ID 73287317a15385, but using the CLI, I can’t find any machines:

➜ flyctl machines list -a openmanifest-staging-db
No machines are available on this app openmanifest-staging-db

If I try to restart that machine, I get:

➜ fly machine restart 73287317a15385                           
Error could not get machine 73287317a15385: failed to get VM 73287317a15385: machine not found

This is strange, because this machine shows up on my dashboard as running, yet it’s nowhere to be found in the CLI?

If I try to connect to postgres, I get:

➜ fly pg connect -a openmanifest-staging-db
Error no active leader found

What’s happening here, and how can I restore access to my databases?

2 Likes

Machine restart worked in my case:

fly machines list --app <pg-app-name>
fly machines restart <machine-id> --app <pg-app-name>

JFYI, checks list looked like following:

fly checks list -a <pg-app-name>

Health Checks for
NAME | STATUS | MACHINE | LAST UPDATED | OUTPUT
--------------------------------------------------------------------------
pg | warning | | 19h5m ago | waiting for status update
--------------------------------------------------------------------------
role | warning | | 19h5m ago | waiting for status update
--------------------------------------------------------------------------
vm | warning | | 19h5m ago | waiting for status update
--------------------------------------------------------------------------

2 Likes