Postgres database unreachable

[nicolaskb]

Hello,

It looks like my postgres db is unresponsive and un-detachable.

$ fly postgres restart -a <app-name-db>
Error: failed to obtain lease: failed to get lease on VM e286560ea91248: dial tcp [fc01:a7b:10c::]:3593: connect: connection refused

$ fly postgres detach -a <app-name> <app-name-db>
Error: command is not compatible with this image

Any idea how to overcome this issue? Thanks a lot.

[kaelynH]

Hey there,

I see you are having some issues with restarting postgres/it being unresponsive. I wonder if it is a connection/network issue at all? Would you be willing to run flyctl wireguard reset? Here is the doc for reference- fly wireguard · Fly Docs

Also would you mind trying to run- fly postgres detach -app <app-name> <app-name-db> instead of fly postgres detach -a <app-name> <app-name-db> and see if that results in anything different?

Of course if you are worried about any data, I would suggest making a backup/snapshot first. Here’s the doc for that- Backup, Restores, & Snapshots · Fly Docs

[nicolaskb]

Thanks! After running flyctl wireguard reset, I now get Error: No active leader found (with the -a or --app flags).

I realized subsequently that I had this status message:

Service Interruption 5 days ago
Affected Apps:

<app-name-db>

2023-09-01 23:36:17 UTC We are performing emergency maintenance on a host some of your apps instances are running on. Apps may be unavailable until the maintenance is completed.

And the following output for $ fly checks list -a <app-name-db>

Health Checks for <app-name-db>
  NAME | STATUS   | MACHINE        | LAST UPDATED         | OUTPUT                                                                      
-------*----------*----------------*----------------------*-----------------------------------------------------------------------------
  pg   | critical | e286560ea91248 | 2023-09-01T23:46:11Z | context deadline exceeded (Client.Timeout exceeded while awaiting headers)  
-------*----------*----------------*----------------------*-----------------------------------------------------------------------------
  role | critical | e286560ea91248 | 3h2m ago             | context deadline exceeded (Client.Timeout exceeded while awaiting headers)  
-------*----------*----------------*----------------------*-----------------------------------------------------------------------------
  vm   | critical | e286560ea91248 | 2023-09-01T23:46:11Z | context deadline exceeded (Client.Timeout exceeded while awaiting headers)  
-------*----------*----------------*----------------------*-----------------------------------------------------------------------------

I guess the physical machine on which the app was running is down? If so, is there anything at all that I can do to copy the data and set up a new DB?

Thanks again,

[kaelynH]

If you are wanting to copy the data you currently have within your db, and then setup a new one, and migrate the data over. I would take a look at this post here- How copy local postgres db to fly.io?.

You could possibly try to restart your pg clusters as well. To do so you could follow this doc here- fly postgres restart · Fly Docs

Hope this helps!