Postgres crashed - cannot restart, restore or fork

Questions / Help
1

I have an PG database that has been running for a year or so.
I have had some OOM errors lately, so I decided to login and see what was going on.

It was working fine, but had a lot of errors in the dashboard, so I decided to try to restart it…
Long story short, I cannot get it up, and I think I am kinda stuck.

What I have tried:

fly pg restart -a snowheads-pg
Error: no active leader found

fly status -a snowheads-pg
ID            	STATE  	ROLE 	REGION	CHECKS            	IMAGE                        	CREATED             	UPDATED              
2865600b7e71d8	stopped	error	ams   	3 total, 3 warning	flyio/postgres:14.6 (v0.0.41)	2023-10-30T21:21:28Z	2023-10-30T21:40:04Z	
1857930a413028	stopped	error	fra   	3 total, 3 warning	flyio/postgres:14.6 (v0.0.41)	2023-09-27T18:31:01Z	2023-10-30T21:40:29Z	
148ed031f27608	stopped	error	fra   	3 total, 3 warning	flyio/postgres:14.6 (v0.0.41)	2023-09-27T18:31:02Z	2023-10-30T21:39:51Z	


fly machine restart <machineid> -a snowhead-pg
# Failed after a while for all machines

^ I tired to adding a third machine, without luck…

fly volumes fork -a snowheads-pg
Error: failed to fork volume: failed to create volume: Some of our regions require a paid plan, due to high demand

^Yeah, It seem like my old volume was in FRA, and since I am on a free plan, I am not able to access that region anymore, and therefor not able to fork into another region…

fly postgres create --snapshot-id vs_1Y266pymp1xQesYl
Error:  "Failed to resolve source machine from snapshot."

Currently I feel kinda stuck and struggles to get the database running again or get my data out.
Hope its possible to resolve this somehow. The DB is powering my hobby projects, which isn’t a large site, but has some visitors every day, so it would be nice if it worked…

2 Likes
2

Not sure why, but tried to start up all machines today and it worked.

3

@Andreas1 how did you restart the Postgres instance?

We tried:

flyctl postgres restart -a dbname

But got:

Error: no active leader found

:person_shrugging:

4

What solved it for me was to stop and start the machines running the Postgres instance.
I had two machines that had stopped during the fly postgres restartstep, so I used fly machine · Fly Docs to first list and then start them again.

5

@Andreas1 thanks for the reply. :ok_hand:

Sadly, it appears that the Volume that was attached to our Fly instance that was running our Postgres DB was wiped so restarting doesn’t work …
New forum topic: Where is our Data? (Postgres Database Volume 0MB) :new:

6

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.