Deleted `postgres` database, now PostgreSQL won't start

Questions / Help
1

As the title says, I deleted the postgres database, as I thought it wasn’t needed (flyctl automatically created a database with the name of my app during setup), and now PostgreSQL won’t start.
I’m getting the following output repeated in the metrics, from a cmd/keeper.go file:

FATAL: database "postgres" does not exist

The reason I deleted it was that I was trying to do a pg_restore of a dump from Heroku, which put all the data in the postgres database, which isn’t the one my app is using, so I thought the easiest way to clean up would be to delete the postgres database.

Has anyone done something equally bad, and found a solution? :slight_smile:

I’m in the progress of deleting the database and creating it again which should solve my issue, but I’ll leave this issue here as this might be something someone else runs in to.

That did not work. After deleting the PostgreSQL instance and creating a new with flyctl pg create it won’t start and keeps looping the following messages:

 2022-06-16T19:42:02.371 app[41c69fac] fra [info] keeper | 2022-06-16T19:42:02.371Z INFO	cmd/keeper.go:1504 our db requested role is master

2022-06-16T19:42:02.372 app[41c69fac] fra [info] keeper | 2022-06-16T19:42:02.372Z INFO	cmd/keeper.go:1542 already master

2022-06-16T19:42:02.395 app[41c69fac] fra [info] keeper | 2022-06-16T19:42:02.395Z INFO	cmd/keeper.go:1675 postgres parameters not changed

Looking around, it seems like you are using stolon - could the keeper have gone into a bad state?

It’s the following health check that keeps failing - same thing happens if I delete and create a new pg app:

pg   | critical | 8b6164c1   | fra    | HTTP | 17m36s ago   | HTTP GET http://X.X.X.X:5500/flycheck/pg: 500 Internal Server Error Output: "failed to connect to proxy: context deadline exceeded"[✓]

Okay, I tried creating a new database with a different name, which works fine. So somehow the name I used when it stopped working is in a bad state, and whenever I try to recreate a database with the same name, it fails.

3

No input on this? It seems quite fragile that a unique app identifier will break like this and just stop working for newly created apps with the same name.

When I delete all my apps and do fly launch from the start, it errors out when trying to create the postgres database with the following error:

Error Failed attaching [appname]-db to the Postgres cluster [appname]: 500: context deadline exceeded.\nTry attaching manually with 'fly postgres attach --app [appname] --postgres-app [appname]-db'

Launching the app in another region works just fine, so I guess the name is only broken in the fra region

4

These are normal.

Can you share the name you used? It sounds like there might be lingering internal DNS records for <name>.internal, so apps/proxies aren’t able to find the real members.

5

Thanks, I didn’t find anything weird with these log statements as well - I just didn’t have anything else to go with.
It was named datavin-db and ran on fra region. I’m running on ams now which is just fine though.