Postgres database down... again.

[foocux]

I’m experiencing the same issue that I had 18 days ago with the same database, for the record, https://community.fly.io/t/fly-pg-suddenly-went-down/11757.

This is the error that I’m getting:

2023-04-14T16:35:16.846 app[e286279ae51686] mia [info] sentinel | 2023-04-14T16:35:16.845Z ERROR cmd/sentinel.go:1018 no eligible masters

When I tried to restart the database using fly pg restart and fly machines restart it fails with these errors:

Error failed to obtain lease: failed to get lease on VM 32874755b13d85: request returned non-2xx status, 504

Error could not stop machine 32874755b13d85: failed to stop VM 32874755b13d85: request returned non-2xx status, 504

Last time I solved it because the restart commands worked after an hour of downtime.

[lillian]

Hello, we’re currently experiencing an incident in the region one of the instances of your database is running in: Fly.io Status - Network disruption in HKG region.
Failing over databases between regions is not supported. I would recommend adding replicas in the hkg region, though unfortunately it wouldn’t have helped with this incident since the network issues are currently impacting all of our hosts in that region.