postgres crashed and unrecoverable (Error: no active leader found)

[fubhy]

This happened out of the blue yesterday at 2023-08-09T13:50

2023-08-09T13:50:23Z app[4d89623b00e087] ams [info]panic: error checking stolon status: cannot create kv store: Put "https://consul-fra-5.fly-shared.net/v1/catalog/register?wait=5000ms": dial tcp [2a09:8280:1::c05a]:443: connect: connection refused
2023-08-09T13:50:23Z app[4d89623b00e087] ams [info]: exit status 1
2023-08-09T13:50:23Z app[4d89623b00e087] ams [info]goroutine 17 [running]:
2023-08-09T13:50:23Z app[4d89623b00e087] ams [info]main.main.func2(0xc000150000, 0xc0000a4000)
2023-08-09T13:50:23Z app[4d89623b00e087] ams [info]     /go/src/github.com/fly-examples/postgres-ha/cmd/start/main.go:81 +0x72c
2023-08-09T13:50:23Z app[4d89623b00e087] ams [info]created by main.main
2023-08-09T13:50:23Z app[4d89623b00e087] ams [info]     /go/src/github.com/fly-examples/postgres-ha/cmd/start/main.go:72 +0x43b
2023-08-09T13:50:23Z app[4d89623b00e087] ams [info]Starting clean up.
2023-08-09T13:50:23Z app[4d89623b00e087] ams [info]Umounting /dev/vdb from /data
2023-08-09T13:50:24Z app[4d89623b00e087] ams [info][    3.539215] reboot: Restarting system

I am unable to restart the db since then.

[Elder]

yesterday at approximately that time there was at least one host down in AMS region, I lost one of my postgres instances to it. I don’t know if this is the cause in your case. That instance didn’t recover, but other instances in the cluster picked up the workload

This is what I had on my dashboard: