Hi everyone,
The Postgres I deployed on Fly is consistently giving the following error today, and fly’s internal apps are not connecting properly:
2022-09-21T12:43:44.607 app[79135861] sjc [info] sentinel | 2022-09-21T12:43:44.607Z ERROR cmd/sentinel.go:1009 no eligible masters
2022-09-21T12:43:45.195 app[79135861] sjc [info] exporter | INFO[2365] Established new database connection to "fdaa:0:737f:a7b:ad0:1:f80d:2:5433". source="postgres_exporter.go:970"
2022-09-21T12:43:45.195 app[79135861] sjc [info] exporter | ERRO[2365] Error opening connection to database (postgresql://flypgadmin:PASSWORD_REMOVED@[fdaa:0:737f:a7b:ad0:1:f80d:2]:5433/postgres?sslmode=disable): dial tcp [fdaa:0:737f:a7b:ad0:1:f80d:2]:5433: connect: connection refused source="postgres_exporter.go:1658"
2022-09-21T12:43:46.195 app[79135861] sjc [info] exporter | ERRO[2366] Error opening connection to database (postgresql://flypgadmin:PASSWORD_REMOVED@[fdaa:0:737f:a7b:ad0:1:f80d:2]:5433/postgres?sslmode=disable): dial tcp [fdaa:0:737f:a7b:ad0:1:f80d:2]:5433: connect: connection refused source="postgres_exporter.go:1658"
2022-09-21T12:43:50.375 app[79135861] sjc [info] sentinel | 2022-09-21T12:43:50.374Z WARN cmd/sentinel.go:276 no keeper info available {"db": "2e3aca53", "keeper": "22951f80f2"}
2022-09-21T12:43:50.376 app[79135861] sjc [info] sentinel | 2022-09-21T12:43:50.376Z ERROR cmd/sentinel.go:1009 no eligible masters
I did the following but it did not resume running:
-
scale count 0
andscale count 1
-
Re-deploy Postgres project
-
Follow this thread for troubleshooting
-
Use
fly proxy
and try to connect locally, failed
=== Keepers ===
UID HEALTHY PG LISTENADDRESS PG HEALTHY PG WANTEDGENERATION PG CURRENTGENERATION
22951f80f2 false fdaa:0:737f:a7b:2295:1:f80f:2:5433 false 4 3
ad01f80d2 true fdaa:0:737f:a7b:ad0:1:f80d:2:5433 false 1 0
The database is set to
dedicated-cpu-1x 2gb
. A total of 40GB of disk abd was used 2.5GB.
Now use fly status
to view the status still visible service is in state running
and has 1 instance ( running (failed to co)
). Now what should I do to get Postgres back up and running?
UPDATE: Running again
I found that Fly Postgres does not work correctly at 1
node, I scaled to 0
and then to 2
and the database started running again.
The confusion is,
- I didn’t change any configuration before the issue occurred.
- I’ve had some other Postgres running on Fly for a while now as well, and they all seem to be
1
node and running well.
Please let me know if there is anything missing.