Postgres having issues: `cannot create kv store`

,
1

We are seeing very severe errors in our Postgres cluster right now. I’ve researched past posts and it seems like it’s typically due to errors on the Fly.io side. Can someone please assist? Our entire service is failing at the moment.

cmd/keeper.go:2118	cannot create keeper: cannot create store: cannot create kv store: Put "https://consul-iad-5.fly-shared.net/v1/catalog/register?wait=5000ms": EOF
2023-12-07T00:52:58Z app[3d8d0e6b0d3189] sjc [info]sentinel | 2023-12-07T00:52:58.496Z	FATAL	cmd/sentinel.go:2030	cannot create sentinel: cannot create store: cannot create kv store: Put "https://consul-iad-5.fly-shared.net/v1/catalog/register?wait=5000ms": EOF

Here is an example of a prior issue with the same error messages: Postgresql "cannot create sentinel: cannot create store: cannot create kv store"

2

We replicated our DB and reattached to the server. It was working briefly until we issued a restart to try to change concurrency limits (which has never seemed to work very well). After the restart, we began seeing the same errors again and everything is failing yet again.

3

There are a couple of other reports of this now, also in iad-5:

https://community.fly.io/t/issues-connecting-to-consul-with-litefs/17167

4

We’re looking into this now, thanks for bringing it up.

  • Daniel
5

I believe we found and resolved the issue with consul-iad-5. Please let me know if you continue to have issues with it.

6

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.