FRA Postgres DB are dead?

Hi @kurt and @jerome,

I’m stuck with two projects I maintain and that are currently facing issues with the Postgres instances.

Here is the recent logs from one of them:

$ fly logs -a mo....-db

Waiting for logs...

2023-01-19T16:03:38.743 app[3d8dd13a49d789] fra [info] sentinel | 2023-01-19T16:03:38.743Z ERROR cmd/sentinel.go:1852 error retrieving cluster data {"error": "Get \"https://consul-fra.fly-shared.net/v1/kv/mo....-db-l8gnr14vge81y7wm/mo....-db/clusterdata?consistent=&wait=5000ms\": EOF"}

2023-01-19T16:03:58.351 app[3d8dd13a49d789] fra [info] keeper | 2023-01-19T16:03:58.351Z ERROR cmd/keeper.go:1041 error retrieving cluster data {"error": "Unexpected response code: 500"}

2023-01-19T16:04:12.678 app[3d8dd13a49d789] fra [info] keeper | 2023-01-19T16:04:12.678Z ERROR cmd/keeper.go:1041 error retrieving cluster data {"error": "Unexpected response code: 500"}

2023-01-19T16:04:21.279 app[3d8dd13a49d789] fra [info] keeper | 2023-01-19T16:04:21.278Z ERROR cmd/keeper.go:870 failed to update keeper info {"error": "cannot set or renew session for ttl, unable to operate on sessions"}

2023-01-19T16:04:28.161 app[3d8dd13a49d789] fra [info] sentinel | 2023-01-19T16:04:28.160Z ERROR cmd/sentinel.go:1889 cannot update sentinel info {"error": "cannot set or renew session for ttl, unable to operate on sessions"}

2023-01-19T16:04:40.353 app[3d8dd13a49d789] fra [info] sentinel | 2023-01-19T16:04:40.352Z ERROR cmd/sentinel.go:1852 error retrieving cluster data {"error": "Unexpected response code: 500"}

2023-01-19T16:04:44.265 app[3d8dd13a49d789] fra [info] keeper | 2023-01-19T16:04:44.265Z ERROR cmd/keeper.go:1041 error retrieving cluster data {"error": "Unexpected response code: 500"}

2023-01-19T16:04:49.269 app[3d8dd13a49d789] fra [info] keeper | 2023-01-19T16:04:49.268Z ERROR cmd/keeper.go:1041 error retrieving cluster data {"error": "Get \"https://consul-fra.fly-shared.net/v1/kv/mo....-db-l8gnr14vge81y7wm/mo....-db/clusterdata?consistent=&wait=5000ms\": EOF"}

2023-01-19T16:04:52.602 app[3d8dd13a49d789] fra [info] sentinel | 2023-01-19T16:04:52.602Z ERROR cmd/sentinel.go:1852 error retrieving cluster data {"error": "Unexpected response code: 500"}

2023-01-19T16:04:53.768 app[3d8dd13a49d789] fra [info] sentinel | panic: close of closed channel

2023-01-19T16:04:53.768 app[3d8dd13a49d789] fra [info] sentinel |

2023-01-19T16:04:53.768 app[3d8dd13a49d789] fra [info] sentinel | goroutine 699 [running]:

2023-01-19T16:04:53.768 app[3d8dd13a49d789] fra [info] sentinel | github.com/superfly/leadership.(*Candidate).initLock(0xc000136460)

2023-01-19T16:04:53.768 app[3d8dd13a49d789] fra [info] sentinel | /go/pkg/mod/github.com/superfly/leadership@v0.2.1/candidate.go:98 +0x2e

2023-01-19T16:04:53.768 app[3d8dd13a49d789] fra [info] sentinel | github.com/superfly/leadership.(*Candidate).campaign(0xc000136460)

2023-01-19T16:04:53.768 app[3d8dd13a49d789] fra [info] sentinel | /go/pkg/mod/github.com/superfly/leadership@v0.2.1/candidate.go:124 +0xc6

2023-01-19T16:04:53.768 app[3d8dd13a49d789] fra [info] sentinel | created by github.com/superfly/leadership.(*Candidate).RunForElection

2023-01-19T16:04:53.768 app[3d8dd13a49d789] fra [info] sentinel | /go/pkg/mod/github.com/superfly/leadership@v0.2.1/candidate.go:60 +0xc5

2023-01-19T16:04:53.770 app[3d8dd13a49d789] fra [info] sentinel | exit status 2

2023-01-19T16:04:53.770 app[3d8dd13a49d789] fra [info] sentinel | restarting in 3s [attempt 2]

2023-01-19T16:04:56.771 app[3d8dd13a49d789] fra [info] sentinel | Running...

2023-01-19T16:05:03.872 app[3d8dd13a49d789] fra [info] sentinel | 2023-01-19T16:05:03.872Z FATAL cmd/sentinel.go:2030 cannot create sentinel: cannot create store: cannot create kv store: Unexpected response code: 500 (No cluster leader)

2023-01-19T16:05:03.873 app[3d8dd13a49d789] fra [info] sentinel | exit status 1

2023-01-19T16:05:03.873 app[3d8dd13a49d789] fra [info] sentinel | restarting in 3s [attempt 3]

2023-01-19T16:05:06.875 app[3d8dd13a49d789] fra [info] sentinel | Running...

2023-01-19T16:05:13.946 app[3d8dd13a49d789] fra [info] sentinel | 2023-01-19T16:05:13.944Z FATAL cmd/sentinel.go:2030 cannot create sentinel: cannot create store: cannot create kv store: Unexpected response code: 500 (No cluster leader)

2023-01-19T16:05:13.946 app[3d8dd13a49d789] fra [info] sentinel | exit status 1

2023-01-19T16:05:13.946 app[3d8dd13a49d789] fra [info] sentinel | restarting in 3s [attempt 4]

2023-01-19T16:05:16.947 app[3d8dd13a49d789] fra [info] sentinel | Running...

2023-01-19T16:05:24.029 app[3d8dd13a49d789] fra [info] sentinel | 2023-01-19T16:05:24.027Z FATAL cmd/sentinel.go:2030 cannot create sentinel: cannot create store: cannot create kv store: Unexpected response code: 500 (No cluster leader)

2023-01-19T16:05:24.029 app[3d8dd13a49d789] fra [info] sentinel | exit status 1

2023-01-19T16:05:24.029 app[3d8dd13a49d789] fra [info] sentinel | restarting in 3s [attempt 5]

2023-01-19T16:05:27.031 app[3d8dd13a49d789] fra [info] sentinel | Running...

2023-01-19T16:05:34.230 app[3d8dd13a49d789] fra [info] sentinel | 2023-01-19T16:05:34.229Z FATAL cmd/sentinel.go:2030 cannot create sentinel: cannot create store: cannot create kv store: Unexpected response code: 500 (No cluster leader)

2023-01-19T16:05:34.232 app[3d8dd13a49d789] fra [info] sentinel | exit status 1

2023-01-19T16:05:34.232 app[3d8dd13a49d789] fra [info] sentinel | restarting in 3s [attempt 6]

2023-01-19T16:05:37.240 app[3d8dd13a49d789] fra [info] sentinel | Running...

2023-01-19T16:05:44.509 app[3d8dd13a49d789] fra [info] sentinel | 2023-01-19T16:05:44.508Z FATAL cmd/sentinel.go:2030 cannot create sentinel: cannot create store: cannot create kv store: Unexpected response code: 500 (No cluster leader)

2023-01-19T16:05:44.511 app[3d8dd13a49d789] fra [info] sentinel | exit status 1

2023-01-19T16:05:44.511 app[3d8dd13a49d789] fra [info] sentinel | restarting in 3s [attempt 7]

2023-01-19T16:05:47.513 app[3d8dd13a49d789] fra [info] sentinel | Running...

2023-01-19T16:05:54.282 app[3d8dd13a49d789] fra [info] keeper | 2023-01-19T16:05:54.274Z ERROR cmd/keeper.go:1041 error retrieving cluster data {"error": "Get \"https://consul-fra.fly-shared.net/v1/kv/mo....-db-l8gnr14vge81y7wm/mo....-db/clusterdata?consistent=&wait=5000ms\": EOF"}

2023-01-19T16:05:54.757 app[3d8dd13a49d789] fra [info] sentinel | 2023-01-19T16:05:54.754Z FATAL cmd/sentinel.go:2030 cannot create sentinel: cannot create store: cannot create kv store: Unexpected response code: 500 (No cluster leader)

2023-01-19T16:05:54.757 app[3d8dd13a49d789] fra [info] sentinel | exit status 1

2023-01-19T16:05:54.757 app[3d8dd13a49d789] fra [info] sentinel | restarting in 3s [attempt 8]

2023-01-19T16:05:57.757 app[3d8dd13a49d789] fra [info] sentinel | Running...

2023-01-19T16:06:05.001 app[3d8dd13a49d789] fra [info] sentinel | 2023-01-19T16:06:04.999Z FATAL cmd/sentinel.go:2030 cannot create sentinel: cannot create store: cannot create kv store: Unexpected response code: 500 (No cluster leader)

2023-01-19T16:06:05.003 app[3d8dd13a49d789] fra [info] sentinel | exit status 1

2023-01-19T16:06:05.003 app[3d8dd13a49d789] fra [info] sentinel | restarting in 3s [attempt 9]

2023-01-19T16:06:08.005 app[3d8dd13a49d789] fra [info] sentinel | Running...

2023-01-19T16:06:59.282 app[3d8dd13a49d789] fra [info] keeper | 2023-01-19T16:06:59.281Z ERROR cmd/keeper.go:1041 error retrieving cluster data {"error": "Get \"https://consul-fra.fly-shared.net/v1/kv/mo....-db-l8gnr14vge81y7wm/mo....-db/clusterdata?consistent=&wait=5000ms\": EOF"}

2023-01-19T16:07:09.163 app[3d8dd13a49d789] fra [info] sentinel | 2023-01-19T16:07:09.162Z ERROR cmd/sentinel.go:102 election loop error {"error": "Put \"https://consul-fra.fly-shared.net/v1/session/create?wait=5000ms\": EOF"}

2023-01-19T16:07:28.246 app[3d8dd13a49d789] fra [info] sentinel | 2023-01-19T16:07:28.246Z WARN cmd/sentinel.go:276 no keeper info available {"db": "58eecf52", "keeper": "23c31583b2"}

2023-01-19T16:07:29.310 app[3d8dd13a49d789] fra [info] sentinel | panic: close of closed channel

2023-01-19T16:07:29.310 app[3d8dd13a49d789] fra [info] sentinel |

2023-01-19T16:07:29.310 app[3d8dd13a49d789] fra [info] sentinel | goroutine 94 [running]:

2023-01-19T16:07:29.310 app[3d8dd13a49d789] fra [info] sentinel | github.com/superfly/leadership.(*Candidate).initLock(0xc000136000)

2023-01-19T16:07:29.310 app[3d8dd13a49d789] fra [info] sentinel | /go/pkg/mod/github.com/superfly/leadership@v0.2.1/candidate.go:98 +0x2e

2023-01-19T16:07:29.310 app[3d8dd13a49d789] fra [info] sentinel | github.com/superfly/leadership.(*Candidate).campaign(0xc000136000)

2023-01-19T16:07:29.310 app[3d8dd13a49d789] fra [info] sentinel | /go/pkg/mod/github.com/superfly/leadership@v0.2.1/candidate.go:124 +0xc6

2023-01-19T16:07:29.310 app[3d8dd13a49d789] fra [info] sentinel | created by github.com/superfly/leadership.(*Candidate).RunForElection

2023-01-19T16:07:29.310 app[3d8dd13a49d789] fra [info] sentinel | /go/pkg/mod/github.com/superfly/leadership@v0.2.1/candidate.go:60 +0xc5

2023-01-19T16:07:29.314 app[3d8dd13a49d789] fra [info] sentinel | exit status 2

2023-01-19T16:07:29.314 app[3d8dd13a49d789] fra [info] sentinel | restarting in 3s [attempt 10]

2023-01-19T16:07:32.316 app[3d8dd13a49d789] fra [info] sentinel | Running...

2023-01-19T16:07:38.106 app[3d8dd13a49d789] fra [info] sentinel | 2023-01-19T16:07:38.104Z WARN cmd/sentinel.go:276 no keeper info available {"db": "58eecf52", "keeper": "23c31583b2"}

2023-01-19T16:07:43.502 app[3d8dd13a49d789] fra [info] sentinel | 2023-01-19T16:07:43.501Z WARN cmd/sentinel.go:276 no keeper info available {"db": "58eecf52", "keeper": "23c31583b2"}

2023-01-19T16:07:48.764 app[3d8dd13a49d789] fra [info] sentinel | 2023-01-19T16:07:48.763Z WARN cmd/sentinel.go:276 no keeper info available {"db": "58eecf52", "keeper": "23c31583b2"}

2023-01-19T16:07:54.296 app[3d8dd13a49d789] fra [info] sentinel | 2023-01-19T16:07:54.294Z WARN cmd/sentinel.go:276 no keeper info available {"db": "58eecf52", "keeper": "23c31583b2"}

2023-01-19T16:10:41.030 app[3d8dd13a49d789] fra [info] sentinel | 2023-01-19T16:10:41.025Z WARN cmd/sentinel.go:276 no keeper info available {"db": "58eecf52", "keeper": "23c31583b2"}

2023-01-19T16:14:08.750 app[3d8dd13a49d789] fra [info] keeper | 2023-01-19T16:14:08.749Z ERROR cmd/keeper.go:1041 error retrieving cluster data {"error": "Unexpected response code: 500"}

2023-01-19T16:15:53.637 app[3d8dd13a49d789] fra [info] sentinel | 2023-01-19T16:15:53.636Z WARN cmd/sentinel.go:276 no keeper info available {"db": "58eecf52", "keeper": "23c31583b2"}

2023-01-19T16:18:16.303 app[3d8dd13a49d789] fra [info] sentinel | 2023-01-19T16:18:16.303Z ERROR cmd/sentinel.go:102 election loop error {"error": "Unexpected response code: 502 ()"}

2023-01-19T16:18:36.451 app[3d8dd13a49d789] fra [info] sentinel | panic: close of closed channel

2023-01-19T16:18:36.451 app[3d8dd13a49d789] fra [info] sentinel |

2023-01-19T16:18:36.451 app[3d8dd13a49d789] fra [info] sentinel | goroutine 453 [running]:

2023-01-19T16:18:36.451 app[3d8dd13a49d789] fra [info] sentinel | github.com/superfly/leadership.(*Candidate).initLock(0xc00010a070)

2023-01-19T16:18:36.451 app[3d8dd13a49d789] fra [info] sentinel | /go/pkg/mod/github.com/superfly/leadership@v0.2.1/candidate.go:98 +0x2e

2023-01-19T16:18:36.451 app[3d8dd13a49d789] fra [info] sentinel | github.com/superfly/leadership.(*Candidate).campaign(0xc00010a070)

2023-01-19T16:18:36.451 app[3d8dd13a49d789] fra [info] sentinel | /go/pkg/mod/github.com/superfly/leadership@v0.2.1/candidate.go:124 +0xc6

2023-01-19T16:18:36.451 app[3d8dd13a49d789] fra [info] sentinel | created by github.com/superfly/leadership.(*Candidate).RunForElection

2023-01-19T16:18:36.451 app[3d8dd13a49d789] fra [info] sentinel | /go/pkg/mod/github.com/superfly/leadership@v0.2.1/candidate.go:60 +0xc5

2023-01-19T16:18:36.452 app[3d8dd13a49d789] fra [info] sentinel | exit status 2

2023-01-19T16:18:36.452 app[3d8dd13a49d789] fra [info] sentinel | restarting in 3s [attempt 11]

2023-01-19T16:18:39.453 app[3d8dd13a49d789] fra [info] sentinel | Running...

2023-01-19T17:02:09.401 app[3d8dd13a49d789] fra [info] keeper | 2023-01-19T17:00:10.697Z ERROR cmd/keeper.go:1041 error retrieving cluster data {"error": "Get \"https://consul-fra.fly-shared.net/v1/kv/mo....-db-l8gnr14vge81y7wm/mo....-db/clusterdata?consistent=&wait=5000ms\": dial tcp: lookup consul-fra.fly-shared.net on [fdaa::3]:53: read udp [fdaa:0:7b2e:a7b:23c3:83cd:7087:2]:58505->[fdaa::3]:53: i/o timeout"}

2023-01-19T17:07:47.379 app[3d8dd13a49d789] fra [info] sentinel | panic: close of closed channel

2023-01-19T17:07:47.379 app[3d8dd13a49d789] fra [info] sentinel |

2023-01-19T17:07:47.379 app[3d8dd13a49d789] fra [info] sentinel | goroutine 1318 [running]:

2023-01-19T17:07:47.379 app[3d8dd13a49d789] fra [info] sentinel | github.com/superfly/leadership.(*Candidate).initLock(0xc000136000)

2023-01-19T17:07:50.385 app[3d8dd13a49d789] fra [info] sentinel | Running...

2023-01-19T17:07:50.410 app[3d8dd13a49d789] fra [info] sentinel | 2023-01-19T17:07:50.409Z FATAL cmd/sentinel.go:2030 cannot create sentinel: cannot create store: cannot create kv store: Put "https://consul-fra.fly-shared.net/v1/catalog/register?wait=5000ms": EOF

2023-01-19T17:07:50.411 app[3d8dd13a49d789] fra [info] sentinel | exit status 1

2023-01-19T17:07:50.411 app[3d8dd13a49d789] fra [info] sentinel | restarting in 3s [attempt 13]

2023-01-19T17:07:53.472 app[3d8dd13a49d789] fra [info] sentinel | Running...

2023-01-19T17:07:56.949 app[3d8dd13a49d789] fra [info] sentinel | 2023-01-19T17:07:56.941Z ERROR cmd/sentinel.go:102 election loop error {"error": "Put \"https://consul-fra.fly-shared.net/v1/session/create?wait=5000ms\": read tcp [2604:1380:4091:3602:0:83cd:7087:1]:51930->[2a09:8280:1::3:c1f]:443: read: connection reset by peer"} >[2a09:8280:1::3:c1f]:443: read: connection reset by peer"}

2023-01-19T17:07:56.949 app[3d8dd13a49d789] fra [info] sentinel | 2023-01-19T17:07:56.941Z ERROR cmd/sentinel.go:102 election loop error {"error": "Put \"https://consul-fra.fly-shared.net/v1/session/create?wait=5000ms\": read tcp [2604:1380:4091:3602:0:83cd:7087:1]:51930->[2a09:8280:1::3:c1f]:443: read: connection reset by peer"}

2023-01-19T17:29:40.250 app[3d8dd13a49d789] fra [info] keeper | 2023-01-19T17:24:19.634Z ERROR cmd/keeper.go:1041 error retrieving cluster data {"error": "Get \"https://consul-fra.fly-shared.net/v1/kv/mo....-db-l8gnr14vge81y7wm/mo....-db/clusterdata?consistent=&wait=5000ms\": dial tcp: lookup consul-fra.fly-shared.net on [fdaa::3]:53: read udp [fdaa:0:7b2e:a7b:23c3:83cd:7087:2]:57062->[fdaa::3]:53: i/o timeout"}

2023-01-19T17:29:40.250 app[3d8dd13a49d789] fra [info] sentinel | 2023-01-19T17:24:26.753Z FATAL cmd/sentinel.go:2030 cannot create sentinel: cannot create store: cannot create kv store: Put "https://consul-fra.fly-shared.net/v1/catalog/register?wait=5000ms": dial tcp: lookup consul-fra.fly-shared.net on [fdaa::3]:53: read udp [fdaa:0:7b2e:a7b:23c3:83cd:7087:2]:57661->[fdaa::3]:53: i/o timeout

2023-01-19T17:29:40.250 app[3d8dd13a49d789] fra [info] sentinel | exit status 1

I tried several things after looking for solution, but nothing seems to work. Could you please have a look and confirm there’s an issue I can’t solve myself? I see nothing on https://status.flyio.net/, but the status seems far away from OK…

1 Like

Hello, we’re aware of issues and actively looking into them.

1 Like