Production dabase unreachable all of a sudden

bkniffler · July 19, 2022, 12:12pm

Our database is not reachable any more. The logs don’t show anything unusual. Is there anything happening??

2022-07-19T12:09:47.820 app[352fd723] fra [info] keeper | 2022-07-19T12:09:47.820Z INFO cmd/keeper.go:1542 already master
2022-07-19T12:09:52.669 app[d44f35e3] fra [info] keeper | 2022-07-19T12:09:52.669Z INFO cmd/keeper.go:1556 our db requested role is standby {"followedDB": "7b0208a7"}
2022-07-19T12:09:52.669 app[d44f35e3] fra [info] keeper | 2022-07-19T12:09:52.669Z INFO cmd/keeper.go:1575 already standby
2022-07-19T12:09:52.695 app[d44f35e3] fra [info] keeper | 2022-07-19T12:09:52.694Z INFO cmd/keeper.go:1675 postgres parameters not changed
2022-07-19T12:09:52.695 app[d44f35e3] fra [info] keeper | 2022-07-19T12:09:52.695Z INFO cmd/keeper.go:1702 postgres hba entries not changed
2022-07-19T12:09:57.759 app[d44f35e3] fra [info] keeper | 2022-07-19T12:09:57.755Z INFO cmd/keeper.go:1556 our db requested role is standby {"followedDB": "7b0208a7"}
2022-07-19T12:09:57.759 app[d44f35e3] fra [info] keeper | 2022-07-19T12:09:57.756Z INFO cmd/keeper.go:1575 already standby
2022-07-19T12:09:57.789 app[d44f35e3] fra [info] keeper | 2022-07-19T12:09:57.788Z INFO cmd/keeper.go:1675 postgres parameters not changed
2022-07-19T12:09:57.789 app[d44f35e3] fra [info] keeper | 2022-07-19T12:09:57.789Z INFO cmd/keeper.go:1702 postgres hba entries not changed
2022-07-19T12:09:58.781 app[352fd723] fra [info] keeper | 2022-07-19T12:09:58.780Z INFO cmd/keeper.go:1504 our db requested role is master
2022-07-19T12:09:58.782 app[352fd723] fra [info] keeper | 2022-07-19T12:09:58.781Z INFO cmd/keeper.go:1542 already master
2022-07-19T12:09:58.806 app[352fd723] fra [info] keeper | 2022-07-19T12:09:58.805Z INFO cmd/keeper.go:1702 postgres hba entries not changed
2022-07-19T12:10:03.580 app[d44f35e3] fra [info] keeper | 2022-07-19T12:10:03.580Z INFO cmd/keeper.go:1556 our db requested role is standby {"followedDB": "7b0208a7"}
2022-07-19T12:10:03.580 app[d44f35e3] fra [info] keeper | 2022-07-19T12:10:03.580Z INFO cmd/keeper.go:1575 already standby
2022-07-19T12:10:03.666 app[d44f35e3] fra [info] keeper | 2022-07-19T12:10:03.666Z INFO cmd/keeper.go:1675 postgres parameters not changed
2022-07-19T12:10:03.667 app[d44f35e3] fra [info] keeper | 2022-07-19T12:10:03.666Z INFO cmd/keeper.go:1702 postgres hba entries not changed
2022-07-19T12:10:08.896 app[d44f35e3] fra [info] keeper | 2022-07-19T12:10:08.894Z INFO cmd/keeper.go:1556 our db requested role is standby {"followedDB": "7b0208a7"}
2022-07-19T12:10:08.896 app[d44f35e3] fra [info] keeper | 2022-07-19T12:10:08.894Z INFO cmd/keeper.go:1575 already standby
2022-07-19T12:10:08.920 app[d44f35e3] fra [info] keeper | 2022-07-19T12:10:08.920Z INFO cmd/keeper.go:1675 postgres parameters not changed
2022-07-19T12:10:08.920 app[d44f35e3] fra [info] keeper | 2022-07-19T12:10:08.920Z INFO cmd/keeper.go:1702 postgres hba entries not changed
2022-07-19T12:10:09.906 app[352fd723] fra [info] keeper | 2022-07-19T12:10:09.905Z INFO cmd/keeper.go:1504 our db requested role is master
2022-07-19T12:10:09.954 app[352fd723] fra [info] keeper | 2022-07-19T12:10:09.952Z INFO cmd/keeper.go:1675 postgres parameters not changed
2022-07-19T12:10:14.069 app[d44f35e3] fra [info] keeper | 2022-07-19T12:10:14.065Z INFO cmd/keeper.go:1556 our db requested role is standby {"followedDB": "7b0208a7"}
2022-07-19T12:10:14.069 app[d44f35e3] fra [info] keeper | 2022-07-19T12:10:14.065Z INFO cmd/keeper.go:1575 already standby
2022-07-19T12:10:14.160 app[d44f35e3] fra [info] keeper | 2022-07-19T12:10:14.155Z INFO cmd/keeper.go:1675 postgres parameters not changed
2022-07-19T12:10:14.160 app[d44f35e3] fra [info] keeper | 2022-07-19T12:10:14.157Z INFO cmd/keeper.go:1702 postgres hba entries not changed
2022-07-19T12:10:15.176 app[352fd723] fra [info] keeper | 2022-07-19T12:10:15.175Z INFO cmd/keeper.go:1504 our db requested role is master
2022-07-19T12:10:15.215 app[352fd723] fra [info] keeper | 2022-07-19T12:10:15.215Z INFO cmd/keeper.go:1702 postgres hba entries not changed
2022-07-19T12:10:19.939 app[d44f35e3] fra [info] keeper | 2022-07-19T12:10:19.938Z INFO cmd/keeper.go:1556 our db requested role is standby {"followedDB": "7b0208a7"}
2022-07-19T12:10:19.939 app[d44f35e3] fra [info] keeper | 2022-07-19T12:10:19.939Z INFO cmd/keeper.go:1575 already standby
2022-07-19T12:10:20.011 app[d44f35e3] fra [info] keeper | 2022-07-19T12:10:19.987Z INFO cmd/keeper.go:1675 postgres parameters not changed
2022-07-19T12:10:20.011 app[d44f35e3] fra [info] keeper | 2022-07-19T12:10:19.987Z INFO cmd/keeper.go:1702 postgres hba entries not changed
2022-07-19T12:10:20.935 app[352fd723] fra [info] keeper | 2022-07-19T12:10:20.934Z INFO cmd/keeper.go:1504 our db requested role is master
2022-07-19T12:10:20.935 app[352fd723] fra [info] keeper | 2022-07-19T12:10:20.935Z INFO cmd/keeper.go:1542 already master
2022-07-19T12:10:20.978 app[352fd723] fra [info] keeper | 2022-07-19T12:10:20.977Z INFO cmd/keeper.go:1675 postgres parameters not changed
2022-07-19T12:10:20.978 app[352fd723] fra [info] keeper | 2022-07-19T12:10:20.977Z INFO cmd/keeper.go:1702 postgres hba entries not changed
2022-07-19T12:10:25.118 app[d44f35e3] fra [info] keeper | 2022-07-19T12:10:25.117Z INFO cmd/keeper.go:1556 our db requested role is standby {"followedDB": "7b0208a7"}
2022-07-19T12:10:25.118 app[d44f35e3] fra [info] keeper | 2022-07-19T12:10:25.118Z INFO cmd/keeper.go:1575 already standby
2022-07-19T12:10:25.146 app[d44f35e3] fra [info] keeper | 2022-07-19T12:10:25.145Z INFO cmd/keeper.go:1675 postgres parameters not changed
2022-07-19T12:10:25.146 app[d44f35e3] fra [info] keeper | 2022-07-19T12:10:25.145Z INFO cmd/keeper.go:1702 postgres hba entries not changed
2022-07-19T12:10:26.243 app[352fd723] fra [info] keeper | 2022-07-19T12:10:26.242Z INFO cmd/keeper.go:1504 our db requested role is master
2022-07-19T12:10:26.248 app[352fd723] fra [info] keeper | 2022-07-19T12:10:26.247Z INFO cmd/keeper.go:1542 already master
2022-07-19T12:10:30.721 app[d44f35e3] fra [info] keeper | 2022-07-19T12:10:30.720Z INFO cmd/keeper.go:1556 our db requested role is standby {"followedDB": "7b0208a7"}
2022-07-19T12:10:30.721 app[d44f35e3] fra [info] keeper | 2022-07-19T12:10:30.720Z INFO cmd/keeper.go:1575 already standby
2022-07-19T12:10:30.748 app[d44f35e3] fra [info] keeper | 2022-07-19T12:10:30.748Z INFO cmd/keeper.go:1675 postgres parameters not changed
2022-07-19T12:10:30.749 app[d44f35e3] fra [info] keeper | 2022-07-19T12:10:30.748Z INFO cmd/keeper.go:1702 postgres hba entries not changed
2022-07-19T12:10:31.513 app[352fd723] fra [info] keeper | 2022-07-19T12:10:31.512Z INFO cmd/keeper.go:1504 our db requested role is master
2022-07-19T12:10:31.513 app[352fd723] fra [info] keeper | 2022-07-19T12:10:31.513Z INFO cmd/keeper.go:1542 already master
2022-07-19T12:10:31.559 app[352fd723] fra [info] keeper | 2022-07-19T12:10:31.559Z INFO cmd/keeper.go:1675 postgres parameters not changed
2022-07-19T12:10:31.559 app[352fd723] fra [info] keeper | 2022-07-19T12:10:31.559Z INFO cmd/keeper.go:1702 postgres hba entries not changed
2022-07-19T12:10:36.159 app[d44f35e3] fra [info] keeper | 2022-07-19T12:10:36.158Z INFO cmd/keeper.go:1556 our db requested role is standby {"followedDB": "7b0208a7"}
2022-07-19T12:10:36.159 app[d44f35e3] fra [info] keeper | 2022-07-19T12:10:36.158Z INFO cmd/keeper.go:1575 already standby
2022-07-19T12:10:36.183 app[d44f35e3] fra [info] keeper | 2022-07-19T12:10:36.183Z INFO cmd/keeper.go:1675 postgres parameters not changed
2022-07-19T12:10:36.183 app[d44f35e3] fra [info] keeper | 2022-07-19T12:10:36.183Z INFO cmd/keeper.go:1702 postgres hba entries not changed
2022-07-19T12:10:37.129 app[352fd723] fra [info] keeper | 2022-07-19T12:10:37.125Z INFO cmd/keeper.go:1504 our db requested role is master
2022-07-19T12:10:37.129 app[352fd723] fra [info] keeper | 2022-07-19T12:10:37.127Z INFO cmd/keeper.go:1542 already master
2022-07-19T12:10:37.221 app[352fd723] fra [info] keeper | 2022-07-19T12:10:37.220Z INFO cmd/keeper.go:1675 postgres parameters not changed
2022-07-19T12:10:41.445 app[d44f35e3] fra [info] keeper | 2022-07-19T12:10:41.444Z INFO cmd/keeper.go:1556 our db requested role is standby {"followedDB": "7b0208a7"}
2022-07-19T12:10:41.445 app[d44f35e3] fra [info] keeper | 2022-07-19T12:10:41.444Z INFO cmd/keeper.go:1575 already standby
2022-07-19T12:10:41.557 app[d44f35e3] fra [info] keeper | 2022-07-19T12:10:41.552Z INFO cmd/keeper.go:1675 postgres parameters not changed
2022-07-19T12:10:41.557 app[d44f35e3] fra [info] keeper | 2022-07-19T12:10:41.552Z INFO cmd/keeper.go:1702 postgres hba entries not changed
2022-07-19T12:10:47.424 app[d44f35e3] fra [info] keeper | 2022-07-19T12:10:47.423Z INFO cmd/keeper.go:1556 our db requested role is standby {"followedDB": "7b0208a7"}
2022-07-19T12:10:47.424 app[d44f35e3] fra [info] keeper | 2022-07-19T12:10:47.423Z INFO cmd/keeper.go:1575 already standby
2022-07-19T12:10:47.506 app[d44f35e3] fra [info] keeper | 2022-07-19T12:10:47.501Z INFO cmd/keeper.go:1675 postgres parameters not changed
2022-07-19T12:10:47.506 app[d44f35e3] fra [info] keeper | 2022-07-19T12:10:47.501Z INFO cmd/keeper.go:1702 postgres hba entries not changed
2022-07-19T12:10:52.751 app[d44f35e3] fra [info] keeper | 2022-07-19T12:10:52.751Z INFO cmd/keeper.go:1556 our db requested role is standby {"followedDB": "7b0208a7"}
2022-07-19T12:10:52.751 app[d44f35e3] fra [info] keeper | 2022-07-19T12:10:52.751Z INFO cmd/keeper.go:1575 already standby
2022-07-19T12:10:52.801 app[d44f35e3] fra [info] keeper | 2022-07-19T12:10:52.801Z INFO cmd/keeper.go:1675 postgres parameters not changed
2022-07-19T12:10:52.801 app[d44f35e3] fra [info] keeper | 2022-07-19T12:10:52.801Z INFO cmd/keeper.go:1702 postgres hba entries not changed
2022-07-19T12:10:57.895 app[d44f35e3] fra [info] keeper | 2022-07-19T12:10:57.888Z INFO cmd/keeper.go:1556 our db requested role is standby {"followedDB": "7b0208a7"}
2022-07-19T12:10:57.895 app[d44f35e3] fra [info] keeper | 2022-07-19T12:10:57.888Z INFO cmd/keeper.go:1575 already standby
2022-07-19T12:10:57.926 app[d44f35e3] fra [info] keeper | 2022-07-19T12:10:57.921Z INFO cmd/keeper.go:1675 postgres parameters not changed
2022-07-19T12:10:57.926 app[d44f35e3] fra [info] keeper | 2022-07-19T12:10:57.922Z INFO cmd/keeper.go:1702 postgres hba entries not changed
2022-07-19T12:10:58.248 app[352fd723] fra [info] keeper | 2022-07-19T12:10:58.247Z INFO cmd/keeper.go:1504 our db requested role is master
2022-07-19T12:10:58.279 app[352fd723] fra [info] keeper | 2022-07-19T12:10:58.278Z INFO cmd/keeper.go:1675 postgres parameters not changed
2022-07-19T12:10:58.279 app[352fd723] fra [info] keeper | 2022-07-19T12:10:58.278Z INFO cmd/keeper.go:1702 postgres hba entries not changed
2022-07-19T12:11:04.150 app[d44f35e3] fra [info] keeper | 2022-07-19T12:11:04.150Z INFO cmd/keeper.go:1556 our db requested role is standby {"followedDB": "7b0208a7"}
2022-07-19T12:11:04.150 app[d44f35e3] fra [info] keeper | 2022-07-19T12:11:04.150Z INFO cmd/keeper.go:1575 already standby
2022-07-19T12:11:04.176 app[d44f35e3] fra [info] keeper | 2022-07-19T12:11:04.175Z INFO cmd/keeper.go:1675 postgres parameters not changed
2022-07-19T12:11:04.176 app[d44f35e3] fra [info] keeper | 2022-07-19T12:11:04.176Z INFO cmd/keeper.go:1702 postgres hba entries not changed
2022-07-19T12:11:05.056 app[352fd723] fra [info] keeper | 2022-07-19T12:11:05.055Z INFO cmd/keeper.go:1504 our db requested role is master
2022-07-19T12:11:05.057 app[352fd723] fra [info] keeper | 2022-07-19T12:11:05.056Z INFO cmd/keeper.go:1542 already master
2022-07-19T12:11:05.083 app[352fd723] fra [info] keeper | 2022-07-19T12:11:05.083Z INFO cmd/keeper.go:1675 postgres parameters not changed
2022-07-19T12:11:05.083 app[352fd723] fra [info] keeper | 2022-07-19T12:11:05.083Z INFO cmd/keeper.go:1702 postgres hba entries not changed
2022-07-19T12:11:09.248 app[d44f35e3] fra [info] keeper | 2022-07-19T12:11:09.247Z INFO cmd/keeper.go:1575 already standby role is standby {"followedDB": "7b0208a7"}
2022-07-19T12:11:09.248 app[d44f35e3] fra [info] keeper | 2022-07-19T12:11:09.247Z INFO cmd/keeper.go:1575 already standby
2022-07-19T12:11:09.272 app[d44f35e3] fra [info] keeper | 2022-07-19T12:11:09.272Z INFO cmd/keeper.go:1675 postgres parameters not changed
2022-07-19T12:11:09.272 app[d44f35e3] fra [info] keeper | 2022-07-19T12:11:09.272Z INFO cmd/keeper.go:1702 postgres hba entries not changed
2022-07-19T12:11:10.148 app[352fd723] fra [info] keeper | 2022-07-19T12:11:10.148Z INFO cmd/keeper.go:1504 our db requested role is master
2022-07-19T12:11:10.149 app[352fd723] fra [info] keeper | 2022-07-19T12:11:10.149Z INFO cmd/keeper.go:1542 already master
2022-07-19T12:11:10.177 app[352fd723] fra [info] keeper | 2022-07-19T12:11:10.176Z INFO cmd/keeper.go:1675 postgres parameters not changed
2022-07-19T12:11:10.177 app[352fd723] fra [info] keeper | 2022-07-19T12:11:10.176Z INFO cmd/keeper.go:1702 postgres hba entries not changed
2022-07-19T12:11:14.544 app[d44f35e3] fra [info] keeper | 2022-07-19T12:11:14.543Z INFO cmd/keeper.go:1556 our db requested role is standby {"followedDB": "7b0208a7"}
2022-07-19T12:11:14.544 app[d44f35e3] fra [info] keeper | 2022-07-19T12:11:14.543Z INFO cmd/keeper.go:1575 already standby
2022-07-19T12:11:14.573 app[d44f35e3] fra [info] keeper | 2022-07-19T12:11:14.572Z INFO cmd/keeper.go:1675 postgres parameters not changed
2022-07-19T12:11:14.575 app[d44f35e3] fra [info] keeper | 2022-07-19T12:11:14.575Z INFO cmd/keeper.go:1702 postgres hba entries not changed
2022-07-19T12:11:15.588 app[352fd723] fra [info] keeper | 2022-07-19T12:11:15.586Z INFO cmd/keeper.go:1504 our db requested role is master
2022-07-19T12:11:15.588 app[352fd723] fra [info] keeper | 2022-07-19T12:11:15.587Z INFO cmd/keeper.go:1542 already master
2022-07-19T12:11:15.616 app[352fd723] fra [info] keeper | 2022-07-19T12:11:15.615Z INFO cmd/keeper.go:1675 postgres parameters not changed
2022-07-19T12:11:15.616 app[352fd723] fra [info] keeper | 2022-07-19T12:11:15.615Z INFO cmd/keeper.go:1702 postgres hba entries not changed
2022-07-19T12:11:19.682 app[d44f35e3] fra [info] keeper | 2022-07-19T12:11:19.672Z INFO cmd/keeper.go:1556 our db requested role is standby {"followedDB": "7b0208a7"}
2022-07-19T12:11:19.682 app[d44f35e3] fra [info] keeper | 2022-07-19T12:11:19.672Z INFO cmd/keeper.go:1575 already standby
2022-07-19T12:11:19.712 app[d44f35e3] fra [info] keeper | 2022-07-19T12:11:19.711Z INFO cmd/keeper.go:1675 postgres parameters not changed
2022-07-19T12:11:19.712 app[d44f35e3] fra [info] keeper | 2022-07-19T12:11:19.711Z INFO cmd/keeper.go:1702 postgres hba entries not changed
2022-07-19T12:11:21.241 app[352fd723] fra [info] keeper | 2022-07-19T12:11:21.240Z INFO cmd/keeper.go:1504 our db requested role is master
2022-07-19T12:11:21.242 app[352fd723] fra [info] keeper | 2022-07-19T12:11:21.241Z INFO cmd/keeper.go:1542 already master
2022-07-19T12:11:21.331 app[352fd723] fra [info] keeper | 2022-07-19T12:11:21.322Z INFO cmd/keeper.go:1675 postgres parameters not changed
2022-07-19T12:11:21.331 app[352fd723] fra [info] keeper | 2022-07-19T12:11:21.322Z INFO cmd/keeper.go:1702 postgres hba entries not changed
2022-07-19T12:11:25.259 app[d44f35e3] fra [info] keeper | 2022-07-19T12:11:25.259Z INFO cmd/keeper.go:1556 our db requested role is standby {"followedDB": "7b0208a7"}
2022-07-19T12:11:25.259 app[d44f35e3] fra [info] keeper | 2022-07-19T12:11:25.259Z INFO cmd/keeper.go:1575 already standby
2022-07-19T12:11:25.316 app[d44f35e3] fra [info] keeper | 2022-07-19T12:11:25.315Z INFO cmd/keeper.go:1675 postgres parameters not changed
2022-07-19T12:11:25.316 app[d44f35e3] fra [info] keeper | 2022-07-19T12:11:25.315Z INFO cmd/keeper.go:1702 postgres hba entries not changed

bkniffler · July 19, 2022, 12:13pm

Fly doctor

Testing authentication token... PASSED
Testing flyctl agent... PASSED
Testing local Docker instance... PASSED
Pinging WireGuard gateway (give us a sec)... PASSED

bkniffler · July 19, 2022, 12:17pm

Tried to rescale. The DB was briefly reachable, but then became unreachable again…

Logs


2022-07-19T12:15:20.404 app[ebca82d6] fra [info] "initMode": "existing",
2022-07-19T12:15:20.404 app[ebca82d6] fra [info] "existingConfig": {
2022-07-19T12:15:20.404 app[ebca82d6] fra [info] "keeperUID": "23c312b842"
2022-07-19T12:15:20.405 app[ebca82d6] fra [info] },
2022-07-19T12:15:20.405 app[ebca82d6] fra [info] "pgParameters": {
2022-07-19T12:15:20.405 app[ebca82d6] fra [info] "effective_cache_size": "6912MB",
2022-07-19T12:15:20.405 app[ebca82d6] fra [info] "effective_io_concurrency": "200",
2022-07-19T12:15:20.405 app[ebca82d6] fra [info] "maintenance_work_mem": "460MB",
2022-07-19T12:15:20.405 app[ebca82d6] fra [info] "max_connections": "300",
2022-07-19T12:15:20.405 app[ebca82d6] fra [info] "max_parallel_workers": "8",
2022-07-19T12:15:20.405 app[ebca82d6] fra [info] "max_parallel_workers_per_gather": "2",
2022-07-19T12:15:20.405 app[ebca82d6] fra [info] "max_worker_processes": "8",
2022-07-19T12:15:20.405 app[ebca82d6] fra [info] "random_page_cost": "1.1",
2022-07-19T12:15:20.405 app[ebca82d6] fra [info] "shared_buffers": "2304MB",
2022-07-19T12:15:20.405 app[ebca82d6] fra [info] "wal_compression": "on",
2022-07-19T12:15:20.405 app[ebca82d6] fra [info] "work_mem": "144MB"
2022-07-19T12:15:20.405 app[ebca82d6] fra [info] },
2022-07-19T12:15:20.405 app[ebca82d6] fra [info] "maxStandbysPerSender": 50
2022-07-19T12:15:20.405 app[ebca82d6] fra [info] }
2022-07-19T12:15:20.406 app[ebca82d6] fra [info] generated new config
2022-07-19T12:15:20.406 app[ebca82d6] fra [info] sentinel | Running...
2022-07-19T12:15:20.407 app[ebca82d6] fra [info] keeper | Running...
2022-07-19T12:15:20.407 app[ebca82d6] fra [info] proxy | Running...
2022-07-19T12:15:20.408 app[ebca82d6] fra [info] exporter | Running...
2022-07-19T12:15:20.530 app[ebca82d6] fra [info] exporter | INFO[0000] Starting Server: :9187 source="postgres_exporter.go:1837"
2022-07-19T12:15:20.552 app[ebca82d6] fra [info] proxy | [WARNING] 199/121520 (555) : parsing [/fly/haproxy.cfg:37]: Missing LF on last line, file might have been truncated at position 171. This will become a hard error in HAProxy 2.3.
2022-07-19T12:15:20.565 app[ebca82d6] fra [info] proxy | [NOTICE] 199/121520 (555) : New worker #1 (580) forked
2022-07-19T12:15:20.766 app[ebca82d6] fra [info] sentinel | 2022-07-19T12:15:20.763Z INFO cmd/sentinel.go:2000 sentinel uid {"uid": "d88abcb8"}
2022-07-19T12:15:20.777 app[ebca82d6] fra [info] keeper | 2022-07-19T12:15:20.775Z INFO cmd/keeper.go:2090 exclusive lock on data dir taken
2022-07-19T12:15:21.409 app[ebca82d6] fra [info] checking stolon status
2022-07-19T12:15:21.592 app[ebca82d6] fra [info] proxy | [WARNING] 199/121521 (580) : bk_db/pg1 changed its IP from (none) to fdaa:0:58b5:a7b:23c3:1:2b84:2 by flydns/dns1.
2022-07-19T12:15:21.592 app[ebca82d6] fra [info] proxy | [WARNING] 199/121521 (580) : Server bk_db/pg1 ('fra.finalytic-hasura-db.internal') is UP/READY (resolves again).
2022-07-19T12:15:21.592 app[ebca82d6] fra [info] proxy | [WARNING] 199/121521 (580) : Server bk_db/pg1 administratively READY thanks to valid DNS answer.
2022-07-19T12:15:22.418 app[d44f35e3] fra [info] keeper | 2022-07-19T12:15:22.416Z INFO cmd/keeper.go:1556 our db requested role is standby {"followedDB": "7b0208a7"}
2022-07-19T12:15:22.418 app[d44f35e3] fra [info] keeper | 2022-07-19T12:15:22.416Z INFO cmd/keeper.go:1575 already standby
2022-07-19T12:15:22.450 app[d44f35e3] fra [info] keeper | 2022-07-19T12:15:22.449Z INFO cmd/keeper.go:1675 postgres parameters not changed
2022-07-19T12:15:22.450 app[d44f35e3] fra [info] keeper | 2022-07-19T12:15:22.449Z INFO cmd/keeper.go:1702 postgres hba entries not changed
2022-07-19T12:15:22.564 app[ebca82d6] fra [info] keeper | 2022-07-19T12:15:22.563Z INFO cmd/keeper.go:568 keeper uid {"uid": "23c312b842"}
2022-07-19T12:15:23.168 app[ebca82d6] fra [info] keeper | 2022-07-19T12:15:23.167Z ERROR cmd/keeper.go:719 cannot get configured pg parameters {"error": "dial unix /tmp/.s.PGSQL.5433: connect: no such file or directory"}
2022-07-19T12:15:23.209 app[ebca82d6] fra [info] keeper | 2022-07-19T12:15:23.208Z INFO cmd/keeper.go:1093 our db boot UID is different than the cluster data one, waiting for it to be updated {"bootUUID": "ddc3d5f0-4ebe-4b0b-a781-54797a9fa212", "clusterBootUUID": "8a5aa749-9133-481a-b1d3-cf5074b4ae18"}
2022-07-19T12:15:23.225 app[ebca82d6] fra [info] keeper is healthy, db is healthy, role: master
2022-07-19T12:15:23.267 app[ebca82d6] fra [info] sentinel | 2022-07-19T12:15:23.266Z INFO cmd/sentinel.go:82 Trying to acquire sentinels leadership
2022-07-19T12:15:24.208 app[d44f35e3] fra [info] sentinel | 2022-07-19T12:15:24.204Z ERROR cmd/sentinel.go:1886 cannot get keepers info {"error": "unexpected end of JSON input"}
2022-07-19T12:15:25.669 app[ebca82d6] fra [info] keeper | 2022-07-19T12:15:25.669Z ERROR cmd/keeper.go:719 cannot get configured pg parameters {"error": "dial unix /tmp/.s.PGSQL.5433: connect: no such file or directory"}
2022-07-19T12:15:26.226 app[d44f35e3] fra [info] keeper | 2022-07-19 12:15:26.223 UTC [20964] FATAL: could not connect to the primary server: connection to server at "fdaa:0:58b5:a7b:23c3:1:2b84:2", port 5433 failed: Connection refused
2022-07-19T12:15:26.226 app[d44f35e3] fra [info] keeper | Is the server running on that host and accepting TCP/IP connections?
2022-07-19T12:15:26.231 app[d44f35e3] fra [info] keeper | 2022-07-19 12:15:26.228 UTC [21017] FATAL: could not connect to the primary server: connection to server at "fdaa:0:58b5:a7b:23c3:1:2b84:2", port 5433 failed: Connection refused
2022-07-19T12:15:26.231 app[d44f35e3] fra [info] keeper | Is the server running on that host and accepting TCP/IP connections?
2022-07-19T12:15:27.553 app[ebca82d6] fra [info] proxy | [WARNING] 199/121527 (580) : Server bk_db/pg1 is DOWN, reason: Layer7 timeout, check duration: 5000ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
2022-07-19T12:15:27.553 app[ebca82d6] fra [info] proxy | [NOTICE] 199/121527 (580) : haproxy version is 2.2.9-2+deb11u3
2022-07-19T12:15:27.553 app[ebca82d6] fra [info] proxy | [NOTICE] 199/121527 (580) : path to executable is /usr/sbin/haproxy
2022-07-19T12:15:27.553 app[ebca82d6] fra [info] proxy | [ALERT] 199/121527 (580) : backend 'bk_db' has no server available!
2022-07-19T12:15:27.724 app[d44f35e3] fra [info] keeper | 2022-07-19T12:15:27.724Z INFO cmd/keeper.go:1556 our db requested role is standby {"followedDB": "7b0208a7"}
2022-07-19T12:15:27.724 app[d44f35e3] fra [info] keeper | 2022-07-19T12:15:27.724Z INFO cmd/keeper.go:1575 already standby
2022-07-19T12:15:27.753 app[d44f35e3] fra [info] keeper | 2022-07-19T12:15:27.750Z INFO cmd/keeper.go:1675 postgres parameters not changed
2022-07-19T12:15:27.753 app[d44f35e3] fra [info] keeper | 2022-07-19T12:15:27.751Z INFO cmd/keeper.go:1702 postgres hba entries not changed
2022-07-19T12:15:28.170 app[ebca82d6] fra [info] keeper | 2022-07-19T12:15:28.170Z ERROR cmd/keeper.go:719 cannot get configured pg parameters {"error": "dial unix /tmp/.s.PGSQL.5433: connect: no such file or directory"}
2022-07-19T12:15:28.226 app[ebca82d6] fra [info] error connecting to local postgres context deadline exceeded
2022-07-19T12:15:28.226 app[ebca82d6] fra [info] checking stolon status
2022-07-19T12:15:28.654 app[ebca82d6] fra [info] keeper | 2022-07-19T12:15:28.653Z INFO cmd/keeper.go:1093 our db boot UID is different than the cluster data one, waiting for it to be updated {"bootUUID": "ddc3d5f0-4ebe-4b0b-a781-54797a9fa212", "clusterBootUUID": "8a5aa749-9133-481a-b1d3-cf5074b4ae18"}
2022-07-19T12:15:29.620 app[ebca82d6] fra [info] keeper is healthy, db is healthy, role: master
2022-07-19T12:15:31.234 app[d44f35e3] fra [info] keeper | 2022-07-19 12:15:31.233 UTC [21037] FATAL: could not connect to the primary server: connection to server at "fdaa:0:58b5:a7b:23c3:1:2b84:2", port 5433 failed: Connection refused
2022-07-19T12:15:31.234 app[d44f35e3] fra [info] keeper | Is the server running on that host and accepting TCP/IP connections?
2022-07-19T12:15:31.548 app[ebca82d6] fra [info] exporter | INFO[0011] Established new database connection to "fdaa:0:58b5:a7b:23c3:1:2b84:2:5433". source="postgres_exporter.go:970"
2022-07-19T12:15:32.549 app[ebca82d6] fra [info] exporter | ERRO[0012] Error opening connection to database (postgresql://flypgadmin:PASSWORD_REMOVED@[fdaa:0:58b5:a7b:23c3:1:2b84:2]:5433/postgres?sslmode=disable): dial tcp [fdaa:0:58b5:a7b:23c3:1:2b84:2]:5433: connect: connection refused source="postgres_exporter.go:1658"
2022-07-19T12:15:33.113 app[d44f35e3] fra [info] keeper | 2022-07-19T12:15:33.112Z INFO cmd/keeper.go:1556 our db requested role is standby {"followedDB": "7b0208a7"}
2022-07-19T12:15:33.113 app[d44f35e3] fra [info] keeper | 2022-07-19T12:15:33.112Z INFO cmd/keeper.go:1575 already standby
2022-07-19T12:15:33.158 app[d44f35e3] fra [info] keeper | 2022-07-19T12:15:33.155Z INFO cmd/keeper.go:1675 postgres parameters not changed
2022-07-19T12:15:33.158 app[d44f35e3] fra [info] keeper | 2022-07-19T12:15:33.155Z INFO cmd/keeper.go:1702 postgres hba entries not changed
2022-07-19T12:15:33.172 app[ebca82d6] fra [info] keeper | 2022-07-19T12:15:33.171Z ERROR cmd/keeper.go:719 cannot get configured pg parameters {"error": "dial unix /tmp/.s.PGSQL.5433: connect: no such file or directory"}
2022-07-19T12:15:33.765 app[ebca82d6] fra [info] keeper | 2022-07-19T12:15:33.764Z INFO cmd/keeper.go:1504 our db requested role is master
2022-07-19T12:15:33.774 app[ebca82d6] fra [info] keeper | 2022-07-19T12:15:33.774Z INFO postgresql/postgresql.go:319 starting database
2022-07-19T12:15:33.850 app[ebca82d6] fra [info] keeper | 2022-07-19 12:15:33.849 UTC [616] LOG: starting PostgreSQL 14.3 (Debian 14.3-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
2022-07-19T12:15:33.850 app[ebca82d6] fra [info] keeper | 2022-07-19 12:15:33.849 UTC [616] LOG: listening on IPv6 address "fdaa:0:58b5:a7b:23c3:1:2b84:2", port 5433
2022-07-19T12:15:33.851 app[ebca82d6] fra [info] keeper | 2022-07-19 12:15:33.851 UTC [616] LOG: listening on Unix socket "/tmp/.s.PGSQL.5433"
2022-07-19T12:15:33.861 app[ebca82d6] fra [info] keeper | 2022-07-19 12:15:33.859 UTC [617] LOG: database system was shut down at 2022-07-19 12:15:05 UTC
2022-07-19T12:15:33.870 app[ebca82d6] fra [info] keeper | 2022-07-19 12:15:33.869 UTC [616] LOG: database system is ready to accept connections
2022-07-19T12:15:34.002 app[ebca82d6] fra [info] keeper | 2022-07-19T12:15:34.001Z INFO cmd/keeper.go:1542 already master
2022-07-19T12:15:34.034 app[ebca82d6] fra [info] keeper | 2022-07-19T12:15:34.033Z INFO cmd/keeper.go:1675 postgres parameters not changed
2022-07-19T12:15:34.034 app[ebca82d6] fra [info] keeper | 2022-07-19T12:15:34.033Z INFO cmd/keeper.go:1702 postgres hba entries not changed
2022-07-19T12:15:34.621 app[ebca82d6] fra [info] error connecting to local postgres context deadline exceeded
2022-07-19T12:15:34.621 app[ebca82d6] fra [info] checking stolon status
2022-07-19T12:15:34.969 app[ebca82d6] fra [info] keeper is healthy, db is healthy, role: master
2022-07-19T12:15:34.976 app[ebca82d6] fra [info] configuring operator
2022-07-19T12:15:34.988 app[ebca82d6] fra [info] operator password does not match config, changing
2022-07-19T12:15:34.991 app[ebca82d6] fra [info] operator ready!
2022-07-19T12:15:34.992 app[ebca82d6] fra [info] configuring repluser
2022-07-19T12:15:34.992 app[ebca82d6] fra [info] repluser password does not match config, changing
2022-07-19T12:15:34.995 app[ebca82d6] fra [info] replication ready!
2022-07-19T12:15:36.247 app[d44f35e3] fra [info] keeper | 2022-07-19 12:15:36.246 UTC [21055] LOG: started streaming WAL from primary at 42/BB000000 on timeline 1
2022-07-19T12:15:38.573 app[ebca82d6] fra [info] proxy | [WARNING] 199/121538 (580) : Server bk_db/pg1 is UP, reason: Layer7 check passed, code: 200, check duration: 9ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
2022-07-19T12:15:38.896 app[d44f35e3] fra [info] keeper | 2022-07-19T12:15:38.895Z INFO cmd/keeper.go:1556 our db requested role is standby {"followedDB": "7b0208a7"}
2022-07-19T12:15:38.896 app[d44f35e3] fra [info] keeper | 2022-07-19T12:15:38.895Z INFO cmd/keeper.go:1575 already standby
2022-07-19T12:15:38.917 app[d44f35e3] fra [info] keeper | 2022-07-19T12:15:38.916Z INFO cmd/keeper.go:1675 postgres parameters not changed
2022-07-19T12:15:38.917 app[d44f35e3] fra [info] keeper | 2022-07-19T12:15:38.917Z INFO cmd/keeper.go:1702 postgres hba entries not changed
2022-07-19T12:15:39.382 app[ebca82d6] fra [info] keeper | 2022-07-19T12:15:39.381Z INFO cmd/keeper.go:1504 our db requested role is master
2022-07-19T12:15:39.384 app[ebca82d6] fra [info] keeper | 2022-07-19T12:15:39.382Z INFO cmd/keeper.go:1542 already master
2022-07-19T12:15:39.411 app[ebca82d6] fra [info] keeper | 2022-07-19T12:15:39.411Z INFO cmd/keeper.go:1675 postgres parameters not changed
2022-07-19T12:15:39.411 app[ebca82d6] fra [info] keeper | 2022-07-19T12:15:39.411Z INFO cmd/keeper.go:1702 postgres hba entries not changed
2022-07-19T12:15:42.113 app[d44f35e3] fra [info] proxy | [WARNING] 199/121542 (581) : Server bk_db/pg1 is UP, reason: Layer7 check passed, code: 200, check duration: 12ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
2022-07-19T12:15:44.148 app[d44f35e3] fra [info] keeper | 2022-07-19T12:15:44.147Z INFO cmd/keeper.go:1556 our db requested role is standby {"followedDB": "7b0208a7"}
2022-07-19T12:15:44.148 app[d44f35e3] fra [info] keeper | 2022-07-19T12:15:44.147Z INFO cmd/keeper.go:1575 already standby
2022-07-19T12:15:44.178 app[d44f35e3] fra [info] keeper | 2022-07-19T12:15:44.177Z INFO cmd/keeper.go:1675 postgres parameters not changed
2022-07-19T12:15:44.178 app[d44f35e3] fra [info] keeper | 2022-07-19T12:15:44.178Z INFO cmd/keeper.go:1702 postgres hba entries not changed
2022-07-19T12:15:44.560 app[ebca82d6] fra [info] keeper | 2022-07-19T12:15:44.547Z INFO cmd/keeper.go:1504 our db requested role is master
2022-07-19T12:15:44.560 app[ebca82d6] fra [info] keeper | 2022-07-19T12:15:44.548Z INFO cmd/keeper.go:1542 already master
2022-07-19T12:15:44.579 app[ebca82d6] fra [info] keeper | 2022-07-19T12:15:44.576Z INFO cmd/keeper.go:1675 postgres parameters not changed
2022-07-19T12:15:44.579 app[ebca82d6] fra [info] keeper | 2022-07-19T12:15:44.577Z INFO cmd/keeper.go:1702 postgres hba entries not changed
2022-07-19T12:15:46.542 app[ebca82d6] fra [info] exporter | INFO[0026] Established new database connection to "fdaa:0:58b5:a7b:23c3:1:2b84:2:5433". source="postgres_exporter.go:970"
2022-07-19T12:15:46.557 app[ebca82d6] fra [info] exporter | INFO[0026] Semantic Version Changed on "fdaa:0:58b5:a7b:23c3:1:2b84:2:5433": 0.0.0 -> 14.3.0 source="postgres_exporter.go:1539"
2022-07-19T12:15:46.587 app[ebca82d6] fra [info] exporter | INFO[0026] Established new database connection to "fdaa:0:58b5:a7b:23c3:1:2b84:2:5433". source="postgres_exporter.go:970"
2022-07-19T12:15:46.598 app[ebca82d6] fra [info] exporter | INFO[0026] Semantic Version Changed on "fdaa:0:58b5:a7b:23c3:1:2b84:2:5433": 0.0.0 -> 14.3.0 source="postgres_exporter.go:1539"
2022-07-19T12:15:46.598 app[ebca82d6] fra [info] exporter | INFO[0026] Established new database connection to "fdaa:0:58b5:a7b:23c3:1:2b84:2:5433". source="postgres_exporter.go:970"
2022-07-19T12:15:46.624 app[ebca82d6] fra [info] exporter | INFO[0026] Semantic Version Changed on "fdaa:0:58b5:a7b:23c3:1:2b84:2:5433": 0.0.0 -> 14.3.0 source="postgres_exporter.go:1539"
2022-07-19T12:15:46.624 app[ebca82d6] fra [info] exporter | INFO[0026] Established new database connection to "fdaa:0:58b5:a7b:23c3:1:2b84:2:5433". source="postgres_exporter.go:970"
2022-07-19T12:15:46.647 app[ebca82d6] fra [info] exporter | INFO[0026] Semantic Version Changed on "fdaa:0:58b5:a7b:23c3:1:2b84:2:5433": 0.0.0 -> 14.3.0 source="postgres_exporter.go:1539"
2022-07-19T12:15:46.647 app[ebca82d6] fra [info] exporter | INFO[0026] Established new database connection to "fdaa:0:58b5:a7b:23c3:1:2b84:2:5433". source="postgres_exporter.go:970"
2022-07-19T12:15:46.661 app[ebca82d6] fra [info] exporter | INFO[0026] Semantic Version Changed on "fdaa:0:58b5:a7b:23c3:1:2b84:2:5433": 0.0.0 -> 14.3.0 source="postgres_exporter.go:1539"
2022-07-19T12:15:50.036 app[ebca82d6] fra [info] keeper | 2022-07-19T12:15:50.035Z INFO cmd/keeper.go:1504 our db requested role is master
2022-07-19T12:15:50.037 app[ebca82d6] fra [info] keeper | 2022-07-19T12:15:50.036Z INFO cmd/keeper.go:1542 already master
2022-07-19T12:15:50.071 app[ebca82d6] fra [info] keeper | 2022-07-19T12:15:50.068Z INFO cmd/keeper.go:1675 postgres parameters not changed
2022-07-19T12:15:50.071 app[ebca82d6] fra [info] keeper | 2022-07-19T12:15:50.068Z INFO cmd/keeper.go:1702 postgres hba entries not changed
2022-07-19T12:15:50.071 app[d44f35e3] fra [info] keeper | 2022-07-19T12:15:50.071Z INFO cmd/keeper.go:1556 our db requested role is standby {"followedDB": "7b0208a7"}
2022-07-19T12:15:50.071 app[d44f35e3] fra [info] keeper | 2022-07-19T12:15:50.071Z INFO cmd/keeper.go:1575 already standby
2022-07-19T12:15:50.116 app[d44f35e3] fra [info] keeper | 2022-07-19T12:15:50.116Z INFO cmd/keeper.go:1675 postgres parameters not changed
2022-07-19T12:15:50.117 app[d44f35e3] fra [info] keeper | 2022-07-19T12:15:50.117Z INFO cmd/keeper.go:1702 postgres hba entries not changed
2022-07-19T12:15:52.220 app[d44f35e3] fra [info] proxy | [WARNING] 199/121552 (581) : Server bk_db/pg1 is DOWN, reason: Layer7 invalid response, info: "HTTP content check did not match", check duration: 18ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
2022-07-19T12:15:52.220 app[d44f35e3] fra [info] proxy | [ALERT] 199/121552 (581) : backend 'bk_db' has no server available!
2022-07-19T12:15:55.145 app[ebca82d6] fra [info] keeper | 2022-07-19T12:15:55.144Z INFO cmd/keeper.go:1504 our db requested role is master
2022-07-19T12:15:55.145 app[ebca82d6] fra [info] keeper | 2022-07-19T12:15:55.145Z INFO cmd/keeper.go:1542 already master
2022-07-19T12:15:55.191 app[d44f35e3] fra [info] keeper | 2022-07-19T12:15:55.190Z INFO cmd/keeper.go:1556 our db requested role is standby {"followedDB": "7b0208a7"}
2022-07-19T12:15:55.191 app[d44f35e3] fra [info] keeper | 2022-07-19T12:15:55.190Z INFO cmd/keeper.go:1575 already standby
2022-07-19T12:15:55.207 app[ebca82d6] fra [info] keeper | 2022-07-19T12:15:55.200Z INFO cmd/keeper.go:1675 postgres parameters not changed
2022-07-19T12:15:55.207 app[ebca82d6] fra [info] keeper | 2022-07-19T12:15:55.200Z INFO cmd/keeper.go:1702 postgres hba entries not changed
2022-07-19T12:15:55.224 app[d44f35e3] fra [info] keeper | 2022-07-19T12:15:55.224Z INFO cmd/keeper.go:1675 postgres parameters not changed
2022-07-19T12:15:55.225 app[d44f35e3] fra [info] keeper | 2022-07-19T12:15:55.224Z INFO cmd/keeper.go:1702 postgres hba entries not changed
2022-07-19T12:16:00.629 app[ebca82d6] fra [info] keeper | 2022-07-19T12:16:00.628Z INFO cmd/keeper.go:1542 already master
2022-07-19T12:16:00.681 app[ebca82d6] fra [info] keeper | 2022-07-19T12:16:00.680Z INFO cmd/keeper.go:1675 postgres parameters not changed
2022-07-19T12:16:00.681 app[ebca82d6] fra [info] keeper | 2022-07-19T12:16:00.680Z INFO cmd/keeper.go:1702 postgres hba entries not changed
2022-07-19T12:16:00.685 app[d44f35e3] fra [info] keeper | 2022-07-19T12:16:00.684Z INFO cmd/keeper.go:1556 our db requested role is standby {"followedDB": "7b0208a7"}
2022-07-19T12:16:00.716 app[d44f35e3] fra [info] keeper | 2022-07-19T12:16:00.714Z INFO cmd/keeper.go:1675 postgres parameters not changed
2022-07-19T12:16:00.716 app[d44f35e3] fra [info] keeper | 2022-07-19T12:16:00.714Z INFO cmd/keeper.go:1702 postgres hba entries not changed
2022-07-19T12:16:06.274 app[ebca82d6] fra [info] keeper | 2022-07-19T12:16:06.274Z INFO cmd/keeper.go:1504 our db requested role is master
2022-07-19T12:16:06.275 app[ebca82d6] fra [info] keeper | 2022-07-19T12:16:06.275Z INFO cmd/keeper.go:1542 already master
2022-07-19T12:16:06.320 app[ebca82d6] fra [info] keeper | 2022-07-19T12:16:06.320Z INFO cmd/keeper.go:1675 postgres parameters not changed
2022-07-19T12:16:06.321 app[ebca82d6] fra [info] keeper | 2022-07-19T12:16:06.320Z INFO cmd/keeper.go:1702 postgres hba entries not changed
2022-07-19T12:16:07.020 app[d44f35e3] fra [info] keeper | 2022-07-19T12:16:07.020Z INFO cmd/keeper.go:1556 our db requested role is standby {"followedDB": "7b0208a7"}
2022-07-19T12:16:07.020 app[d44f35e3] fra [info] keeper | 2022-07-19T12:16:07.020Z INFO cmd/keeper.go:1575 already standby
2022-07-19T12:16:07.052 app[d44f35e3] fra [info] keeper | 2022-07-19T12:16:07.052Z INFO cmd/keeper.go:1675 postgres parameters not changed
2022-07-19T12:16:07.052 app[d44f35e3] fra [info] keeper | 2022-07-19T12:16:07.052Z INFO cmd/keeper.go:1702 postgres hba entries not changed
2022-07-19T12:16:11.411 app[ebca82d6] fra [info] keeper | 2022-07-19T12:16:11.410Z INFO cmd/keeper.go:1504 our db requested role is master
2022-07-19T12:16:11.412 app[ebca82d6] fra [info] keeper | 2022-07-19T12:16:11.411Z INFO cmd/keeper.go:1542 already master
2022-07-19T12:16:11.434 app[ebca82d6] fra [info] keeper | 2022-07-19T12:16:11.434Z INFO cmd/keeper.go:1675 postgres parameters not changed
2022-07-19T12:16:11.434 app[ebca82d6] fra [info] keeper | 2022-07-19T12:16:11.434Z INFO cmd/keeper.go:1702 postgres hba entries not changed
2022-07-19T12:16:12.116 app[d44f35e3] fra [info] keeper | 2022-07-19T12:16:12.115Z INFO cmd/keeper.go:1556 our db requested role is standby {"followedDB": "7b0208a7"}
2022-07-19T12:16:12.116 app[d44f35e3] fra [info] keeper | 2022-07-19T12:16:12.115Z INFO cmd/keeper.go:1575 already standby
2022-07-19T12:16:12.142 app[d44f35e3] fra [info] keeper | 2022-07-19T12:16:12.142Z INFO cmd/keeper.go:1675 postgres parameters not changed
2022-07-19T12:16:12.143 app[d44f35e3] fra [info] keeper | 2022-07-19T12:16:12.142Z INFO cmd/keeper.go:1702 postgres hba entries not changed
2022-07-19T12:16:17.447 app[d44f35e3] fra [info] keeper | 2022-07-19T12:16:17.446Z INFO cmd/keeper.go:1556 our db requested role is standby {"followedDB": "7b0208a7"}
2022-07-19T12:16:17.447 app[d44f35e3] fra [info] keeper | 2022-07-19T12:16:17.446Z INFO cmd/keeper.go:1575 already standby
2022-07-19T12:16:17.471 app[d44f35e3] fra [info] keeper | 2022-07-19T12:16:17.470Z INFO cmd/keeper.go:1675 postgres parameters not changed
2022-07-19T12:16:17.471 app[d44f35e3] fra [info] keeper | 2022-07-19T12:16:17.471Z INFO cmd/keeper.go:1702 postgres hba entries not changed
2022-07-19T12:16:17.490 app[ebca82d6] fra [info] keeper | 2022-07-19T12:16:17.488Z INFO cmd/keeper.go:1504 our db requested role is master
2022-07-19T12:16:17.490 app[ebca82d6] fra [info] keeper | 2022-07-19T12:16:17.490Z INFO cmd/keeper.go:1542 already master
2022-07-19T12:16:17.559 app[ebca82d6] fra [info] keeper | 2022-07-19T12:16:17.558Z INFO cmd/keeper.go:1675 postgres parameters not changed
2022-07-19T12:16:17.559 app[ebca82d6] fra [info] keeper | 2022-07-19T12:16:17.559Z INFO cmd/keeper.go:1702 postgres hba entries not changed
2022-07-19T12:16:22.533 app[d44f35e3] fra [info] keeper | 2022-07-19T12:16:22.533Z INFO cmd/keeper.go:1556 our db requested role is standby {"followedDB": "7b0208a7"}
2022-07-19T12:16:22.533 app[d44f35e3] fra [info] keeper | 2022-07-19T12:16:22.533Z INFO cmd/keeper.go:1575 already standby
2022-07-19T12:16:22.561 app[d44f35e3] fra [info] keeper | 2022-07-19T12:16:22.560Z INFO cmd/keeper.go:1675 postgres parameters not changed
2022-07-19T12:16:22.561 app[d44f35e3] fra [info] keeper | 2022-07-19T12:16:22.561Z INFO cmd/keeper.go:1702 postgres hba entries not changed
2022-07-19T12:16:22.709 app[ebca82d6] fra [info] keeper | 2022-07-19T12:16:22.708Z INFO cmd/keeper.go:1504 our db requested role is master
2022-07-19T12:16:22.709 app[ebca82d6] fra [info] keeper | 2022-07-19T12:16:22.709Z INFO cmd/keeper.go:1542 already master
2022-07-19T12:16:22.731 app[ebca82d6] fra [info] keeper | 2022-07-19T12:16:22.730Z INFO cmd/keeper.go:1675 postgres parameters not changed
2022-07-19T12:16:22.731 app[ebca82d6] fra [info] keeper | 2022-07-19T12:16:22.731Z INFO cmd/keeper.go:1702 postgres hba entries not changed
2022-07-19T12:16:27.806 app[d44f35e3] fra [info] keeper | 2022-07-19T12:16:27.805Z INFO cmd/keeper.go:1556 our db requested role is standby {"followedDB": "7b0208a7"}
2022-07-19T12:16:27.806 app[d44f35e3] fra [info] keeper | 2022-07-19T12:16:27.805Z INFO cmd/keeper.go:1575 already standby
2022-07-19T12:16:27.844 app[d44f35e3] fra [info] keeper | 2022-07-19T12:16:27.843Z INFO cmd/keeper.go:1675 postgres parameters not changed
2022-07-19T12:16:27.844 app[d44f35e3] fra [info] keeper | 2022-07-19T12:16:27.843Z INFO cmd/keeper.go:1702 postgres hba entries not changed
2022-07-19T12:16:27.847 app[ebca82d6] fra [info] keeper | 2022-07-19T12:16:27.846Z INFO cmd/keeper.go:1504 our db requested role is master
2022-07-19T12:16:27.848 app[ebca82d6] fra [info] keeper | 2022-07-19T12:16:27.847Z INFO cmd/keeper.go:1542 already master
2022-07-19T12:16:27.875 app[ebca82d6] fra [info] keeper | 2022-07-19T12:16:27.875Z INFO cmd/keeper.go:1675 postgres parameters not changed
2022-07-19T12:16:27.875 app[ebca82d6] fra [info] keeper | 2022-07-19T12:16:27.875Z INFO cmd/keeper.go:1702 postgres hba entries not changed
2022-07-19T12:16:33.202 app[d44f35e3] fra [info] keeper | 2022-07-19T12:16:33.201Z INFO cmd/keeper.go:1556 our db requested role is standby {"followedDB": "7b0208a7"}
2022-07-19T12:16:33.202 app[d44f35e3] fra [info] keeper | 2022-07-19T12:16:33.201Z INFO cmd/keeper.go:1575 already standby
2022-07-19T12:16:33.235 app[d44f35e3] fra [info] keeper | 2022-07-19T12:16:33.234Z INFO cmd/keeper.go:1675 postgres parameters not changed
2022-07-19T12:16:33.235 app[d44f35e3] fra [info] keeper | 2022-07-19T12:16:33.234Z INFO cmd/keeper.go:1702 postgres hba entries not changed
2022-07-19T12:16:33.450 app[ebca82d6] fra [info] keeper | 2022-07-19T12:16:33.449Z INFO cmd/keeper.go:1504 our db requested role is master
2022-07-19T12:16:33.451 app[ebca82d6] fra [info] keeper | 2022-07-19T12:16:33.450Z INFO cmd/keeper.go:1542 already master
2022-07-19T12:16:33.470 app[ebca82d6] fra [info] keeper | 2022-07-19T12:16:33.469Z INFO cmd/keeper.go:1675 postgres parameters not changed
2022-07-19T12:16:33.470 app[ebca82d6] fra [info] keeper | 2022-07-19T12:16:33.470Z INFO cmd/keeper.go:1702 postgres hba entries not changed
2022-07-19T12:16:38.411 app[d44f35e3] fra [info] keeper | 2022-07-19T12:16:38.410Z INFO cmd/keeper.go:1575 already standby role is standby {"followedDB": "7b0208a7"}
2022-07-19T12:16:38.411 app[d44f35e3] fra [info] keeper | 2022-07-19T12:16:38.410Z INFO cmd/keeper.go:1575 already standby
2022-07-19T12:16:38.438 app[d44f35e3] fra [info] keeper | 2022-07-19T12:16:38.437Z INFO cmd/keeper.go:1675 postgres parameters not changed
2022-07-19T12:16:38.438 app[d44f35e3] fra [info] keeper | 2022-07-19T12:16:38.438Z INFO cmd/keeper.go:1702 postgres hba entries not changed
2022-07-19T12:16:38.718 app[ebca82d6] fra [info] keeper | 2022-07-19T12:16:38.718Z INFO cmd/keeper.go:1504 our db requested role is master
2022-07-19T12:16:38.722 app[ebca82d6] fra [info] keeper | 2022-07-19T12:16:38.719Z INFO cmd/keeper.go:1542 already master
2022-07-19T12:16:38.762 app[ebca82d6] fra [info] keeper | 2022-07-19T12:16:38.761Z INFO cmd/keeper.go:1675 postgres parameters not changed
2022-07-19T12:16:38.762 app[ebca82d6] fra [info] keeper | 2022-07-19T12:16:38.762Z INFO cmd/keeper.go:1702 postgres hba entries not changed
2022-07-19T12:16:44.173 app[d44f35e3] fra [info] keeper | 2022-07-19T12:16:44.165Z INFO cmd/keeper.go:1556 our db requested role is standby {"followedDB": "7b0208a7"}
2022-07-19T12:16:44.173 app[d44f35e3] fra [info] keeper | 2022-07-19T12:16:44.165Z INFO cmd/keeper.go:1575 already standby
2022-07-19T12:16:44.179 app[ebca82d6] fra [info] keeper | 2022-07-19T12:16:44.178Z INFO cmd/keeper.go:1504 our db requested role is master
2022-07-19T12:16:44.180 app[ebca82d6] fra [info] keeper | 2022-07-19T12:16:44.179Z INFO cmd/keeper.go:1542 already master
2022-07-19T12:16:44.216 app[d44f35e3] fra [info] keeper | 2022-07-19T12:16:44.215Z INFO cmd/keeper.go:1675 postgres parameters not changed
2022-07-19T12:16:44.216 app[d44f35e3] fra [info] keeper | 2022-07-19T12:16:44.216Z INFO cmd/keeper.go:1702 postgres hba entries not changed
2022-07-19T12:16:44.284 app[ebca82d6] fra [info] keeper | 2022-07-19T12:16:44.283Z INFO cmd/keeper.go:1675 postgres parameters not changed
2022-07-19T12:16:44.284 app[ebca82d6] fra [info] keeper | 2022-07-19T12:16:44.283Z INFO cmd/keeper.go:1702 postgres hba entries not changed
2022-07-19T12:16:49.352 app[ebca82d6] fra [info] keeper | 2022-07-19T12:16:49.352Z INFO cmd/keeper.go:1504 our db requested role is master
2022-07-19T12:16:49.353 app[ebca82d6] fra [info] keeper | 2022-07-19T12:16:49.352Z INFO cmd/keeper.go:1542 already master
2022-07-19T12:16:49.376 app[ebca82d6] fra [info] keeper | 2022-07-19T12:16:49.375Z INFO cmd/keeper.go:1675 postgres parameters not changed
2022-07-19T12:16:49.376 app[ebca82d6] fra [info] keeper | 2022-07-19T12:16:49.376Z INFO cmd/keeper.go:1702 postgres hba entries not changed
2022-07-19T12:16:49.759 app[d44f35e3] fra [info] keeper | 2022-07-19T12:16:49.758Z INFO cmd/keeper.go:1556 our db requested role is standby {"followedDB": "7b0208a7"}
2022-07-19T12:16:49.759 app[d44f35e3] fra [info] keeper | 2022-07-19T12:16:49.759Z INFO cmd/keeper.go:1575 already standby
2022-07-19T12:16:49.790 app[d44f35e3] fra [info] keeper | 2022-07-19T12:16:49.788Z INFO cmd/keeper.go:1675 postgres parameters not changed
2022-07-19T12:16:49.790 app[d44f35e3] fra [info] keeper | 2022-07-19T12:16:49.788Z INFO cmd/keeper.go:1702 postgres hba entries not changed
2022-07-19T12:16:54.900 app[ebca82d6] fra [info] keeper | 2022-07-19T12:16:54.899Z INFO cmd/keeper.go:1504 our db requested role is master
2022-07-19T12:16:54.914 app[ebca82d6] fra [info] keeper | 2022-07-19T12:16:54.914Z INFO cmd/keeper.go:1542 already master
2022-07-19T12:16:54.943 app[ebca82d6] fra [info] keeper | 2022-07-19T12:16:54.942Z INFO cmd/keeper.go:1675 postgres parameters not changed
2022-07-19T12:16:54.943 app[ebca82d6] fra [info] keeper | 2022-07-19T12:16:54.943Z INFO cmd/keeper.go:1702 postgres hba entries not changed
2022-07-19T12:16:55.164 app[d44f35e3] fra [info] keeper | 2022-07-19T12:16:55.164Z INFO cmd/keeper.go:1556 our db requested role is standby {"followedDB": "7b0208a7"}
2022-07-19T12:16:55.164 app[d44f35e3] fra [info] keeper | 2022-07-19T12:16:55.164Z INFO cmd/keeper.go:1575 already standby
2022-07-19T12:16:55.186 app[d44f35e3] fra [info] keeper | 2022-07-19T12:16:55.186Z INFO cmd/keeper.go:1675 postgres parameters not changed
2022-07-19T12:16:55.186 app[d44f35e3] fra [info] keeper | 2022-07-19T12:16:55.186Z INFO cmd/keeper.go:1702 postgres hba entries not changed

bkniffler · July 19, 2022, 12:19pm

Really need some help @kurt or anyone from team. Our system has a lot of customers who’re already pinging us…

bkniffler · July 19, 2022, 12:21pm

Rescale seems stuck, 1 instance is updated, the other one still on prev version:

bkniffler · July 19, 2022, 12:32pm

Just saw Fly.io Status - Host kernel upgrades

Wouldn’t this be supposed to NOT disrupt our services if we have multiple instances running…?

eli · July 19, 2022, 12:36pm

You’re correct; I don’t think that we’ve gotten to your region in the maintenance yet, so it might be something else.
Checking on our end, I can see that your allocations appear to agree on the version now (v7)
It also looks like your disks might be getting a bit too full. We just released fly volume extend to help with this.

bkniffler · July 19, 2022, 12:45pm

Thanks, looks like extending the volumes helped! The new command is super helpful.

bkniffler · July 19, 2022, 1:24pm

Not all health checks are working, is this something I need to worry about @eli ? Thanks

bkniffler · July 19, 2022, 1:28pm

Nevermind, they are now all passing, guess it just needed some time to readjust itself

Topic		Replies	Views
Production database randomly going down and not recovering	6	965	July 18, 2022
Failing Postgres DB connection from app here and there Questions / Help	45	2158	April 26, 2023
Lost Postgres DB connection Questions / Help	4	528	December 14, 2021
2 database instances but none showing in fly status	11	344	March 22, 2022
postgres our db requested role is standby Questions / Help postgres	1	256	April 21, 2023

Production dabase unreachable all of a sudden

Related topics