Postgres DB instance down

On my elixir app it’s just spewing out these errors:
[error] Postgrex.Protocol (#PID<0.2953.0>) failed to connect: ** (DBConnection.ConnectionError) tcp recv (idle): closed

On the dashboard, when I click into the db instance it says “Deployment is running”:

"Deploying v1

Deployment is running."

But the console output looks like it’s going in circles. The words “this shouldn’t happen” in the logs don’t look good.

2022-10-29T00:08:25.159 app[8120e026] yyz [info] exporter | INFO[9874] Established new database connection to "fdaa:0:43c2:a7b:88dc:2:41ae:2:5433". source="postgres_exporter.go:970"

2022-10-29T00:08:26.160 app[8120e026] yyz [info] exporter | ERRO[9875] Error opening connection to database (postgresql://flypgadmin:PASSWORD_REMOVED@[fdaa:0:43c2:a7b:88dc:2:41ae:2]:5433/postgres?sslmode=disable): dial tcp [fdaa:0:43c2:a7b:88dc:2:41ae:2]:5433: connect: connection refused source="postgres_exporter.go:1658"

2022-10-29T00:08:26.355 app[8120e026] yyz [info] keeper | 2022-10-29T00:08:26.355Z INFO cmd/keeper.go:1141 current db UID different than cluster data db UID {"db": "c5487d8c", "cdDB": "c425c7c8"}

2022-10-29T00:08:26.355 app[8120e026] yyz [info] keeper | 2022-10-29T00:08:26.355Z ERROR cmd/keeper.go:1452 different local dbUID but init mode is none, this shouldn't happen. Something bad happened to the keeper data. Check that keeper data is on a persistent volume and that the keeper state files weren't removed

2022-10-29T00:08:27.521 app[8120e026] yyz [info] sentinel | 2022-10-29T00:08:27.521Z WARN cmd/sentinel.go:287 received db state for unexpected db uid {"receivedDB": "c5487d8c", "db": "c425c7c8", "keeper": "aa2074f12"}

2022-10-29T00:08:27.521 app[8120e026] yyz [info] sentinel | 2022-10-29T00:08:27.521Z WARN cmd/sentinel.go:276 no keeper info available {"db": "ab48744d", "keeper": "aa4074f02"}

2022-10-29T00:08:27.523 app[8120e026] yyz [info] sentinel | 2022-10-29T00:08:27.523Z INFO cmd/sentinel.go:995 master db is failed {"db": "ab48744d", "keeper": "aa4074f02"}

2022-10-29T00:08:27.523 app[8120e026] yyz [info] sentinel | 2022-10-29T00:08:27.523Z INFO cmd/sentinel.go:1006 trying to find a new master to replace failed master

2022-10-29T00:08:27.524 app[8120e026] yyz [info] sentinel | 2022-10-29T00:08:27.523Z ERROR cmd/sentinel.go:1009 no eligible masters

2022-10-29T00:08:27.657 app[8120e026] yyz [info] keeper | 2022-10-29T00:08:27.657Z ERROR cmd/keeper.go:720 cannot get configured pg parameters {"error": "dial unix /tmp/.s.PGSQL.5433: connect: no such file or directory"}

2022-10-29T00:08:30.158 app[8120e026] yyz [info] keeper | 2022-10-29T00:08:30.157Z ERROR cmd/keeper.go:720 cannot get configured pg parameters {"error": "dial unix /tmp/.s.PGSQL.5433: connect: no such file or directory"}

2022-10-29T00:08:31.376 app[8120e026] yyz [info] keeper | 2022-10-29T00:08:31.376Z INFO cmd/keeper.go:1141 current db UID different than cluster data db UID {"db": "c5487d8c", "cdDB": "c425c7c8"}

2022-10-29T00:08:31.376 app[8120e026] yyz [info] keeper | 2022-10-29T00:08:31.376Z ERROR cmd/keeper.go:1452 different local dbUID but init mode is none, this shouldn't happen. Something bad happened to the keeper data. Check that keeper data is on a persistent volume and that the keeper state files weren't removed

2022-10-29T00:08:32.659 app[8120e026] yyz [info] keeper | 2022-10-29T00:08:32.658Z ERROR cmd/keeper.go:720 cannot get configured pg parameters {"error": "dial unix /tmp/.s.PGSQL.5433: connect: no such file or directory"}

2022-10-29T00:08:32.684 app[8120e026] yyz [info] sentinel | 2022-10-29T00:08:32.684Z WARN cmd/sentinel.go:276 no keeper info available {"db": "ab48744d", "keeper": "aa4074f02"}

2022-10-29T00:08:32.684 app[8120e026] yyz [info] sentinel | 2022-10-29T00:08:32.684Z WARN cmd/sentinel.go:287 received db state for unexpected db uid {"receivedDB": "c5487d8c", "db": "c425c7c8", "keeper": "aa2074f12"}

2022-10-29T00:08:32.686 app[8120e026] yyz [info] sentinel | 2022-10-29T00:08:32.685Z INFO cmd/sentinel.go:995 master db is failed {"db": "ab48744d", "keeper": "aa4074f02"}

2022-10-29T00:08:32.686 app[8120e026] yyz [info] sentinel | 2022-10-29T00:08:32.686Z INFO cmd/sentinel.go:1006 trying to find a new master to replace failed master

2022-10-29T00:08:32.686 app[8120e026] yyz [info] sentinel | 2022-10-29T00:08:32.686Z ERROR cmd/sentinel.go:1009 no eligible masters

2022-10-29T00:08:35.159 app[8120e026] yyz [info] keeper | 2022-10-29T00:08:35.159Z ERROR cmd/keeper.go:720 cannot get configured pg parameters {"error": "dial unix /tmp/.s.PGSQL.5433: connect: no such file or directory"}

2022-10-29T00:08:36.396 app[8120e026] yyz [info] keeper | 2022-10-29T00:08:36.396Z INFO cmd/keeper.go:1141 current db UID different than cluster data db UID {"db": "c5487d8c", "cdDB": "c425c7c8"}

2022-10-29T00:08:36.397 app[8120e026] yyz [info] keeper | 2022-10-29T00:08:36.396Z ERROR cmd/keeper.go:1452 different local dbUID but init mode is none, this shouldn't happen. Something bad happened to the keeper data. Check that keeper data is on a persistent volume and that the keeper state files weren't removed

2022-10-29T00:08:37.660 app[8120e026] yyz [info] keeper | 2022-10-29T00:08:37.660Z ERROR cmd/keeper.go:720 cannot get configured pg parameters {"error": "dial unix /tmp/.s.PGSQL.5433: connect: no such file or directory"}

2022-10-29T00:08:37.844 app[8120e026] yyz [info] sentinel | 2022-10-29T00:08:37.844Z WARN cmd/sentinel.go:276 no keeper info available {"db": "ab48744d", "keeper": "aa4074f02"}

2022-10-29T00:08:37.845 app[8120e026] yyz [info] sentinel | 2022-10-29T00:08:37.844Z WARN cmd/sentinel.go:287 received db state for unexpected db uid {"receivedDB": "c5487d8c", "db": "c425c7c8", "keeper": "aa2074f12"}

2022-10-29T00:08:37.848 app[8120e026] yyz [info] sentinel | 2022-10-29T00:08:37.848Z INFO cmd/sentinel.go:995 master db is failed {"db": "ab48744d", "keeper": "aa4074f02"}

2022-10-29T00:08:37.849 app[8120e026] yyz [info] sentinel | 2022-10-29T00:08:37.848Z INFO cmd/sentinel.go:1006 trying to find a new master to replace failed master

2022-10-29T00:08:37.849 app[8120e026] yyz [info] sentinel | 2022-10-29T00:08:37.849Z ERROR cmd/sentinel.go:1009 no eligible masters

2022-10-29T00:08:40.159 app[8120e026] yyz [info] exporter | INFO[9889] Established new database connection to "fdaa:0:43c2:a7b:88dc:2:41ae:2:5433". source="postgres_exporter.go:970"

2022-10-29T00:08:40.161 app[8120e026] yyz [info] keeper | 2022-10-29T00:08:40.161Z ERROR cmd/keeper.go:720 cannot get configured pg parameters {"error": "dial unix /tmp/.s.PGSQL.5433: connect: no such file or directory"}

2022-10-29T00:08:41.160 app[8120e026] yyz [info] exporter | ERRO[9890] Error opening connection to database (postgresql://flypgadmin:PASSWORD_REMOVED@[fdaa:0:43c2:a7b:88dc:2:41ae:2]:5433/postgres?sslmode=disable): dial tcp [fdaa:0:43c2:a7b:88dc:2:41ae:2]:5433: connect: connection refused source="postgres_exporter.go:1658"

2022-10-29T00:08:41.428 app[8120e026] yyz [info] keeper | 2022-10-29T00:08:41.428Z INFO cmd/keeper.go:1141 current db UID different than cluster data db UID {"db": "c5487d8c", "cdDB": "c425c7c8"}

2022-10-29T00:08:41.428 app[8120e026] yyz [info] keeper | 2022-10-29T00:08:41.428Z ERROR cmd/keeper.go:1452 different local dbUID but init mode is none, this shouldn't happen. Something bad happened to the keeper data. Check that keeper data is on a persistent volume and that the keeper state files weren't removed

2022-10-29T00:08:42.662 app[8120e026] yyz [info] keeper | 2022-10-29T00:08:42.662Z ERROR cmd/keeper.go:720 cannot get configured pg parameters {"error": "dial unix /tmp/.s.PGSQL.5433: connect: no such file or directory"}

2022-10-29T00:08:43.028 app[8120e026] yyz [info] sentinel | 2022-10-29T00:08:43.027Z WARN cmd/sentinel.go:287 received db state for unexpected db uid {"receivedDB": "c5487d8c",

2022-10-29T00:08:43.028 app[8120e026] yyz [info] sentinel | 2022-10-29T00:08:43.027Z WARN cmd/sentinel.go:287 received db state for unexpected db uid {"receivedDB": "c5487d8c", "db": "c425c7c8", "keeper": "aa2074f12"}

2022-10-29T00:08:43.030 app[8120e026] yyz [info] sentinel | 2022-10-29T00:08:43.029Z INFO cmd/sentinel.go:995 master db is failed {"db": "ab48744d", "keeper": "aa4074f02"}

2022-10-29T00:08:43.030 app[8120e026] yyz [info] sentinel | 2022-10-29T00:08:43.030Z INFO cmd/sentinel.go:1006 trying to find a new master to replace failed master

2022-10-29T00:08:43.030 app[8120e026] yyz [info] sentinel | 2022-10-29T00:08:43.030Z ERROR cmd/sentinel.go:1009 no eligible masters

2022-10-29T00:08:45.163 app[8120e026] yyz [info] keeper | 2022-10-29T00:08:45.163Z ERROR cmd/keeper.go:720 cannot get configured pg parameters {"error": "dial unix /tmp/.s.PGSQL.5433: connect: no such file or directory"}

2022-10-29T00:08:46.450 app[8120e026] yyz [info] keeper | 2022-10-29T00:08:46.449Z INFO cmd/keeper.go:1141 current db UID different than cluster data db UID {"db": "c5487d8c", "cdDB": "c425c7c8"}

2022-10-29T00:08:46.450 app[8120e026] yyz [info] keeper | 2022-10-29T00:08:46.449Z ERROR cmd/keeper.go:1452 different local dbUID but init mode is none, this shouldn't happen. Something bad happened to the keeper data. Check that keeper data is on a persistent volume and that the keeper state files weren't removed

2022-10-29T00:08:47.665 app[8120e026] yyz [info] keeper | 2022-10-29T00:08:47.664Z ERROR cmd/keeper.go:720 cannot get configured pg parameters {"error": "dial unix /tmp/.s.PGSQL.5433: connect: no such file or directory"}

2022-10-29T00:08:48.189 app[8120e026] yyz [info] sentinel | 2022-10-29T00:08:48.189Z WARN cmd/sentinel.go:276 no keeper info available {"db": "ab48744d", "keeper": "aa4074f02"}

2022-10-29T00:08:48.189 app[8120e026] yyz [info] sentinel | 2022-10-29T00:08:48.189Z WARN cmd/sentinel.go:287 received db state for unexpected db uid {"receivedDB": "c5487d8c", "db": "c425c7c8", "keeper": "aa2074f12"}

2022-10-29T00:08:48.193 app[8120e026] yyz [info] sentinel | 2022-10-29T00:08:48.193Z INFO cmd/sentinel.go:995 master db is failed {"db": "ab48744d", "keeper": "aa4074f02"}

2022-10-29T00:08:48.193 app[8120e026] yyz [info] sentinel | 2022-10-29T00:08:48.193Z INFO cmd/sentinel.go:1006 trying to find a new master to replace failed master

2022-10-29T00:08:48.194 app[8120e026] yyz [info] sentinel | 2022-10-29T00:08:48.193Z ERROR cmd/sentinel.go:1009 no eligible masters

2022-10-29T00:08:50.165 app[8120e026] yyz [info] keeper | 2022-10-29T00:08:50.165Z ERROR cmd/keeper.go:720 cannot get configured pg parameters {"error": "dial unix /tmp/.s.PGSQL.5433: connect: no such file or directory"}

2022-10-29T00:08:51.469 app[8120e026] yyz [info] keeper | 2022-10-29T00:08:51.469Z INFO cmd/keeper.go:1141 current db UID different than cluster data db UID {"db": "c5487d8c", "cdDB": "c425c7c8"}

2022-10-29T00:08:51.470 app[8120e026] yyz [info] keeper | 2022-10-29T00:08:51.469Z ERROR cmd/keeper.go:1452 different local dbUID but init mode is none, this shouldn't happen. Something bad happened to the keeper data. Check that keeper data is on a persistent volume and that the keeper state files weren't removed

2022-10-29T00:08:52.666 app[8120e026] yyz [info] keeper | 2022-10-29T00:08:52.666Z ERROR cmd/keeper.go:720 cannot get configured pg parameters {"error": "dial unix /tmp/.s.PGSQL.5433: connect: no such file or directory"}

2022-10-29T00:08:53.383 app[8120e026] yyz [info] sentinel | 2022-10-29T00:08:53.383Z WARN cmd/sentinel.go:276 no keeper info available {"db": "ab48744d", "keeper": "aa4074f02"}

2022-10-29T00:08:53.383 app[8120e026] yyz [info] sentinel | 2022-10-29T00:08:53.383Z WARN cmd/sentinel.go:287 received db state for unexpected db uid {"receivedDB": "c5487d8c", "db": "c425c7c8", "keeper": "aa2074f12"}

2022-10-29T00:08:53.385 app[8120e026] yyz [info] sentinel | 2022-10-29T00:08:53.385Z INFO cmd/sentinel.go:995 master db is failed {"db": "ab48744d", "keeper": "aa4074f02"}

2022-10-29T00:08:53.385 app[8120e026] yyz [info] sentinel | 2022-10-29T00:08:53.385Z INFO cmd/sentinel.go:1006 trying to find a new master to replace failed master

2022-10-29T00:08:53.385 app[8120e026] yyz [info] sentinel | 2022-10-29T00:08:53.385Z ERROR cmd/sentinel.go:1009 no eligible masters

2022-10-29T00:08:55.159 app[8120e026] yyz [info] exporter | INFO[9904] Established new database connection to "fdaa:0:43c2:a7b:88dc:2:41ae:2:5433". source="postgres_exporter.go:970"

2022-10-29T00:08:55.166 app[8120e026] yyz [info] keeper | 2022-10-29T00:08:55.166Z ERROR cmd/keeper.go:720 cannot get configured pg parameters {"error": "dial unix /tmp/.s.PGSQL.5433: connect: no such file or directory"}

2022-10-29T00:08:56.160 app[8120e026] yyz [info] exporter | ERRO[9905] Error opening connection to database (postgresql://flypgadmin:PASSWORD_REMOVED@[fdaa:0:43c2:a7b:88dc:2:41ae:2]:5433/postgres?sslmode=disable): dial tcp [fdaa:0:43c2:a7b:88dc:2:41ae:2]:5433: connect: connection refused source="postgres_exporter.go:1658"

2022-10-29T00:08:56.494 app[8120e026] yyz [info] keeper | 2022-10-29T00:08:56.493Z INFO cmd/keeper.go:1141 current db UID different than cluster data db UID {"db": "c5487d8c", "cdDB": "c425c7c8"}

2022-10-29T00:08:56.494 app[8120e026] yyz [info] keeper | 2022-10-29T00:08:56.494Z ERROR cmd/keeper.go:1452 different local dbUID but init mode is none, this shouldn't happen. Something bad happened to the keeper data. Check that keeper data is on a persistent volume and that the keeper state files weren't removed

2022-10-29T00:08:57.667 app[8120e026] yyz [info] keeper | 2022-10-29T00:08:57.667Z ERROR cmd/keeper.go:720 cannot get configured pg parameters {"error": "dial unix /tmp/.s.PGSQL.5433: connect: no such file or directory"}

2022-10-29T00:08:58.546 app[8120e026] yyz [info] sentinel | 2022-10-29T00:08:58.545Z WARN cmd/sentinel.go:276 no keeper info available {"db": "ab48744d", "keeper": "aa4074f02"}

2022-10-29T00:08:58.546 app[8120e026] yyz [info] sentinel | 2022-10-29T00:08:58.546Z WARN cmd/sentinel.go:287 received db state for unexpected db uid {"receivedDB": "c5487d8c", "db": "c425c7c8", "keeper": "aa2074f12"}

2022-10-29T00:08:58.549 app[8120e026] yyz [info] sentinel | 2022-10-29T00:08:58.549Z INFO cmd/sentinel.go:1006 trying to find a new master to replace failed master 74f02"}

2022-10-29T00:08:58.549 app[8120e026] yyz [info] sentinel | 2022-10-29T00:08:58.549Z INFO cmd/sentinel.go:1006 trying to find a new master to replace failed master

2022-10-29T00:08:58.549 app[8120e026] yyz [info] sentinel | 2022-10-29T00:08:58.549Z ERROR cmd/sentinel.go:1009 no eligible masters

2022-10-29T00:09:00.168 app[8120e026] yyz [info] keeper | 2022-10-29T00:09:00.168Z ERROR cmd/keeper.go:720 cannot get configured pg parameters {"error": "dial unix /tmp/.s.PGSQL.5433: connect: no such file or directory"}

2022-10-29T00:09:01.531 app[8120e026] yyz [info] keeper | 2022-10-29T00:09:01.530Z INFO cmd/keeper.go:1141 current db UID different than cluster data db UID {"db": "c5487d8c", "cdDB": "c425c7c8"}

2022-10-29T00:09:01.531 app[8120e026] yyz [info] keeper | 2022-10-29T00:09:01.530Z ERROR cmd/keeper.go:1452 different local dbUID but init mode is none, this shouldn't happen. Something bad happened to the keeper data. Check that keeper data is on a persistent volume and that the keeper state files weren't removed

2022-10-29T00:09:02.670 app[8120e026] yyz [info] keeper | 2022-10-29T00:09:02.670Z ERROR cmd/keeper.go:720 cannot get configured pg parameters {"error": "dial unix /tmp/.s.PGSQL.5433: connect: no such file or directory"}

2022-10-29T00:09:03.715 app[8120e026] yyz [info] sentinel | 2022-10-29T00:09:03.713Z WARN cmd/sentinel.go:287 received db state for unexpected db uid {"receivedDB": "c5487d8c", "db": "c425c7c8", "keeper": "aa2074f12"}

2022-10-29T00:09:03.715 app[8120e026] yyz [info] sentinel | 2022-10-29T00:09:03.713Z WARN cmd/sentinel.go:276 no keeper info available {"db": "ab48744d", "keeper": "aa4074f02"}

2022-10-29T00:09:03.717 app[8120e026] yyz [info] sentinel | 2022-10-29T00:09:03.716Z INFO cmd/sentinel.go:995 master db is failed {"db": "ab48744d", "keeper": "aa4074f02"}

2022-10-29T00:09:03.717 app[8120e026] yyz [info] sentinel | 2022-10-29T00:09:03.716Z INFO cmd/sentinel.go:1006 trying to find a new master to replace failed master

2022-10-29T00:09:03.717 app[8120e026] yyz [info] sentinel | 2022-10-29T00:09:03.716Z ERROR cmd/sentinel.go:1009 no eligible masters

2022-10-29T00:09:05.178 app[8120e026] yyz [info] keeper | 2022-10-29T00:09:05.174Z ERROR cmd/keeper.go:720 cannot get configured pg parameters {"error": "dial unix /tmp/.s.PGSQL.5433: connect: no such file or directory"}

2022-10-29T00:09:06.555 app[8120e026] yyz [info] keeper | 2022-10-29T00:09:06.554Z INFO cmd/keeper.go:1141 current db UID different than cluster data db UID {"db": "c5487d8c", "cdDB": "c425c7c8"}

2022-10-29T00:09:06.555 app[8120e026] yyz [info] keeper | 2022-10-29T00:09:06.554Z ERROR cmd/keeper.go:1452 different local dbUID but init mode is none, this shouldn't happen. Something bad happened to the keeper data. Check that keeper data is on a persistent volume and that the keeper state files weren't removed

2022-10-29T00:09:07.676 app[8120e026] yyz [info] keeper | 2022-10-29T00:09:07.675Z ERROR cmd/keeper.go:720 cannot get configured pg parameters {"error": "dial unix /tmp/.s.PGSQL.5433: connect: no such file or directory"}

2022-10-29T00:09:08.953 app[8120e026] yyz [info] sentinel | 2022-10-29T00:09:08.953Z WARN cmd/sentinel.go:276 no keeper info available {"db": "ab48744d", "keeper": "aa4074f02"}

2022-10-29T00:09:08.954 app[8120e026] yyz [info] sentinel | 2022-10-29T00:09:08.953Z WARN cmd/sentinel.go:287 received db state for unexpected db uid {"receivedDB": "c5487d8c", "db": "c425c7c8", "keeper": "aa2074f12"}

2022-10-29T00:09:08.957 app[8120e026] yyz [info] sentinel | 2022-10-29T00:09:08.957Z INFO cmd/sentinel.go:995 master db is failed {"db": "ab48744d", "keeper": "aa4074f02"}

2022-10-29T00:09:08.957 app[8120e026] yyz [info] sentinel | 2022-10-29T00:09:08.957Z INFO cmd/sentinel.go:1006 trying to find a new master to replace failed master

2022-10-29T00:09:08.958 app[8120e026] yyz [info] sentinel | 2022-10-29T00:09:08.957Z ERROR cmd/sentinel.go:1009 no eligible masters

2022-10-29T00:09:10.160 app[8120e026] yyz [info] exporter | INFO[9919] Established new database connection to "fdaa:0:43c2:a7b:88dc:2:41ae:2:5433". source="postgres_exporter.go:970"

2022-10-29T00:09:10.176 app[8120e026] yyz [info] keeper | 2022-10-29T00:09:10.176Z ERROR cmd/keeper.go:720 cannot get configured pg parameters {"error": "dial unix /tmp/.s.PGSQL.5433: connect: no such file or directory"}

This was caused by our scheduler outage last night. We’re working on fixing your db up now.

1 Like

Sorry for the slow reply, you should be good as of about 90 minutes ago.

2 Likes

same issue

ERROR cmd/keeper.go:719 cannot get configured pg parameters {"error": "pq: the database system is in recovery mode"}