Recently we’ve been receiving a lot of connection on our app and it started refusing processing events
We are suspecting that we reached the limit of connections to the database. Is there a way we can change the limit?
Here are the database logs:
2021-08-12T13:39:15.610207428Z app[901039f9] syd [info] "max_connections": "300",
2021-08-12T13:39:15.610766501Z app[901039f9] syd [info] "max_parallel_workers": "8",
2021-08-12T13:39:15.611450883Z app[901039f9] syd [info] "max_parallel_workers_per_gather": "2",
2021-08-12T13:39:15.612090297Z app[901039f9] syd [info] "max_worker_processes": "8",
2021-08-12T13:39:15.612660854Z app[901039f9] syd [info] "random_page_cost": "1.1",
2021-08-12T13:39:15.613219580Z app[901039f9] syd [info] "shared_buffers": "512MB",
2021-08-12T13:39:15.613656772Z app[901039f9] syd [info] "work_mem": "32MB"
2021-08-12T13:39:15.613813156Z app[901039f9] syd [info] }
2021-08-12T13:39:15.613906932Z app[901039f9] syd [info] }
2021-08-12T13:39:15.614222753Z app[901039f9] syd [info] generated new config
2021-08-12T13:39:18.169167809Z app[901039f9] syd [info] system | Tmux socket name: overmind-fly-b5QYOwWzfOthl6ORaRJhy2
2021-08-12T13:39:18.171078862Z app[901039f9] syd [info] system | Tmux session ID: fly
2021-08-12T13:39:18.172365307Z app[901039f9] syd [info] system | Listening at ./.overmind.sock
2021-08-12T13:39:18.767481683Z app[901039f9] syd [info] update_config | Started with pid 585...
2021-08-12T13:39:18.769295023Z app[901039f9] syd [info] keeper | Started with pid 574...
2021-08-12T13:39:18.771169592Z app[901039f9] syd [info] sentinel | Started with pid 577...
2021-08-12T13:39:18.773114463Z app[901039f9] syd [info] proxy | Started with pid 579...
2021-08-12T13:39:18.775017855Z app[901039f9] syd [info] postgres_exporter | Started with pid 582...
2021-08-12T13:39:27.176531220Z app[901039f9] syd [info] postgres_exporter | INFO[0000] Starting Server: :9187
source="postgres_exporter.go:1837"
2021-08-12T13:39:34.121255974Z app[901039f9] syd [info] sentinel | 2021-08-12T13:39:34.018Z INFO cmd/sentinel.go:2000 sentinel uid {"uid": "c8fbe650"}
2021-08-12T13:39:36.306267435Z app[901039f9] syd [info] sentinel | 2021-08-12T13:39:36.300Z INFO cmd/sentinel.go:82 Trying to acquire sentinels leadership
2021-08-12T13:39:37.479921974Z app[8ff3cc74] lhr [info] sentinel | 2021-08-12T13:39:37.476Z ERROR cmd/sentinel.go:1893 failed to get proxies info {"error": "unexpected end of JSON input"}
2021-08-12T13:39:37.487585190Z app[901039f9] syd [info] update_config | === Active sentinels ===
2021-08-12T13:39:37.489214564Z app[901039f9] syd [info] update_config |
2021-08-12T13:39:37.492730968Z app[901039f9] syd [info] update_config | ID LEADER
2021-08-12T13:39:37.494081297Z app[901039f9] syd [info] update_config | 02b3698a true
2021-08-12T13:39:37.495365015Z app[901039f9] syd [info] update_config | f7801acc false
2021-08-12T13:39:37.496090602Z app[901039f9] syd [info] update_config |
2021-08-12T13:39:37.497160155Z app[901039f9] syd [info] update_config | === Active proxies ===
2021-08-12T13:39:37.497970622Z app[901039f9] syd [info] update_config |
2021-08-12T13:39:37.498864889Z app[901039f9] syd [info] update_config | ID
2021-08-12T13:39:37.499878408Z app[901039f9] syd [info] update_config | 09de0f34
2021-08-12T13:39:37.500936887Z app[901039f9] syd [info] update_config | b2fe1907
2021-08-12T13:39:37.501747265Z app[901039f9] syd [info] update_config |
2021-08-12T13:39:37.502768413Z app[901039f9] syd [info] update_config | === Keepers ===
2021-08-12T13:39:37.503516919Z app[901039f9] syd [info] update_config |
2021-08-12T13:39:37.506816928Z app[901039f9] syd [info] update_config | UID HEALTHY PG LISTENADDRESPG HEALTHY PG WANTEDGENERATION PG CURRENTGENERATION
2021-08-12T13:39:37.509996416Z app[901039f9] syd [info] update_config | fdaa01af2a7ba9801e822 true fdaa:0:1af2:a7b:a98:0:1e82:2:5433 true 3 3
2021-08-12T13:39:37.514007138Z app[901039f9] syd [info] update_config | fdaa01af2a7ba9a01e812 true fdaa:0:1af2:a7b:a9a:0:1e81:2:5433 true 5 5
2021-08-12T13:39:37.517697256Z app[901039f9] syd [info] update_config |
2021-08-12T13:39:37.519623241Z app[901039f9] syd [info] update_config | === Cluster Info ===
2021-08-12T13:39:37.520834278Z app[901039f9] syd [info] update_config |
2021-08-12T13:39:37.522988200Z app[901039f9] syd [info] update_config | Master Keeper: fdaa01af2a7ba9a01e812
2021-08-12T13:39:38.036503204Z app[901039f9] syd [info] update_config |
2021-08-12T13:39:38.037987487Z app[901039f9] syd [info] update_config | ===== Keepers/DB tree =====
2021-08-12T13:39:38.038964073Z app[901039f9] syd [info] update_config |
2021-08-12T13:39:38.040393982Z app[901039f9] syd [info] update_config | fdaa01af2a7ba9a01e812 (master)
2021-08-12T13:39:38.041883008Z app[901039f9] syd [info] update_config | └─fdaa01af2a7ba9801e822
2021-08-12T13:39:38.042904161Z app[901039f9] syd [info] update_config |
2021-08-12T13:39:51.502321201Z app[8ff3cc74] lhr [info] sentinel | 2021-08-12T13:39:51.498Z INFO cmd/sentinel.go:1503 added new standby db {"db": "b9027623", "keeper": "fdaa01af2a7bb92032e52"}
2021-08-12T13:40:00.084456040Z app[8ff3cc74] lhr [info] sentinel | 2021-08-12T13:40:00.080Z WARN cmd/sentinel.go:287 received db state for unexpected db uid {"receivedDB": "", "db": "b9027623", "keeper": "fdaa01af2a7bb92032e52"}
2021-08-12T13:40:00.181170831Z app[901039f9] syd [info] keeper | pg_basebackup: initiating base backup, waiting for checkpoint to complete
2021-08-12T13:40:00.999357683Z app[b487a403] lhr [info] keeper | 2021-08-12 13:40:00.997 UTC [2201] WARNING: skipping special file "./postgresql.auto.conf"
2021-08-12T13:40:01.129524594Z app[901039f9] syd [info] keeper | WARNING: skipping special file "./postgresql.auto.conf"
2021-08-12T13:40:01.130586449Z app[901039f9] syd [info] keeper | pg_basebackup: checkpoint completed
2021-08-12T13:40:01.132127532Z app[901039f9] syd [info] keeper | pg_basebackup: write-ahead log start point: 1/20000028 on timeline 2
2021-08-12T13:40:01.133386009Z app[901039f9] syd [info] keeper | pg_basebackup: starting background WAL receiver
2021-08-12T13:40:02.405020653Z app[901039f9] syd [info] keeper | 0/56420 kB (0%), 0/1 tablespace (/data/postgres/backup_label )
2021-08-12T13:40:02.483382075Z app[901039f9] syd [info] postgres_exporter | INFO[0035] Established new database connection to "fdaa:0:1af2:a7b:b92:0:32e5:2:5433". source="postgres_exporter.go:970"
2021-08-12T13:40:03.035402657Z app[901039f9] syd [info] keeper | 279/56420 kB (0%), 0/1 tablespace (/data/postgres/base/13408/1259 )
2021-08-12T13:40:03.531094464Z app[901039f9] syd [info] postgres_exporter | ERRO[0036] Error opening connection to database (postgresql://flypgadmin:PASSWORD_REMOVED@[fdaa:0:1af2:a7b:b92:0:32e5:2]:5433/postgres?sslmode=disable): dial tcp [fdaa:0:1af2:a7b:b92:0:32e5:2]:5433: connect: connection refused source="postgres_exporter.go:1658"
2021-08-12T13:40:04.094255055Z app[901039f9] syd [info] keeper | 3274/56420 kB (5%), 0/1 tablespace (/data/postgres/base/13408/2609 )
2021-08-12T13:40:05.010561326Z app[901039f9] syd [info] keeper | 8429/56420 kB (14%), 0/1 tablespace (/data/postgres/base/13407/3456_vm )
2021-08-12T13:40:06.125097721Z app[901039f9] syd [info] keeper | 14724/56420 kB (26%), 0/1 tablespace (/data/postgres/base/13407/1255 )
2021-08-12T13:40:07.007539263Z app[901039f9] syd [info] keeper | 21414/56420 kB (37%), 0/1 tablespace (/data/postgres/base/16389/13268_fsm)
2021-08-12T13:40:08.008708227Z app[901039f9] syd [info] keeper | 29732/56420 kB (52%), 0/1 tablespace (/data/postgres/base/1/3380 )
2021-08-12T13:40:09.007874882Z app[901039f9] syd [info] keeper | 39990/56420 kB (70%), 0/1 tablespace (/data/postgres/base/16387/1247 )
2021-08-12T13:40:10.055408311Z app[901039f9] syd [info] keeper | 51875/56420 kB (91%), 0/1 tablespace (/data/postgres/base/16387/25342 )
2021-08-12T13:40:10.212885417Z app[b487a403] lhr [info] keeper | 2021-08-12 13:40:10.210 UTC [2201] WARNING: skipping special file "./postgresql.auto.conf"
2021-08-12T13:40:10.347474967Z app[901039f9] syd [info] keeper | WARNING: skipping special file "./postgresql.auto.conf"
2021-08-12T13:40:10.387957958Z app[901039f9] syd [info] keeper | 56430/56430 kB (100%), 0/1 tablespace (/data/postgres/global/pg_control )
2021-08-12T13:40:10.394445162Z app[901039f9] syd [info] keeper | 56430/56430 kB (100%), 1/1 tablespace
2021-08-12T13:40:10.397799212Z app[901039f9] syd [info] keeper | pg_basebackup: write-ahead log end point: 1/20000138
2021-08-12T13:40:10.399196700Z app[901039f9] syd [info] keeper | pg_basebackup: waiting for background process to finish streaming ...
2021-08-12T13:40:11.477746745Z app[901039f9] syd [info] keeper | 2021-08-12T13:40:11.469Z ERROR cmd/keeper.go:688 cannot get configured pg parameters {"error": "dial unix /tmp/.s.PGSQL.5433: connect: no such file or directory"}
2021-08-12T13:40:11.660026488Z app[901039f9] syd [info] keeper | pg_basebackup: syncing data to disk ...
2021-08-12T13:40:11.936237554Z app[901039f9] syd [info] keeper | pg_basebackup: base backup completed
2021-08-12T13:40:12.152505691Z app[901039f9] syd [info] keeper | 2021-08-12 13:40:12.147 UTC [667] LOG: starting PostgreSQL 12.5 (Debian 12.5-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
2021-08-12T13:40:12.157387258Z app[901039f9] syd [info] keeper | 2021-08-12 13:40:12.153 UTC [667] LOG: listening on IPv6 address "fdaa:0:1af2:a7b:b92:0:32e5:2", port 5433
2021-08-12T13:40:12.160426903Z app[901039f9] syd [info] keeper | 2021-08-12 13:40:12.158 UTC [667] LOG: listening on Unix socket "/tmp/.s.PGSQL.5433"
2021-08-12T13:40:12.260311975Z app[901039f9] syd [info] keeper | 2021-08-12 13:40:12.256 UTC [674] FATAL: the database system is starting up
2021-08-12T13:40:12.263190738Z app[901039f9] syd [info] keeper | 2021-08-12T13:40:12.260Z ERROR cmd/keeper.go:990 failed to get replication slots {"error": "pq: the database system is starting up"}
2021-08-12T13:40:12.266179513Z app[901039f9] syd [info] keeper | 2021-08-12T13:40:12.263Z ERROR cmd/keeper.go:1584 error updating replication slots {"error": "pq: the database system is starting up"}
2021-08-12T13:40:12.270695471Z app[901039f9] syd [info] keeper | 2021-08-12 13:40:12.266 UTC [673] LOG: database system was interrupted; last known up at 2021-08-12 13:40:00 UTC
2021-08-12T13:40:12.372830023Z app[901039f9] syd [info] keeper | 2021-08-12 13:40:12.369 UTC [677] FATAL: the database system is starting up
2021-08-12T13:40:12.376605712Z app[901039f9] syd [info] keeper | 2021-08-12T13:40:12.373Z ERROR cmd/keeper.go:1686 failed to check if restart is required {"error": "pq: the database system is starting up"}
2021-08-12T13:40:12.386134562Z app[901039f9] syd [info] keeper | 2021-08-12 13:40:12.384 UTC [673] LOG: entering standby mode
2021-08-12T13:40:12.389833985Z app[901039f9] syd [info] keeper | 2021-08-12 13:40:12.388 UTC [673] LOG: redo starts at 1/20000028
2021-08-12T13:40:12.392749613Z app[901039f9] syd [info] keeper | 2021-08-12 13:40:12.390 UTC [673] LOG: consistent recovery state reached at 1/20000138
2021-08-12T13:40:12.394993423Z app[901039f9] syd [info] keeper | 2021-08-12 13:40:12.392 UTC [667] LOG: database system is ready to accept read only connections
2021-08-12T13:40:13.950058094Z app[901039f9] syd [info] keeper | 2021-08-12 13:40:13.943 UTC [681] LOG: started streaming WAL from primary at 1/21000000 on timeline 2
2021-08-12T13:40:17.469076269Z app[901039f9] syd [info] postgres_exporter | INFO[0050] Established new database connection to "fdaa:0:1af2:a7b:b92:0:32e5:2:5433". source="postgres_exporter.go:970"
2021-08-12T13:40:17.573985500Z app[901039f9] syd [info] postgres_exporter | INFO[0050] Established new database connection to "fdaa:0:1af2:a7b:b92:0:32e5:2:5433". source="postgres_exporter.go:970"
2021-08-12T13:40:17.587896126Z app[901039f9] syd [info] postgres_exporter | INFO[0051] Semantic Version Changed on "fdaa:0:1af2:a7b:b92:0:32e5:2:5433": 0.0.0 -> 12.5.0 source="postgres_exporter.go:1539"
2021-08-12T13:40:17.649549049Z app[901039f9] syd [info] postgres_exporter | INFO[0051] Established new database connection to "fdaa:0:1af2:a7b:b92:0:32e5:2:5433". source="postgres_exporter.go:970"
2021-08-12T13:40:17.665345489Z app[901039f9] syd [info] postgres_exporter | INFO[0051] Semantic Version Changed on "fdaa:0:1af2:a7b:b92:0:32e5:2:5433": 0.0.0 -> 12.5.0 source="postgres_exporter.go:1539"
2021-08-12T13:40:17.670165672Z app[901039f9] syd [info] postgres_exporter | INFO[0051] Semantic Version Changed on "fdaa:0:1af2:a7b:b92:0:32e5:2:5433": 0.0.0 -> 12.5.0 source="postgres_exporter.go:1539"
2021-08-12T13:41:16.613781454Z app[8ff3cc74] lhr [info] sentinel | 2021-08-12T13:41:16.609Z WARN cmd/sentinel.go:276 no keeper info available {"db": "2739d9d0", "keeper": "fdaa01af2a7ba9801e822"}
2021-08-12T13:41:44.622141647Z app[8ff3cc74] lhr [info] sentinel | 2021-08-12T13:41:44.617Z WARN cmd/sentinel.go:276 no keeper info available {"db": "b9027623", "keeper": "fdaa01af2a7bb92032e52"}
2021-08-12T13:43:45.374627494Z app[8ff3cc74] lhr [info] sentinel | 2021-08-12T13:43:45.370Z WARN cmd/sentinel.go:276 no keeper info available {"db": "b9027623", "keeper": "fdaa01af2a7bb92032e52"}
2021-08-12T13:48:47.027814381Z app[8ff3cc74] lhr [info] sentinel | 2021-08-12T13:48:47.024Z WARN cmd/sentinel.go:276 no keeper info available {"db": "1c4d83a2", "keeper": "fdaa01af2a7ba9a01e812"}
2021-08-12T13:49:10.905443936Z app[8ff3cc74] lhr [info] keeper | 2021-08-12T13:49:10.900Z ERROR cmd/keeper.go:1010 error retrieving cluster data {"error": "Unexpected response code: 429"}
2021-08-12T13:49:10.968201702Z app[b487a403] lhr [info] keeper | 2021-08-12T13:49:10.963Z ERROR cmd/keeper.go:1010 error retrieving cluster data {"error": "Unexpected response code: 429"}
2021-08-12T13:50:22.520061925Z app[8ff3cc74] lhr [info] keeper | 2021-08-12T13:50:22.515Z ERROR cmd/keeper.go:1010 error retrieving cluster data {"error": "Unexpected response code: 429"}
2021-08-12T13:50:33.442130943Z app[8ff3cc74] lhr [info] sentinel | 2021-08-12T13:50:33.438Z WARN cmd/sentinel.go:276 no keeper info available {"db": "b9027623", "keeper": "fdaa01af2a7bb92032e52"}