After the incident this weekend, my app hasn’t been able to successfully reconnect with my postgres instance. The logs in my app are regularly hitting errors such as these:
proxy[19baed19] iad [error] instance refused connection
and a ton of proxy errors from different regions:
2023-01-23T19:49:37.620 proxy[19baed19] iad [warn] Could not proxy HTTP request. Retrying in 1000 ms (attempt 20)
2023-01-23T19:49:38.174 proxy[19baed19] mia [warn] Could not proxy HTTP request. Retrying in 1000 ms (attempt 20)
2023-01-23T19:49:38.278 proxy[19baed19] sjc [warn] Could not proxy HTTP request. Retrying in 1000 ms (attempt 20)
2023-01-23T19:49:38.620 proxy[19baed19] hkg [warn] Could not proxy HTTP request. Retrying in 1000 ms (attempt 20)
2023-01-23T19:49:38.894 proxy[19baed19] sea [warn] Could not proxy HTTP request. Retrying in 1000 ms (attempt 20)
2023-01-23T19:49:38.954 proxy[19baed19] fra [warn] Could not proxy HTTP request. Retrying in 1000 ms (attempt 20)
2023-01-23T19:49:39.089 proxy[19baed19] syd [warn] Could not proxy HTTP request. Retrying in 1000 ms (attempt 20)
And timing out:
2023-01-23T19:46:23.392 proxy[19baed19] iad [error] timed out while connecting to instance
2023-01-23T19:46:23.441 proxy[19baed19] iad [error] timed out while connecting to instance
2023-01-23T19:46:24.442 proxy[19baed19] iad [error] timed out while connecting to instance
2023-01-23T19:46:25.441 proxy[19baed19] iad [error] timed out while connecting to instance
2023-01-23T19:46:25.739 proxy[19baed19] iad [error] timed out while connecting to instance
2023-01-23T19:46:25.746 proxy[19baed19] iad [error] timed out while connecting to instance
2023-01-23T19:46:26.895 proxy[19baed19] iad [error] timed out while connecting to instance
2023-01-23T19:46:28.138 runner[19baed19] iad [info] Starting instance
2023-01-23T19:46:28.594 proxy[19baed19] iad [error] timed out while connecting to instance
2023-01-23T19:46:28.936 proxy[19baed19] iad [error] timed out while connecting to instance
2023-01-23T19:46:28.990 proxy[19baed19] iad [error] timed out while connecting to instance
On the database side, I’ve restarted a couple times but haven’t seen success. Seeing these errors regularly:
iad [info] keeper | 2023-01-23 19:36:54.834 UTC [29704] FATAL: connection to client lost
2023-01-23T19:43:21.853 app[73287903f11685] iad [info] keeper | 2023-01-23 19:43:21.850 UTC [32016] LOG: could not receive data from client: Connection reset by peer
2023-01-23T19:43:21.863 app[73287903f11685] iad [info] keeper | 2023-01-23 19:43:21.863 UTC [32015] LOG: could not send data to client: Connection reset by peer
I’m also seeing keeper errors on the DB side:
WARN cmd/sentinel.go:276 no keeper info available {"db": "fe981a24", "keeper": "92f03176832"}