We’re having an issue with a production app since yesterday, suddenly the max concurrency limit was reached (2022-09-13T12:16:00.711 proxy[7fa6e5c2] iad [warn] Instance reached connections hard limit of 25) and since it doesn’t have scaling set up I’m guessing the proxy started bouncing/queuing all the connections, effectively making the app unreachable (phoenix liveview app).
Because the load is very light, I upped the connection hard limit and requested a re-deploy but immediately noticed a steady increase in the amount of connections even when the app wasn’t being used, to the point where I had to restart the app because it had reached the new limit after some hours. Additionally, ghost vms were showing in the metrics tab(with maxed out connections).
The puzzle is that we haven’t really pushed any new code and the behavior started suddenly yesterday, I increased the hardlimit yet again and requested a new deploy but the ever increasing connections keep growing at a steady pace, even during off time for the app.
Currently there’s two vm showing in the metrics.
zombie vm: 6a99f75f - iad
reachable vm: 9ca7bf8a - iad
This application had no problem with the default hardlimit of 25. I can’t rule out an error in our configuration but it’s still very puzzling to have this happen with no recent deploys.
Any help would be much appreciated because we’re currently at a point were I’m having to manually restart the app when it’s reaching the set hardlimit.