Incorrect app concurrency metrics affecting Fly Proxy scaling

[mayailurus]

Hm… The left part of that last graph in particular looks like the old “zeroing” behavior, :

@PeterCxy

Long-running connections, such as Websockets and database connections, often [did] not get counted. This [was] an unfortunate side effect of garbage collection done in metrics-exporter-prometheus. GC’ing unused metrics is essential as machines and apps [got] created and deleted all the time on our platform, but it also [meant] that any app who mostly handles longer connections (longer than a minute or two) [would] have their load metric reset to 0 very often.

That was changed subsequently, but maybe a few proxy nodes have reverted to the old ways?

---

I’d also suggest saying a bit more, if you could, about what the clients are doing overall. E.g., are these WebSockets that are mostly idle, or very active RTMP, …? And it would probably be prudent to post the full fly.toml, since there were some nuances mentioned in the past with multiple services, etc.

(Feel free to * out any names that you consider sensitive, but show the full structure.)

Hope this helps a little!