HTTP latency jumps on Prometheus

Quick bit of context: Running a Phoenix app, has an internal endpoint as well as an external endpoint. Did try to switch to multi-region by scaling in amsterdam, (and in sydney) but the latency seemed to be too far between the DB and servers. Understandable.

So I rolled that back, but then looking at the metrics noticed massive baseline in all latencies as well as large numbers of requests all-of-a-sudden.

These increases seemed to happen around when I was doing the work, but when I zoomed out, I noticed that this was how it appeared prior to November. Throught Nov the graphs where approximately as I would expect.

The weird thing is, I can’t identify any endpoints that are slow and the TLS handshakes isn’t spiking either (suggesting this is an internal process hitting, and reporting slow?)

There wasn’t any deploys around the start of November (last was 29th October). What makes this all even stranger is that the app is performing as expected. Fast responses etc.

I’ve also had a thought about this potentially being about liveview longpolls? If so, would love some recommendations as to how to get visibility on that. (weird that it’s constantly longpolling though.

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.