Managed Prometheus drops our 53 KiB metrics response and Grafana data becomes intermittent

Questions / Help
1

Hi folks.
Today we encountered this situation: we exposed a metrics endpoint to Fly.io Prometheus for metric collection. On top of the original metrics, we added some new metrics and deployed the update at 17:30. However, after the deployment, the lines on the graph became intermittent and broken.

image
image632×268 19.3 KB

This is a sample response from our metrics endpoint.

image
image1494×938 158 KB

We later found in the official documentation that responses larger than 16 KiB are discarded. Could that be the cause?

image
image1293×643 12.4 KB

2

Hi there, where did you find that responses larger than 16Kb are discarded?

3

image
image2560×1398 195 KB

4

You can click this website and find the 16 KiB limit there.

5

Thanks! That’s an error “off by 3 orders of magnitude”. The Prometheus metrics payload size limit is 16MB.

Cheers!