Rendering of metrics is wrong for some longer duration graphs

Hello Fly friends!

I think I found a bug on the metrics screen. Notice that pattern of the 429s that start a few minutes after an hour and then stop for a bit right at the hour? Notice how they appear to stop around noon?

I was looking at the 2day graph of 429s to track down being throttled and saw data that suggested they had dropped. This was actually incorrect upon inspection as shown in my second screenshot and also when I shortened the period and looked at the graphs there.

This graph led to a misconception that caused me to think an issue was resolved when it wasn’t! When you have a chance, I’d love it if you could take a look.

p.s. it looks like there’s some sort of query language or something for metrics… where do I go to learn more on that?

Ah I think it’s collapsing all the null values left. That’s interesting, we’ll have a look.

The query language is from Prometheus, you can access it directly when our graphs aren’t enough: Early access: build Grafana dashboard from Fly metrics