Inconsistent Metrics Polling

Hello! I’m noticing some weirdness with Prometheus metrics getting polled from my service. The documentation on metrics says that it will be periodically polled a few times per minute.

Here’s some sampled timestamps when /metrics is called. This endpoint is the only one on port 9091 and only available on port 9091, so it shouldn’t be receiving calls from anywhere else.

Timestamps

2022-02-11 17:49:46.858
2022-02-11 17:49:48.605
2022-02-11 17:50:01.863
2022-02-11 17:50:02.162
2022-02-11 17:50:16.792
2022-02-11 17:50:16.873
2022-02-11 17:50:31.858
2022-02-11 17:50:34.479
2022-02-11 17:50:46.857
2022-02-11 17:50:47.004
2022-02-11 17:51:00.526
2022-02-11 17:51:01.854
2022-02-11 17:51:16.858
2022-02-11 17:51:18.816
2022-02-11 17:51:31.855
2022-02-11 17:51:33.184
2022-02-11 17:51:45.100
2022-02-11 17:51:46.857
2022-02-11 17:52:01.853
2022-02-11 17:52:04.550
2022-02-11 17:52:16.861
2022-02-11 17:52:18.507
2022-02-11 17:52:31.859
2022-02-11 17:52:33.378
2022-02-11 17:52:46.855
2022-02-11 17:52:47.439
2022-02-11 17:53:00.126
2022-02-11 17:53:01.858
2022-02-11 17:53:16.858
2022-02-11 17:53:19.011
2022-02-11 17:53:31.860
2022-02-11 17:53:31.870
2022-02-11 17:53:45.265
2022-02-11 17:53:46.855
2022-02-11 17:54:01.854
2022-02-11 17:54:03.964

It seems like the scrape interval is about 15 seconds, and this is what I have my meter registry configured for when generating the endpoint contents. Is 15 seconds the correct scrape interval? It would be helpful to know exactly what it should be set to.

Is this potentially why custom metrics and metrics from Fly have null values occasionally?

Example Custom Metric null gap

image

Example Fly Metric null gap (fly_instance_net_sent_bytes, fly_instance_net_recv_bytes)

image