Fly.io Prometheus as Datasource causes DatasourceNoData

[Akito]

My setup pulls in some stats from Fly.io’s Prometheus, based on this guide.

Self-managed monitoring and alerting on Fly.io with Prometheus + Alertmanager

Alerts on Fly.io? While currently Fly.io does not have built-in alerting functionality (not entirely accurate, we alert via e-mail when an application OOMs), the good news is that you can set up your own monitoring and alerting cluster! You can configure it to alert on almost any condition that can be monitored and expressed as a time-series metric (read: basically anything!), and send notifications to individuals or groups with a dozen supported mechanisms out of the box (Slack, e-mail, PagerDu…

Encountered this issue, except the dashboard confirms those alerts.

Fly prometheus metrics have been unreliable

I run grafana on-premises and query fly’s prometheus metrics to fire alerts. I’ve set up a grafana alert on fly_instance_up to detect downtime and the alert has been going crazy today for some reason. This is my alert query: [image] But when I query the metric from the explore tab, the graph doesn’t look like that. This isn’t the only instance of my alerts becoming unreliable. Frequently, I get alerted for DatasourceNoData or DatasourceError, and when I go try it manually, it’s perfectly fi…

Since this linked issue is ~1.5 years old, I wanted to have the solution clarified and make sure, it is up to date.

Is the DatasourceNoData alert an indicator of pulling metrics too frequently? As I set up the alerting based on the official guide, linked above, I would’ve expected it to use compatible frequencies. I did not make them stricter, than that.

Or is Fly.io’s datasource really that unreliable?

How can I solve this issue, without increasing the metrics pull frequency to a relatively large duration?