Dashboard metrics are not available

Looks like fly dashboard metrics disappeared for all my apps and showing “No data available” (dc: lax)

2 Likes

Can confirm also.

Seems like metrics are dead across the board. My grafana server can’t read metrics either.

Yeah, all metrics queries are coming back with a 500 it seems.

Still down for me as well.

Have been down for me since yesterday.

The status page is showing No Downtime or issues though: https://status.flyio.net/

It seems to be affecting at least a handful of customers, it would be good to have someone from Fly give an update or assistance.

Thanks for the reports, we’re looking into this and will keep you updated.

Confirmed the issue, there is a bug that’s currently causing 500 errors for all Prometheus queries to personal organizations.

A fix for the bug has been deployed, metrics queries should be working for personal organizations again. Please let me know if you’re experiencing any further issues!

1 Like

Would it be possible to apply a permanent fix for the other Prometheus (vector) issue: Lack of Prometheus metrics, sometimes, after deploy (host specific?)?

I appreciate that Prometheus metrics may be “beta” functionality but the problem manifested over a week ago and I don’t believe the root cause has yet been resolved, nor any mention of the issue on https://status.flyio.net/ .

I still can’t get any metrics in my Grafana dashboards so it seems the issue is still present. I can see all the metrics in my “/metrics” endpoint with no problem.

The issue described in this thread has been fixed, I can confirm that Prometheus queries to personal organizations are still working properly.

It’s possible you’re facing a separate configuration issue, to diagnose further, first take a look at our docs on setting up a Grafana datasource to make sure your configuration matches. If you’re setting up any custom metrics make sure that you’ve configured your app to scrape your /metrics endpoint. If that doesn’t work, try doing a manual curl request and share the results if the API is not working correctly for you.