Prometheus not collecting or exposing metrics?

Questions / Help
1

Hello Fly :wave:

I’ve had an app running on Fly free tier for quite some time now. I didn’t pay much attention to it, but now I was looking through the Grafana Cloud (it’s at v9.1.5-0100a6a) dashboards I had setup in the past and it seems like all metrics are gone.

Upon further investigation it seems like the Prometheus instance from Fly is either not collection or exposing them since when I try to perform queries in Grafana my app’s custom metrics are absent (e.g., don’t appear in the auto-complete suggestions).

Like I said I haven’t touched this app in a while so I’m not sure what could have gone wrong. The app itself hasn’t had any updates related to the telemetry and metrics component. I’ve also checked the new Grafana from Fly and it looks the same.

Any help would be appreciated!

EDIT: At least the metrics from Postgres Exporter is being consumed correctly by Prometheus, so I think this is only affecting the app itself.

1 Like
4

Hi, we were testing some new routing policies on a few hosts the day you posted this, and ran into some issues retrieving prometheus data. We rolled back the new policies and things should be working again. Let us know if that’s not the case.

5

Hey @tvdfly, I just checked my Grafana instance and there’s still no data. Is there something I should do on my end to fix this?

6

@tvdfly would you be able to help with this issue? :pray: I can’t find out what’s wrong with Prometheus/Grafana :frowning:

7

I’m suspicious that Prometheus isn’t fetching my metrics correctly. However, I can’t check that because the /targets endpoint doesn’t seem to work:

% curl https://api.fly.io/prometheus/$ORG_SLUG/api/v1/targets \
  -H "Authorization: Bearer $TOKEN"

remoteAddr: "10.123.11.60:42216", X-Forwarded-For: "2001:8a0:7244:1800:c97:5a86:bdde:f492, 2a09:8280:1:f28:246e:d6a:949:dbbf, 213.188.208.17, 172.19.0.17"; requestURI: /select/38056/prometheus/api/v1/targets; unsupported path requested: "/select/38056/prometheus/api/v1/targets"
8

The [metrics] section now seems necessary to Prometheus correctly scrape the metrics endpoint. I think this wasn’t needed in the past? Or maybe I deleted it by accident. But it’s solved now! :slight_smile:

Thank you for your time @tvdfly

1 Like
9

The custom metrics implementation actually doesn’t support the standard Prometheus /targets endpoint, even though the current documentation does incorrectly list /targets as supported- apologies for the confusion, we will get this documentation error fixed.

We’re actually working on adding extra visibility into custom-metrics scraping operations. When it’s ready, it will initially be in the form of additional metrics and log entries, rather than a Prometheus-compatible /targets endpoint.

3 Likes