# Grafana Cloud Fly dashboard/Prometheus issue? **URL:** \ **Category:** General\ **Created:** [June 11, 2022, 11:00am UTC](https://community.fly.io/t/grafana-cloud-fly-dashboard-prometheus-issue/5586 "2022-06-11T11:00:12Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![Whistler](https://avatars.discourse-cdn.com/v4/letter/w/a3d4f5/32.png) [@Whistler](https://community.fly.io/u/Whistler)\ **Post date:** [June 11, 2022, 11:00am UTC](https://community.fly.io/t/grafana-cloud-fly-dashboard-prometheus-issue/5586/1 "2022-06-11T11:00:12Z")
Has anyone else’s Grafana Cloud [Fly dashboard](https://grafana.com/grafana/dashboards/14741)(s) stopped working when trying to open/load (note: _if the dashboard is already loaded in your browser it’s probably working ok_)? and are you able to to Explore your [Fly Prometheus source(s)](https://fly.io/docs/reference/metrics/) from Grafana Cloud? Putting aside the specifics of the dashboard problem, where is seems to fail to fetch the region/host/app details, I’ve also noticed that Explore doesn’t appear to work ("! Unexpected Error") ☹ . As far as I’m aware my (Fly) Prometheus sources haven’t changed within Grafana Cloud and have been working well for quite some time. Another existing (not the [Fly](https://grafana.com/grafana/dashboards/14741)) dashboard I have in Grafana Cloud using a Fly Prometheus source is also working ok. ---
**Author:** ![sashaafm](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/sashaafm/32/3027_2.png) [@sashaafm](https://community.fly.io/u/sashaafm)\ **Post date:** [June 14, 2022, 1:04pm UTC](https://community.fly.io/t/grafana-cloud-fly-dashboard-prometheus-issue/5586/2 "2022-06-14T13:04:28Z")
I believe I’m having the same issue! ---
**Author:** ![tao](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/tao/32/3504_2.png) [@tao](https://community.fly.io/u/tao)\ **Post date:** [June 14, 2022, 2:08pm UTC](https://community.fly.io/t/grafana-cloud-fly-dashboard-prometheus-issue/5586/3 "2022-06-14T14:08:35Z")
Same, I set mine up last week and now it’s not working anymore… just get a bunch of errors popping up. ---
**Author:** ![fubhy](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/fubhy/32/14421_2.png) [@fubhy](https://community.fly.io/u/fubhy)\ **Post date:** [June 14, 2022, 3:33pm UTC](https://community.fly.io/t/grafana-cloud-fly-dashboard-prometheus-issue/5586/4 "2022-06-14T15:33:16Z")
Facing the same issue ---
**Author:** ![kurt](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/kurt/32/43757_2.png) [@kurt](https://community.fly.io/u/kurt)\ **Post date:** [June 14, 2022, 3:40pm UTC](https://community.fly.io/t/grafana-cloud-fly-dashboard-prometheus-issue/5586/5 "2022-06-14T15:40:56Z")
Can you all screenshot the errors you’re seeing? My test dashboards aren’t showing this problem. Or I’m not looking in the right place. ---
**Author:** ![Whistler](https://avatars.discourse-cdn.com/v4/letter/w/a3d4f5/32.png) [@Whistler](https://community.fly.io/u/Whistler)\ **Post date:** [June 14, 2022, 3:45pm UTC](https://community.fly.io/t/grafana-cloud-fly-dashboard-prometheus-issue/5586/6 "2022-06-14T15:45:10Z")
kurt - are you able to `Explore` your Fly Prometheus source(s) from within Grafana Cloud? and also, are you loading your Dashboard(s) afresh (e.g. new tab) as opposed to looking at a Dashboard(s) that you already have open (which I suspect are still ok)? ---
**Author:** ![kurt](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/kurt/32/43757_2.png) [@kurt](https://community.fly.io/u/kurt)\ **Post date:** [June 14, 2022, 3:48pm UTC](https://community.fly.io/t/grafana-cloud-fly-dashboard-prometheus-issue/5586/7 "2022-06-14T15:48:51Z")
Yes, I see this from a fresh, no cache load: ![image](https://global.discourse-cdn.com/fly1/original/3X/c/8/c8ce6f7edc7c0a523f55eddee8b659c3ebbbe7f6.png) It took a little while for the region/host to load when I first logged in. Maybe 10s of spinners. But it worked. I can also see all the available series in the explore view: ![image](https://global.discourse-cdn.com/fly1/original/3X/9/6/965e4010f145ee3fb0ab53bb5fdade0551fda0e7.png) I don’t use Grafana cloud though. This is a Grafana app I run on [Fly.io](http://Fly.io). ---
**Author:** ![Whistler](https://avatars.discourse-cdn.com/v4/letter/w/a3d4f5/32.png) [@Whistler](https://community.fly.io/u/Whistler)\ **Post date:** [June 14, 2022, 3:57pm UTC](https://community.fly.io/t/grafana-cloud-fly-dashboard-prometheus-issue/5586/8 "2022-06-14T15:57:15Z")
This is the error that appears when going to Explore on Grafana Cloud: ![Grafana Issue](https://global.discourse-cdn.com/fly1/original/3X/e/1/e10279dee414ff56e993e9d449cfea64baa14d88.jpeg) ---
**Author:** ![kurt](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/kurt/32/43757_2.png) [@kurt](https://community.fly.io/u/kurt)\ **Post date:** [June 14, 2022, 3:58pm UTC](https://community.fly.io/t/grafana-cloud-fly-dashboard-prometheus-issue/5586/9 "2022-06-14T15:58:31Z")
I don’t know enough about Grafana, but can you see if the browser console shows any errors? Also it’d be worth trying Grafana on [Fly.io](http://Fly.io) to see if you can replicate. This should get you going: [GitHub - fly-apps/grafana: Run Grafana on Fly](https://github.com/fly-apps/grafana) ---
**Author:** ![fubhy](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/fubhy/32/14421_2.png) [@fubhy](https://community.fly.io/u/fubhy)\ **Post date:** [June 14, 2022, 4:02pm UTC](https://community.fly.io/t/grafana-cloud-fly-dashboard-prometheus-issue/5586/10 "2022-06-14T16:02:13Z")
Hey @kurt . These are the errors I’m seeing. Note: I also can’t curl the Prometheus API resources (with a working API token) either. ![Screenshot from 2022-06-14 18-01-09](https://global.discourse-cdn.com/fly1/original/3X/6/1/612187066a5e567795130dec8fa90d35ed697f71.png) ---
**Author:** ![kurt](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/kurt/32/43757_2.png) [@kurt](https://community.fly.io/u/kurt)\ **Post date:** [June 14, 2022, 4:03pm UTC](https://community.fly.io/t/grafana-cloud-fly-dashboard-prometheus-issue/5586/11 "2022-06-14T16:03:51Z")
Will you paste the curl output? What response are you actually getting? ---
**Author:** ![fubhy](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/fubhy/32/14421_2.png) [@fubhy](https://community.fly.io/u/fubhy)\ **Post date:** [June 14, 2022, 4:08pm UTC](https://community.fly.io/t/grafana-cloud-fly-dashboard-prometheus-issue/5586/12 "2022-06-14T16:08:04Z")
```auto curl 'https://api.fly.io/prometheus/avantgarde-finance/api/v1/status/buildinfo' --header 'Authorization: Bearer ' ``` ```auto remoteAddr: "10.123.11.76:53346", X-Forwarded-For: "84.115.209.22, 77.83.143.220, 213.188.208.17, 205.234.149.66"; requestURI: /select/36393/prometheus/api/v1/status/buildinfo; unsupported path requested: "/select/36393/prometheus/api/v1/status/buildinfo" ``` Status code is 400 ---
**Author:** ![kurt](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/kurt/32/43757_2.png) [@kurt](https://community.fly.io/u/kurt)\ **Post date:** [June 14, 2022, 4:27pm UTC](https://community.fly.io/t/grafana-cloud-fly-dashboard-prometheus-issue/5586/13 "2022-06-14T16:27:47Z")
Try this one? That `buildinfo` path is not supported in VictoriaMetrics (what we use under the covers). It’s normal for that to error. This should return something though: ```auto curl 'https://api.fly.io/prometheus/fly/api/v1/label/region/values' --header "Authorization: Bearer $(fly auth token)" ``` ---
**Author:** ![fubhy](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/fubhy/32/14421_2.png) [@fubhy](https://community.fly.io/u/fubhy)\ **Post date:** [June 14, 2022, 4:30pm UTC](https://community.fly.io/t/grafana-cloud-fly-dashboard-prometheus-issue/5586/14 "2022-06-14T16:30:09Z")
That works but it’s not the path that Grafana is trying to call it seems. I’m fairly confident that it worked a few days / weeks ago. ---
**Author:** ![kurt](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/kurt/32/43757_2.png) [@kurt](https://community.fly.io/u/kurt)\ **Post date:** [June 14, 2022, 4:35pm UTC](https://community.fly.io/t/grafana-cloud-fly-dashboard-prometheus-issue/5586/15 "2022-06-14T16:35:39Z")
That’s the path to the first error. Grafana is showing a 401, though, which is what you get when you’re not authenticated. Can you try re-inputing your auth token in the Grafana source? ---
**Author:** ![fubhy](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/fubhy/32/14421_2.png) [@fubhy](https://community.fly.io/u/fubhy)\ **Post date:** [June 14, 2022, 4:41pm UTC](https://community.fly.io/t/grafana-cloud-fly-dashboard-prometheus-issue/5586/16 "2022-06-14T16:41:02Z")
Just did. The metrics and my dashboards work but the query builder & explorer doesn’t work (same error). So the auth token works for scraping the metrics but not for the other endpoints? ---
**Author:** ![fubhy](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/fubhy/32/14421_2.png) [@fubhy](https://community.fly.io/u/fubhy)\ **Post date:** [June 14, 2022, 4:57pm UTC](https://community.fly.io/t/grafana-cloud-fly-dashboard-prometheus-issue/5586/17 "2022-06-14T16:57:37Z")
Btw. the endpoint it’s trying (and failing) to call when loading the list of available metrics in the Explorer / Inspector is ```auto https://avantgarde.grafana.net/api/datasources/32/resources/api/v1/label/ __name__ /values?start=1655222080&end=1655225680 ``` And that returns a `401` despite the rest of the metrics working via the same datasource. ---
**Author:** ![kurt](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/kurt/32/43757_2.png) [@kurt](https://community.fly.io/u/kurt)\ **Post date:** [June 14, 2022, 5:37pm UTC](https://community.fly.io/t/grafana-cloud-fly-dashboard-prometheus-issue/5586/18 "2022-06-14T17:37:51Z")
See if this works for you over curl? ```auto curl 'https://api.fly.io/prometheus/avantgarde-finance/api/v1/label/ __name__ /values' --header "Authorization: Bearer $(fly auth token)" -D - ``` ---
**Author:** ![ian1](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/ian1/32/3620_2.png) [@ian1](https://community.fly.io/u/ian1)\ **Post date:** [June 14, 2022, 6:25pm UTC](https://community.fly.io/t/grafana-cloud-fly-dashboard-prometheus-issue/5586/19 "2022-06-14T18:25:57Z")
Not sure if related, but I stopped getting metrics from AMS and SJC at 9:20am PT today although the instances are still healthy and serving traffic. My project’s other region (HKG) is still getting metrics in Grafana. I have a separate project in AMS (same organization) that still has metrics. Edit: `fly restart` fixed it, but a bit worrisome that metrics just stopped from some instances. ---
**Author:** ![Whistler](https://avatars.discourse-cdn.com/v4/letter/w/a3d4f5/32.png) [@Whistler](https://community.fly.io/u/Whistler)\ **Post date:** [June 14, 2022, 10:07pm UTC](https://community.fly.io/t/grafana-cloud-fly-dashboard-prometheus-issue/5586/20 "2022-06-14T22:07:30Z")
Having now reverse proxied the Grafana Cloud\>Fly API Prometheus requests, I believe I can see the problem… Grafana Cloud _does_ send the `authorization:` header when clicking Save & Test on the Prometheus source. It _ **doesn’t** _ however send the `authorization:` header when fetching the list of Metrics with Explore (and as a result Fly responds with a HTTP 401). If, on the reverse proxy, I force the `authorization: Bearer ` into the requests - it then works (HTTP 200 from Fly). With no knowledge of what is “normal” for Prometheus requests (i.e. are some available without authentication?) I don’t know if a request (without the `authorization:` header) for: > GET /prometheus/\/api/v1/rules Would normally be responded to with a HTTP 200. If Prometheus requests, or at least some paths, are usually allowed (_or have previously been allowed by Fly_, i.e. prior to late last week) - without an `authorization:` header - then this may be resolvable by Fly. If however `authorization:` has always been required by Fly for _all_ Prometheus paths - then I can only assume Grafana Cloud have made a breaking change 😢 . [Next page](https://community.fly.io/t/grafana-cloud-fly-dashboard-prometheus-issue/5586.md?page=2)