fly.io metrics endpoint return 401

[bachree]

When I use the prometheus on Grafana it keeps returning 401.

When I test with curl in Powershell there is no return body and:

curl.exe -v “https://api.fly.io/prometheus/personal/api/v1/query” --data-urlencode "query=sum(increase(fly_edge_http_responses_count)) by (app, status)"
-H “Authorization: FlyV1 MYTOKEN”

output

• Host api.fly.io:443 was resolved.

• IPv6: (none)

• IPv4: 77.83.143.220

• Trying 77.83.143.220:443…

• schannel: disabled automatic use of client certificate

• ALPN: curl offers http/1.1

• ALPN: server accepted http/1.1

• Connected to api.fly.io (77.83.143.220) port 443

• using HTTP/1.x
  POST /prometheus/personal/api/v1/query HTTP/1.1
  Host: api.fly.io
  User-Agent: curl/8.14.1
  Accept: /
  Authorization: FlyV1 MYTOKEN
  Content-Length: 8
  Content-Type: application/x-www-form-urlencoded

• upload completely sent off: 8 bytes

• schannel: remote party requests renegotiation

• schannel: renegotiating SSL/TLS connection

• schannel: SSL/TLS connection renegotiated

• schannel: server close notification received (close_notify)
  < HTTP/1.1 400 Bad Request
  < content-length: 0
  < date: Tue, 14 Oct 2025 21:57:14 GMT
  <

• Connection #0 to host api.fly.io left intact

What could be the problem? The token is generated using

fly tokens create org

[mayailurus]

@bachree

< HTTP/1.1 400 Bad Request

Hm… Note that this one is 400 Bad Request and not 401 Unauthorized, so it might not be the token that’s the problem here.

(Usually it is the token, so that wasn’t a bad guess. Maybe that should be one of the logon poems, actually, but I don’t have the knack—unfortunately.)

Anyway, an almost identical sequence to the one you tried just worked for me on Debian Bookworm. (I used fly tokens create readonly --expiry 2h --name grafana-tst for the Macaroon, but I doubt the read-only-ness or the short expiration made a difference.)

One discrepancy that caught my eye is that you reported Content-Length: 8 for the request body, which is much shorter than the --data-urlencode string. Mine was 81, so perhaps it was just a glitch during anonymization. Still, it might be worth double-checking that you weren’t bumping into an odd corner case in PowerShell string-literal syntax, etc.

Hope this helps a little!

[bachree]

Here is another attempt from a linux system:

curl -v "https://api.fly.io/prometheus/personal/api/v1/query" --data-urlencode "query=sum(increase(fly_edge_http_responses_count)) by (app, status)" -H "Authorization: FlyV1 $TOKEN"
* Host api.fly.io:443 was resolved.
* IPv6: 2a09:8280:1:f28:246e:d6a:949:dbbf
* IPv4: 77.83.143.220
*   Trying [2a09:8280:1:f28:246e:d6a:949:dbbf]:443...
* Immediate connect fail for 2a09:8280:1:f28:246e:d6a:949:dbbf: Network is unreachable
*   Trying 77.83.143.220:443...
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: none
*  CApath: /etc/ssl/certs
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / x25519 / id-ecPublicKey
* ALPN: server accepted http/1.1
* Server certificate:
*  subject: CN=api.fly.io
*  start date: Sep 15 11:20:57 2025 GMT
*  expire date: Dec 14 11:20:56 2025 GMT
*  subjectAltName: host "api.fly.io" matched cert's "api.fly.io"
*  issuer: C=US; O=Let's Encrypt; CN=E7
*  SSL certificate verify ok.
*   Certificate level 0: Public key type EC/prime256v1 (256/128 Bits/secBits), signed using ecdsa-with-SHA384
*   Certificate level 1: Public key type EC/secp384r1 (384/192 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 2: Public key type RSA (4096/152 Bits/secBits), signed using sha256WithRSAEncryption
* Connected to api.fly.io (77.83.143.220) port 443
* using HTTP/1.x
> POST /prometheus/personal/api/v1/query HTTP/1.1
> Host: api.fly.io
> User-Agent: curl/8.14.0
> Accept: */*
> Authorization: FlyV1 $TOKEN
> Content-Length: 81
> Content-Type: application/x-www-form-urlencoded
> 
* upload completely sent off: 81 bytes
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
< HTTP/1.1 200 OK
< access-control-allow-origin: *
< content-type: application/json
< date: Wed, 15 Oct 2025 01:37:30 GMT
< fly-request-id: 01K7JR7RWY8S3HNP42Q3WCMT07-dfw
< server: Fly/5d9a8537e (2025-10-13)
< vary: Accept-Encoding
< via: 1.1 fly.io, 1.1 fly.io, 1.1 fly.io, 1.1 fly.io
< x-server-hostname: 217811eeec9389
< transfer-encoding: chunked
< 
* Connection #0 to host api.fly.io left intact
{"status":"success","isPartial":false,"data":{"resultType":"vector","result":[{"metric":{"app":"$APPNAME","status":"200"},"value":[1760492250,"5"]},{"metric":{"app":"$APPNAME","status":"301"}
,"value":[1760492250,"6"]},{"metric":{"app":"$APPNAME","status":"404"},"value":[1760492250,"2"]}]},"stats":{"seriesFetched": "11","executionTimeMsec":267}}

This time it returns 200.

So, back to Grafana, I don’t understand why it return 401

I checked countless times to make sure the address and the header value is correct but it keep returning 401

[bachree]

I got it to work eventually. It must have been a line break or something. Pasted the key to notepad first and then copied to prometheus and it worked.