Cannot query prometheus volume metrics

[amo]

I am trying to retrieve the used or available space for volumes in machines of one of my apps.
I do:

curl -H "Authorization: Bearer XXXXXXXXXXXX" "https://api.fly.io/prometheus/appname/api/v1/query" --data-urlencode 'query=fly_instance_filesystem_blocks_avail{app=~"^appname$", mount="/volume"}'

and

curl -H "Authorization: Bearer XXXXXXXXXXXX" "https://api.fly.io/prometheus/appname/api/v1/query" --data-urlencode 'query=fly_volume_used_pct{id="vol_xxxxxxxx"}'

…but both return empty:

{
  "status": "success",
  "isPartial": false,
  "data": {
    "resultType": "matrix",
    "result": []
  },
  "stats": {
    "seriesFetched": "0"
  }
}

Yesterday, the first one did return what I want but now does not any more. The volumes are still there and the machines are stopped.

What I’d really love to achieve is to programmatically get the data available in the dashboard with a cron job in my application:

Is there an elegant way to achieve this?

Any help is greatly appreciated!

[MatthewIngwersen]

Hi @amo, as far as I know these metrics are produced only when a Machine is running (they’re actually collected in the running Machine itself!). If your Machines will be stopped much of the time, then using the Prometheus endpoint probably won’t work well.

However, some volume usage data is also kept with a volume’s metadata, and it’s periodically updated from the same source while the Machine is running. It’s exposed via the Machines API, and it should be available even when the Machine is stopped. E.g. if you try:

curl -H "Authorization: Bearer $(fly auth token)" \
    "https://api.machines.dev/v1/apps/<app-name>/volumes/vol_<id>" | jq

then you should receive something like:

{
  "id": "vol_<id>",
  "name": "<name>",
  "blocks": 103211758,
  "block_size": 4096,
  "blocks_free": 24842847,
  "blocks_avail": 20642351,
  ...
}

(The full Machines API docs are here.)

Hope this helps!

[amo]

Awesome, thank you so much for the link to the full API !
That is exactly what I was looking for.