We’ve just “soft” launched our new custom metrics features 
Here are some docs if you want to get all technical right away.
In essence: we’re starting to collect and store custom prometheus metrics for apps hosted on Fly.
Inserting metrics
- Instrument your app with a prometheus library
- Expose metrics (make sure it binds to
0.0.0.0) - Configure your
fly.toml:
[metrics]
port = 9091
path = "/metrics"
We’ll pull from your instances a few times per minute.
Querying metrics
With this new feature, we’re slowly deprecating our current prometheus API and introducing a new one with a few differences:
- Base URL:
https://api.fly.io/prometheus/{org_slug}- Find your
{org_list}by listing orgs withflyctl orgs list
- Find your
- Headers:
Authorization: Bearer {token}- Find
{token}withflyctl auth token
- Find
- All apps for the same organization are accessible from the same endpoint
- Custom app metrics and Fly metrics (proxy, instance, volumes, prefixed with
fly_) are all available through the same endpoint. - Your own metrics are not namespaced, but we will automatically add the
app,instance,hostandregionlabels. - Full list of Fly metrics available in the docs
pg_series are not prefixed withfly_because they work just like custom metrics (itsfly.tomlis configured for it
This new API works just like the prometheus API
curl "https://api.fly.io/prometheus/{org_slug}/api/v1/query_range?step=30" \
--data-urlencode 'sum(rate(fly_edge_http_responses_count{app="{app}"}[5m])) by (status)' \
-H "Authorization: Bearer {token}"
A grafana setup looks like this:
“Normal” grafana features for discovering labels and series should work as-is. The old API did not support that.
Pricing
We’re still working on this. Keeping your series count under 10,000 should awlays be free.
You’ve probably noted that we’re including our own fly-specific metrics in there too. We’re hoping they don’t account for too many series! However, with the number of regions we have and if you have a lot of instances in different regions, that could grow fast. We’ll decide on pricing and limits based on what we observe here.
During this “beta” phase, we won’t be enforcing any specific limits, but we will be on the lookout for excessive usage.
, this came in faster than I expected. I will deploy a couple my apps tommorow to test, just need to modify them a little. The prometheus endpoint is currently exposed on the same port as the main app api.
Got my Grafana dashboard up and humming along nicely.