Early access: build Grafana dashboard from Fly metrics

kurt · September 9, 2020, 1:34am

Check out our shiny new metrics engine: Feature preview: Custom metrics

We have a chunky Prometheus (well, VictoriaMetrics) cluster with all kinds of useful application metrics. It’s what powers the graphs on our web UI, but there’s a bunch more in there.

You can use these with Grafana to make neat dashboards, alert on metrics, etc.

We’re working up a prebuild Grafana dashboard with some interesting graphs, but if you want to try it out before instructions examples, you should!

API Details

The Prometheus API is available at https://api.fly.io/prometheus/api/v1/
Send an Authorization: Bearer <TOKEN> to authenticate (you can run flyctl auth token to get your token)
Run some queries

Grafana Cloud Setup

If you don’t have Grafana yet, the free Grafana Cloud plan will work fine for this: Grafana Cloud | Observability platform overview

Once you’re in your spiff new Grafana instance, add a Prometheus source like so (note the URL and “Custom HTTP Headers” section:

From there, you can create a Dashboard, then add a Panel. Try a query like this:

sum(rate(edge_http_responses_count{app="<APP NAME>"}[$__interval])) by (status)

Series and Labels

We’ve exposed a bunch of different series, you will need to supply an app="<NAME>" argument to each. Here’s a quick list:

// responses from load balancer
edge_http_responses_count
edge_http_response_time_seconds_bucket

// connections to anycast IPs
edge_tcp_connects_count
edge_tcp_disconnects_count

// responses from app vms using http handler
app_http_responses_count
app_http_response_time_seconds_bucket
app_local_connect_time_seconds_bucket

// tcp connections to app vms
app_local_connects_count
app_local_disconnects_count

// data out through load balancer
anycast_data_out
anycast_data_in

// vm memory metrics
firecracker_vm_memory_buffers
firecracker_vm_memory_cached
firecracker_vm_memory_mem_free
firecracker_vm_memory_mem_available
firecracker_vm_memory_mem_total
firecracker_vm_memory_swap_cached
firecracker_vm_memory_vmalloc_used
firecracker_vm_memory_active
firecracker_vm_memory_inactive

// other vm metrics
firecracker_vm_cpu
firecracker_vm_load_average
firecracker_vm_net_sent_bytes
firecracker_vm_disk_time_io

// network interface metrics
node_network_transmit_bytes
node_network_receive_bytes

Give it a try, let us know what you think.

Known missing pieces

We’re missing a few features to really make Grafana nice:

Support for autocomplete in queries. This will be fixed before we officially™️ ship this.
Querying metrics for multiple apps: right now, queries need to include an app name, there’s no way to get metrics that combine apps.

kurt · September 9, 2020, 5:19pm

Grafana has a neat map visualization:

Just add this: Worldmap Panel plugin for Grafana | Grafana Labs

Use a query like this:

sum(rate(edge_http_responses_count{app="<NAME>"}[$__interval])) by (region)

Then set the map data JSON endpoint to https://api.fly.io/meta/regions.json:

danwetherald · September 9, 2020, 5:22pm

This is amazing!

kurt · September 10, 2020, 10:04pm

Heatmaps are pretty great for showing response times:

Settings

Query:

 sum(increase(edge_http_response_time_ns_bucket{app="$app"}[$__interval])) by (le)

Legend: {{le}}
Min step: 1m
Format: Heatmap
Visualization: Heat map

Axes: Y-Axis
- Unit: seconds
- Data format: Time series buckets
Display
- Colors: spectrum
- Scheme: Plasma
- Color scale min: 0
- Show legend: ✓

Grafana JSON

{
  "type": "heatmap",
  "title": "Response Times",
  "gridPos": {
    "x": 9,
    "y": 14,
    "w": 9,
    "h": 9
  },
  "id": 23763571993,
  "targets": [
    {
      "expr": "sum(increase(edge_http_response_time_ns_bucket{app=\"$app\"}[$__interval])) by (le)",
      "legendFormat": "{{le}}",
      "interval": "1m",
      "refId": "A",
      "format": "heatmap"
    }
  ],
  "fieldConfig": {
    "defaults": {
      "custom": {}
    },
    "overrides": []
  },
  "pluginVersion": "7.1.5",
  "legend": {
    "show": true
  },
  "tooltip": {
    "show": true,
    "showHistogram": false
  },
  "heatmap": {},
  "cards": {
    "cardPadding": null,
    "cardRound": null
  },
  "color": {
    "mode": "spectrum",
    "cardColor": "#b4ff00",
    "colorScale": "sqrt",
    "exponent": 0.5,
    "colorScheme": "interpolatePlasma",
    "min": 0
  },
  "dataFormat": "tsbuckets",
  "yBucketBound": "middle",
  "xAxis": {
    "show": true
  },
  "yAxis": {
    "show": true,
    "format": "s",
    "decimals": 0,
    "logBase": 1,
    "splitFactor": null,
    "min": "0",
    "max": null
  },
  "highlightCards": true,
  "timeFrom": null,
  "timeShift": null,
  "reverseYBuckets": false,
  "xBucketSize": null,
  "xBucketNumber": null,
  "yBucketSize": null,
  "yBucketNumber": null,
  "hideZeroBuckets": false,
  "datasource": null
}

jveres · September 23, 2020, 8:39am

Hi! I’d be eager to try this out but I can’t get it work.
https://api.fly.io/prometheus/api/v1 responds with 404.
Is this still available for us to play with?

Codepope · September 23, 2020, 2:45pm

It should be, but the endpoint should be https://api.fly.io/prometheus/ for the Grafana data sources (as per the screenshot).

Dj

jveres · September 23, 2020, 3:22pm

That doesn’t work either:(

kurt · September 23, 2020, 3:25pm

Ah! We’ve only implemented these two URLs so far:

Grafana “knows” about these, so when you set it up all you have to tell it is https://api.fly.io/prometheus/. We use the Ruby Prometheus client, which also just needs the base URL. But for something like cURL you’ll need a more specific endpoint.

jveres · September 23, 2020, 3:35pm

Now it works, thanks!

kurt · September 23, 2020, 6:34pm

Good deal! Definitely let us know if you run into any problems, this could be quite powerful and we want to make sure it’s nice and solid before we launch it.

jveres · September 24, 2020, 3:29pm

Works fine so far, impressive work! I can’t get the worldmap working though.

kurt · September 24, 2020, 4:05pm

The worldmap is a little flakey it seems like. Here’s the panel JSON for the one I have working:

{
  "circleMaxSize": "10",
  "circleMinSize": 2,
  "colors": [
    "rgba(245, 54, 54, 0.9)",
    "rgba(237, 129, 40, 0.89)",
    "rgba(50, 172, 45, 0.97)"
  ],
  "decimals": 0,
  "esMetric": "Count",
  "fieldConfig": {
    "defaults": {
      "custom": {
        "align": null
      },
      "mappings": [],
      "thresholds": {
        "mode": "absolute",
        "steps": [
          {
            "color": "green",
            "value": null
          },
          {
            "color": "red",
            "value": 80
          }
        ]
      }
    },
    "overrides": []
  },
  "gridPos": {
    "h": 14,
    "w": 18,
    "x": 0,
    "y": 0
  },
  "hideEmpty": false,
  "hideZero": false,
  "id": 4,
  "initialZoom": 1,
  "jsonUrl": "https://api.fly.io/meta/regions.json",
  "locationData": "json endpoint",
  "mapCenter": "(0°, 0°)",
  "mapCenterLatitude": 0,
  "mapCenterLongitude": 0,
  "maxDataPoints": 1,
  "mouseWheelZoom": false,
  "pluginVersion": "7.1.5",
  "showLegend": true,
  "stickyLabels": false,
  "tableQueryOptions": {
    "geohashField": "geohash",
    "latitudeField": "latitude",
    "longitudeField": "longitude",
    "metricField": "metric",
    "queryType": "geohash"
  },
  "targets": [
    {
      "expr": "sum(rate(edge_http_responses_count{app=\"$app\"}[$__interval])) by (region)",
      "interval": "",
      "legendFormat": "{{status}}",
      "refId": "A"
    }
  ],
  "thresholds": "0,10",
  "timeFrom": null,
  "timeShift": null,
  "title": "Requests by Region",
  "type": "grafana-worldmap-panel",
  "unitPlural": "",
  "unitSingle": "",
  "valueName": "total",
  "datasource": null
}

jveres · September 24, 2020, 6:19pm

Thanks, it helped. "legendFormat": "{{status}}" was missing for me.

kurt · September 24, 2020, 7:28pm

Oh nice! If you end up doing other interesting stuff in Grafana, will you post about it here?

jveres · September 24, 2020, 8:48pm

Sure. Everything looks pretty solid – maybe the negative memory consumption once today was a bit strange.

kurt · September 24, 2020, 9:42pm

Oh that’s odd! We’re not that good at VMs yet.

What query gave you negatives on memory? I can have a look and see why it might’ve done that.

jveres · September 25, 2020, 7:44am

avg(firecracker_vm_memory_mem_total{app="$__app"}) - avg(firecracker_vm_memory_mem_free{app="$__app"})

jveres · September 25, 2020, 9:12am

Btw, would it be possible to collect app specific metrics too? Maybe as a backing service similarly to Redis?

kurt · September 25, 2020, 3:49pm

Yes we’d like to let people collect custom metrics. It’s a fun technical challenge because Prometheus like databases have problems with too many different series (metric + label). High cardinality breaks things and/or makes them expensive to run.

We’ve experimented with this a little, if you send a fly-cache-status: HIT header you’ll see a chart appear on our ui. You can use any status you want there. We might expose other named metrics and let apps populate them.

The next step is to make it easy to scrape metrics from your app instances so you can turn on something like paid Grafana cloud and at least get them there!

jveres · September 25, 2020, 6:25pm

Great, keep us informed!
Not sure how do you mean sending a fly-cache-status. Send it from an instance in an HTTP response?

Topic		Replies	Views
Preview: Managed Grafana Dashboards for Fly Apps metrics , announcement	24	3244	October 21, 2023
Grafana Dashboard for Fly Apps metrics	1	1182	July 19, 2021
can i modify the Grafana dashboard to view HTTP response times to a lower granularity Questions / Help metrics , grafana	4	469	May 4, 2024
Custom app metrics not making it to grafana cloud Questions / Help metrics , elixir , grafana	18	1849	March 24, 2022
In Grafana - how do I separate metrics by route grafana	1	21	November 22, 2024