Does Fly's Prometheus API endpoint support federation? `remote_read`?

How To
#1

I have Prometheus and Alertmanger (w/ Pusher) configured to monitor various platforms on which I deploy services and alert me if I leave resources deployed. I don’t use Grafana because I don’t need it for this use.

I’d like to include fly.io using the Prometheus endpoint but I’ve been unable to determine whether the API accepts federate|remote_read` requests.

curl \
--silent \
--header "Authorization: Bearer ${TOKEN}"  \
--write-out '%{response_code}' \
http://api.fly.io/prometheus/{$ORG}/api/v1/federate/match[]={__name__=~".+"}"

And permutations of:

curl \
--silent \
--header "Authorization: Bearer ${TOKEN}"  \
--write-out '%{response_code}' \
http://api.fly.io/prometheus/${ORG}/api/v1/read

Return 400 so I’m assuming not.

remote_read:
  - url: "https://api.fly.io/prometheus/{org}/api/v1/read"
    authorization:
      credentials: "{token}"

If I can configure the Prometheus server to scrape e.g. fly_instance_up, I could configure a rule to alert for it being non-zero.

Or am I missing an obvious alternative?

Thanks!

#2

If it’s not possible to remote_read metrics such as fly_instance_up from api.fly.io, should I write a Fly (org) metrics exporter that leverages Fly’s Go API?

I’m looking to enumerate the list of deployed apps so that I can alert on the number.

Thanks!

1 Like
#3

I wrote a very basic Prometheus Exporter to count apps.

Export a TOKEN

# HELP build_info A metric with a constant '1' value labeled by OS version, Go version, and the Git commit of the exporter
# TYPE build_info counter
build_info{git_commit="b6d8c2fd562e5f482e709e229af0910ce8502969",go_version="go1.18.2",os_version="5.15.32-v8+"} 1
# HELP fly_exporter_apps Total Number of Apps
# TYPE fly_exporter_apps counter
fly_exporter_apps{deployed="true",id="foo",name="foo",org_slug="personal",status="running"} 1
fly_exporter_apps{deployed="true",id="bar",name="bar",org_slug="personal",status="running"} 1
# HELP start_time Exporter start time in Unix epoch seconds
# TYPE start_time gauge
start_time 1.652975685e+09
Prometheus Federation - worked for 2-3 days, now it's failing
#4

Hi @DazWilkin,

Fly’s Prometheus API runs on VictoriaMetrics Cluster and supports all of the Prometheus querying API endpoints documented there.
VictoriaMetrics notably does not support the remote_read API, but federate should be supported.

Your curl federate example has a syntax error, the match[] should be passed as a URL parameter (not part of the path), and you need to either URL-encode all URL-unsafe chars or pass the param as url-encoded POST data, either:

  • /federate?match[]=%7B__name__=~%22.%2B%22%7D
  • /federate --data-urlencode 'match[]={__name__=~".+"}'
#5

Aha!

Thank you. Yes, that works.

curl \
--silent \
--header "Authorization: Bearer ${TOKEN}"  \
--write-out '%{response_code}' \
http://api.fly.io/prometheus/{$ORG}/federate \
--data-urlencode 'match[]={__name__=~".+"}' \
| awk '/^fly_instance_up/ {print}'

Yields:

fly_instance_up{instance="{instance}",host="{host}",app="my-app-01",region="sea"} 1 1658427728952
fly_instance_up{instance="{instance}",host="{host}",app="my-app-02",region="sea"} 1 1658427728946
#6

Great! Also note that if you want to just match a single series you can just do that directly:

curl -H "Authorization: Bearer $TOKEN" \
  api.fly.io/prometheus/$ORG/federate?match[]=fly_instance_up