Prometheus scrape endpoint to move metrics into Honeycomb

1

We’re trying to setup Open Telemetry collector to move Fly metrics into Honeycomb.

Links:

But we are hitting a wall which seems to be that the Fly prometheus service does not have the scrape API available.

Is this true/correct? Is there a way to enable it?

2

Which endpoint are you trying to scrape? The /federate endpoint should be available to scrape arbitrary metrics from Fly Prometheus to another service.

3

We are using opentelemetry-collector with this receiver config:

receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: otel-collector
          scrape_interval: 10s
          metrics_path: /prometheus/prisma-preview
          honor_labels: true
          params:
            'match[]':
              - '{job="prometheus"}'
              - '{__name__=~"job:.*"}'
          scheme: https
          static_configs:
            - targets: ['api.fly.io']
          #          authorization:
          #            credentials: 'BEARER_TOKEN'
          bearer_token: 'BEARER_TOKEN'

We tried changing the metrics_path to /prometheus/prisma-preview/federate too but we still get the same error.
warn internal/otlp_metricsbuilder.go:164 Failed to scrape Prometheus endpoint {"kind": "receiver", "name": "prometheus", "pipeline": "metrics", "scrape_timestamp": 1671722291735, "target_labels": "map[__name__:up instance:api.fly.io:9091 job:otel-collector]"}

Is there a better way to scrape the metrics from Fly.io prometheus and export them to Honeycomb?

4

That error message is showing port 9091, perhaps there’s a default port setting somewhere getting in the way- could you maybe try setting port 443?

/federate should definitely work, others have been using that endpoint to scrape metrics to different systems.

1 Like
5

Oh, sorry… I was playing with different ports. None of them worked, 443, 80, 9090, 9091 :slight_smile:
Maybe I am missing something else.

6

Instead of us exporting the metrics to honeycomb, is there a way to setup metrics alerting in Fly.io?
I don’t see this enabled in Fly.io Grafana.

7

it seems that federate endpoint isn’t designed for scraping all metrics but instead something aggregated? At least some stack over flow threads suggest this. That would be a problem for us since we want the raw data in honeycomb.

8

Thanks for all the info so far. Next thing I would suggest trying is a manual curl request to make sure the endpoint itself is working, then if so you can focus on your library’s scrape config and try to figure out why it doesn’t match. See this thread for details on crafting a curl response that works with the /federate endpoint.

The federate endpoint should be able to scrape all metrics, you just need to pass it a selector that matches everything. (In fact, perhaps the selector should be '{__name__=~".*"}' instead of '{__name__=~"job:.*"}' as in your example?)

1 Like
9

Changing the selector to '{__name__=~".*"}' worked for on of our organizations.
I am using the same config and targeting our other organization and I get the same error as before:
2023-01-02T13:26:32.169Z warn internal/otlp_metricsbuilder.go:164 Failed to scrape Prometheus endpoint {"kind": "receiver", "name": "prometheus", "pipeline": "metrics", "scrape_timestamp": 1672665991366, "target_labels": "map[__name__:up instance:api.fly.io:443 job:otel-collector]"}

Not sure what could be different. I will do some more research :thinking:

1 Like
10

It seems to be working now for both organizations.
@wjordan do you know maybe why some metrics have empty columns? Is there a way to filter just the ones we need, when exporting to honeycomb?

image
image2560×1484 417 KB