Custom prometheus metrics not working... Fly.io not querying /metrics endpoint

1

Sorry, maybe I am missing something obvious.

An app is natively listening port :4000 and exposes, among others, /metrics endpoint.

Not sure if this is relevant but I am running the app with container’s entrypoint set to supervisord that concurrently starts the App itself, and also Caddy. Caddy is reverse proxying port 8080 to 4000, only certain selected public endpoints - not everything.

My fly.yaml is

[[services]]
  protocol = 'tcp'
  internal_port = 8080
  auto_stop_machines = 'off'
  auto_start_machines = false
  min_machines_running = 2

  [[services.ports]]
    port = 443
    handlers = ['tls', 'http']

  [[services.ports]]
    port = 80
    handlers = ['http']
    force_https = true

  [[services.http_checks]]
    interval = '10s'
    timeout = '1s'
    grace_period = '60s'
    method = 'get'
    path = '/health/liveness'
    protocol = 'http'
    tls_skip_verify = false

[[vm]]
  cpu_kind = 'performance'
  cpus = 1
  memory_mb = 2048

[[metrics]]
  port = 4000
  path = '/metrics'
  https = false

Everything works as designed for app access. I can access allowed public endpoints through app’s domain and HTTPs (via Caddy); Liveness checks work. I can also access all other endpoints internally via Flycast network (and Wireguard) using .internal domain at the app’s original :4000 port.

I can also manually open http://app.internal:4000/metrics and see the prometheus metrics details, just not via :8080 and Caddy. When I do so, I see “INFO: fdaa:c:d4fe:a7b:ce2:0:a:102:49822 - “GET /metrics/ HTTP/1.1” 200 OK” in the Live Logs.

However, fly.io is not picking up these metrics. There are no automated requests to /metrics/ endpoint (unless I request it manually), and I don’t see any custom metrics in Grafana dashboard.

What am I missing?

Thanks!

3

I wonder if you need to provide processes when using an array of tables ([[metrics]]):

Via App configuration (fly.toml) · Fly Docs

## regular table
[metrics]
  port = 4000
  path = '/metrics'

## array of tables
[[metrics]]
  port = 4000
  path = '/metrics'
  processes = ['app']  ## default process is "app" unless specified
4

Thanks for responding! Nope that has not changed anything…

5

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.