Getting Caddy Metrics

Questions / Help
1

Hey folks :wave:

I’ve got a Caddy server deployed on Fly and I’m trying to get metrics out of it but am having issues figuring out the correct config.

I have a simple Caddyfile that looks like this:

http://localhost, http://my.custom.domain {
        bind 0.0.0.0
        root * /usr/share/caddy
        try_files {path} /{path}.html
        file_server
        metrics /metrics
}

And I’m able to query the metrics with curl https://my.custom.domain/metrics ( I also tried with a separate handler on :9090 but that didn’t work).

In my fly.toml I have the following config:

[metrics]
  port = 80
  path = "/metrics"

I have set up the data source in Grafana, and I can see the default Fly metrics, so that integration works, but I don’t see the metrics exported by caddy anywhere in the prometheus instance.

Anyone has an idea what I might be doing wrong? :thinking:

2

Beep boop, bumping this up!

3

I also connected via ssh to see what’s going on, looks like Caddy is binding on :::80 and :::9091 instead of 0.0.0.0 ? Not sure if that could be the issue…

/ # netstat -ltnup
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 127.0.0.1:2019          0.0.0.0:*               LISTEN      515/caddy
tcp        0      0 fdaa:0:5c86:a7b:88dc:80ab:7419:2:22 :::*                    LISTEN      516/hallpass
tcp        0      0 :::9091                 :::*                    LISTEN      515/caddy
tcp        0      0 :::80                   :::*                    LISTEN      515/caddy
4

That should be ok. Binding to :: binds to IPv4 and IPv6.

Let me take a quick look at it.

5

I manually curled your metrics endpoint and just got a 200 OK response with no body.

There are no metrics being output from Caddy.

6

yeah that’s what I think too, I also curled locally on 0.0.0.0 and it works so that’s a good sign

I tried a few things since I posted this, latest try I have metrics served on :9091 so it’s probably not accessible externally, but the metrics are there, that much I’m sure.

7 
$ curl -iv http://172.19.6.26:9091/metrics
*   Trying 172.19.6.26...
* TCP_NODELAY set
* Connected to 172.19.6.26 (172.19.6.26) port 9091 (#0)
> GET /metrics HTTP/1.1
> Host: 172.19.6.26:9091
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Server: Caddy
Server: Caddy
< Date: Thu, 19 May 2022 13:48:18 GMT
Date: Thu, 19 May 2022 13:48:18 GMT
< Content-Length: 0
Content-Length: 0

<
* Connection #0 to host 172.19.6.26 left intact

That IP is your VM’s IP on the host.

If you SSH in and run curl, you should get the same blank response.

8

mhm, interesting, so I know that Caddy is kinda picky there, if the host isn’t the same as defined in the Caddyfile, it won’t serve the content (I’m not actually sure what’s the behavior there, still pretty new to Caddy). Since my config is set to serve on localhost:9091 I’m guessing Caddy doesn’t recognize that IP and that’s why it’s not serving the content?

I think I’ll need to ask on the Caddy discourse then, not sure how I can have it pick up the host IP :thinking:

9

Possibly this:

10

Yeah, seems to work! Awesome, thank you so much for your time eh!

11

Hey @wperron , I’m also setting this up but having trouble – I’m able to ssh into the caddy instance and access /metrics however nothing seems to have changed on Fly.io / Grafana’s side. Perhaps it cannot access it, but unsure how to debug that.

With this connected were you able to see HTTP status codes reflected in the Metrics page in Fly?

12

Hey @tomm , You won’t see your own metrics in the Fly console, you’ll need to add it as a datasource to Grafana (or whatever dashboarding tool you like) and then you’ll be able to query your metrics.

13

Thanks for jumping in on this old issue, this was the clue I needed. For anyone that ends up here in the future – Head over to the fly-hosted Grafana and click on Explore in the left menu

image
image1106×1100 95.7 KB

From there you should be able to see that your caddy_ metrics are being ingested and can clone and modify the existing dashboards to include them or create a new one.