Getting Caddy Metrics

Hey folks :wave:

I’ve got a Caddy server deployed on Fly and I’m trying to get metrics out of it but am having issues figuring out the correct config.

I have a simple Caddyfile that looks like this:

http://localhost, http://my.custom.domain {
        bind 0.0.0.0
        root * /usr/share/caddy
        try_files {path} /{path}.html
        file_server
        metrics /metrics
}

And I’m able to query the metrics with curl https://my.custom.domain/metrics ( I also tried with a separate handler on :9090 but that didn’t work).

In my fly.toml I have the following config:

[metrics]
  port = 80
  path = "/metrics"

I have set up the data source in Grafana, and I can see the default Fly metrics, so that integration works, but I don’t see the metrics exported by caddy anywhere in the prometheus instance.

Anyone has an idea what I might be doing wrong? :thinking:

Beep boop, bumping this up!

I also connected via ssh to see what’s going on, looks like Caddy is binding on :::80 and :::9091 instead of 0.0.0.0 ? Not sure if that could be the issue…

/ # netstat -ltnup
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 127.0.0.1:2019          0.0.0.0:*               LISTEN      515/caddy
tcp        0      0 fdaa:0:5c86:a7b:88dc:80ab:7419:2:22 :::*                    LISTEN      516/hallpass
tcp        0      0 :::9091                 :::*                    LISTEN      515/caddy
tcp        0      0 :::80                   :::*                    LISTEN      515/caddy

That should be ok. Binding to :: binds to IPv4 and IPv6.

Let me take a quick look at it.

I manually curled your metrics endpoint and just got a 200 OK response with no body.

There are no metrics being output from Caddy.

yeah that’s what I think too, I also curled locally on 0.0.0.0 and it works so that’s a good sign

I tried a few things since I posted this, latest try I have metrics served on :9091 so it’s probably not accessible externally, but the metrics are there, that much I’m sure.

$ curl -iv http://172.19.6.26:9091/metrics
*   Trying 172.19.6.26...
* TCP_NODELAY set
* Connected to 172.19.6.26 (172.19.6.26) port 9091 (#0)
> GET /metrics HTTP/1.1
> Host: 172.19.6.26:9091
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Server: Caddy
Server: Caddy
< Date: Thu, 19 May 2022 13:48:18 GMT
Date: Thu, 19 May 2022 13:48:18 GMT
< Content-Length: 0
Content-Length: 0

<
* Connection #0 to host 172.19.6.26 left intact

That IP is your VM’s IP on the host.

If you SSH in and run curl, you should get the same blank response.

mhm, interesting, so I know that Caddy is kinda picky there, if the host isn’t the same as defined in the Caddyfile, it won’t serve the content (I’m not actually sure what’s the behavior there, still pretty new to Caddy). Since my config is set to serve on localhost:9091 I’m guessing Caddy doesn’t recognize that IP and that’s why it’s not serving the content?

I think I’ll need to ask on the Caddy discourse then, not sure how I can have it pick up the host IP :thinking:

Possibly this:

Yeah, seems to work! Awesome, thank you so much for your time eh!