Issues deploying a caddy server to fly

1

Hi all,

I am currently having an issue with what I think is the fly proxy.
I am trying to run a Caddy server in order to provide automatic ssl for my client domain names. For the context this is my Caddyfile sigle-proxy/Caddyfile at main · pradel/sigle-proxy · GitHub. What I want is to have any domain being able to add a CNAME pointing to alias.sigle.io (during my tests I am using test.leopradel.com) and then the Caddy server will generate a new ssl certificate if needed and reverse proxy the request to another application running on app.sigle.io.

I tried to deploy the exact same Caddyfile on a fresh DO droplet and everything is working fine, my ssl certificates are properly generated and the page is loaded.
But when I deploy the same configuration on Fly when I try to access test.leopradel.com it looks like my caddy server is not even reached and the request is stopped with “ERR_CONNECTION_CLOSED”. My guess is that my request is actually never calling my application but stopped somewhere before by your proxy?

DNS on Fly:

DNS working during my test on DO:

I am not sure how to debug this further or if it’s a limitation of the platform. Any help welcome :slight_smile:

1 Like
2

I didn’t find in the documentation a way to bypass the fly proxy so not sure if it’s possible.

3

If you’re using the tls handler, then we will close connections for which we don’t have a certificate because we can’t handshake. The hostname test.leopradel.com is not associated with your app, so we can’t generate a TLS certificate.

It looks like what you’re trying to do is let your server handle TLS handshakes. You’ll need to remove both tls and http handlers from your service ports. That will make our proxy pass-through TCP directly to your app.

Your fly.toml should look like:

[[services]]
  internal_port = 8080 # port Caddy exposes
  protocol = "tcp"
[[services.ports]]
    port = "80"
[[services.ports]]
    port = "443"

(or something like that)

1 Like
4

It’s worth noting that if you require the “real” remote address from clients, you will need to accept the haproxy protocol and use the proxy_proto handler for your ports.

5

Yeah indeed my config was like this:

  [[services.ports]]
    handlers = ["http"]
    port = "80"

  [[services.ports]]
    handlers = ["tls", "http"]
    port = "443"

I will try with you suggestion, thanks for the super quick response :slight_smile:

6

I did a lot of tries with the config you suggested (so removing the handlers) and I feel like my server is actually never called. I changed my caddy file to log any incoming request, but the logs are not showing anything when I make the request (just from the health check).

Fly config:

app = "snowy-wildflower-7335"

kill_signal = "SIGINT"
kill_timeout = 5

[[services]]
  internal_port = 80
  protocol = "tcp"

  [[services.ports]]
    port = "80"

  [[services.ports]]
    port = "443"

  [[services.http_checks]]
    interval = 60000
    method = "get"
    path = "/health"
    protocol = "http"
    timeout = 2000
    tls_skip_verify = true

Caddyfile

{
    debug

    on_demand_tls {
        ask https://app.sigle.io/api/caddy-endpoint
    }

    # TODO find a way for persistent storage
    # storage dynamodb caddy_ssl_certificates
}

:80 {
    log {
       level DEBUG
       output stderr
    }

    respond /health "Im healthy!" 200
}

:443 {
    log {
       level DEBUG
       output stderr
    }

    tls sigle@protonmail.com {
        on_demand
    }

    reverse_proxy https://app.sigle.io {
        header_up Host app.sigle.io
        header_up User-Custom-Domain {host}
        header_up X-Forwarded-Port {server_port}

        health_timeout 5s
    }
}

I really tried a lot of various config but without any luck :frowning:

Note: I have setup both A and AAAA records.

7

I believe it’s being called because:

$ curl http://test.leopradel.com/health
Im healthy!

I just noticed that you probably need 2 services. Both “external” ports 80 and 443 will hit your internal port 80. Looking at your config, you probably want 2 services, mapped respectively 80:80 and 443:443.

Something like:

app = "snowy-wildflower-7335"

kill_signal = "SIGINT"
kill_timeout = 5

[[services]]
  internal_port = 80
  protocol = "tcp"

  [[services.ports]]
    port = "80"

  [[services.http_checks]]
    interval = 60000
    method = "get"
    path = "/health"
    protocol = "http"
    timeout = 2000
    tls_skip_verify = true

[[services]]
  internal_port = 443
  protocol = "tcp"

  [[services.ports]]
    port = "443"
1 Like
8

Okay I didn’t try with curl :man_facepalming:. And with curl I can see the request in the console

When I open the connection in the browser I get this “ERR_CONNECTION_CLOSED” error message. I will try with 2 services to see if it’s helping.

9

Ok so deploying with 2 services helped! Now I am getting rate limited by letsencrypt so I guess it’s working as it should, thanks for the help :smiley:

10

Sure thing! Happy to help :slight_smile: let us know if you need any more help.

11

Oh btw, we can handle certificates for you if you add every hostname you need one for via flyctl or our graphql API (currently undocumented, but I can help you get setup).

Unless there’s a specific reason you want to generate them yourself. We’re delighted that it works for you!

12

Looks like we have those mutations and queries documented! Custom Domains and SSL Certificates · Fly Docs

1 Like
13

I decided to go with this approach because usually, platforms allowing you to add custom domains have a hard limit at max 50 domain names per app (I think it’s an AWS limitation). So I decided to create my own reverse proxy as I would need more than 50.

14

I am definitely not opposed using your API, but would it be possible for my users to point a CNAME record to alias.sigle.io? From my understanding, they would have to point the record to “something.fly.io” right?

15

We’ll flatten CNAMEs another level, so it should work if you have customers point to your name that goes back to us. We have a few other customers doing this but I don’t remember the specifics, @Jerome anything special we need to do to make that work?

16

@pradel.leo you also might want to cache those certs globally, otherwise any instance of your app will ask LE to issue all certs.

1 Like
17

Yes definitely, I will add some storage, I saw that caddy is supporting s3, dynamodb and some other storages.

18

Just to add one more good reason to use us for TLS handshakes:

If you’re handshaking TLS from your app, you won’t benefit from the faster handshakes we’re able to do from our edge locations. We’ll have to proxy packets back and forth between your app and our edge to handshake TLS.

1 Like
19

That’s indeed an interesting part and less maintenance for my app which is always great :wink:
Just two small questions then, can you confirm that you don’t have a limit in the number of custom domains I can add? (I saw that it’s 0.10$ per custom domain per month, which is totally fine)

And would this be possible?

20

There is no limit on the number of hostnames / certificates.

Yes, you can point alias.sigle.io A and AAAA records to your IPv4 and IPv6 for your app and tell your users to CNAME their hostnames to alias.sigle.io. We check for IPv6! CNAMEing alias.sigle.io to yourapp.fly.dev would also work if your DNS provider does proper flattening including IPv6.