Hosting reverse proxy for custom SaaS domains on Fly

1

Hi, I’m trying to spin up a reverse proxy to host custom (customer) domains on Fly.io for our SAAS.

The proxy is Caddy and works fine in isolation, however it seems that Fly.io forces https before the traffic reaches the proxy (which provides HTTPS provisioning for the customer domains) and this causes problems.

I understand the workaround is to add the domain/certificate to the service in Fly? But that means all of our customers custom domains will need to be added to the Fly service, correct? I really don’t want to be doing that, the whole point of this service was that it would provision SSL on-demand.

2

Don’t use TLS / HTTP handlers, but raw TCP instead (docs | code), and the app should now be able to serve raw TCP and do TLS termination as need be. If you’re doing QUIC (UDP), it is much more trickier on Fly (see).

You can most definitely vend certs elsewhere and plonk them in to your Fly app via fly secrets (that’s what we do).

You shouldn’t have to move the registration of your domain over to Fly, I don’t think so. Per docs, it is enough to set up either CNAME or AAAA / A records to have Fly vend Let’s Encrypt TLS certs and manage them for you at their (edge) proxy.

This is a better option since you don’t have to worry about renewing certs on time, revoking certs in case of compromise, or handling its private keys. Note that, vending beyond 10 Fly-managed TLS certs isn’t free (pricing).

That said, you could also explore using a L3 (IP) load balancer (typically expensive, like Cloudflare’s MagicTransit) than the L4/L7 (TCP/UDP/HTTP) one that Fly has.

3

Thanks, for now I can disable HTTP3/UDP to keep things easier.

Removing the handlers from the TOML seems like the key, so this was really useful. Although after doing that I’m just getting the following on repeat.

2022-10-10T02:48:05.685 proxy[9db4d44b] iad [error] Error: while TCP-proxying data to/from app: failed to copy (direction=client->server, error=Transport endpoint is not connected (os error 107))

2022-10-10T02:48:39.563 proxy[9db4d44b] iad [error] Error: while TCP-proxying data to/from app: failed to copy (direction=client->server, error=Transport endpoint is not connected (os error 107))

TOML file currently looks like:

app = "caddy-proxy"
kill_signal = "SIGINT"
kill_timeout = 5
processes = []

[env]

[experimental]
  allowed_public_ports = []
  auto_rollback = true

[[services]]
  http_checks = []
  script_checks = []
  internal_port = 80
  protocol = "tcp"
  [services.concurrency]
    hard_limit = 25
    soft_limit = 20
    type = "connections"

  [[services.ports]]
    handlers = ["http"]
    port = 80
    force_https = true

  [[services.ports]]
    port = 443

  [[services.tcp_checks]]
    grace_period = "1s"
    interval = "15s"
    restart_limit = 0
    timeout = "2s"
4 
...
    type = "connections"

  [[services.ports]]
-   handlers = ["http"]
    port = 80
-   force_https = true

  [[services.ports]]
    port = 443
...
  1. (optionally) Remove those two lines to check if those are what’s causing trouble… they shouldn’t.
  2. Make sure your app is listening on (both ipv4/ipv6) the same port as declared in the app’s services.internal_port (which is 80 per the config you just shared).
5

Yea, no difference regarding the http redirect, i initially removed all the handlers and then added the redirect back.

The app itself is just the caddy server at this time which binds to all interfaces by default and listening on 80/443. It was “working” / connecting before with Fly.io terminating the TLS.

6

Ah, I had to change internal_port=443 now and that seems to have addressed it, this will work with Fly doing the forwarding to https

1 Like
7

FWIW I am still seeing these errors semi-regularly:

2022-10-12T22:14:34Z proxy[02331348] ams [error]Error: while TCP-proxying data to/from app: failed to copy (direction=client->server, error=Transport endpoint is not connected (os error 107))
2022-10-12T22:14:34Z proxy[02331348] ams [error]Error: while TCP-proxying data to/from app: failed to copy (direction=client->server, error=Transport endpoint is not connected (os error 107))
2022-10-12T22:14:39Z proxy[02331348] ams [error]Error: while TCP-proxying data to/from app: failed to copy (direction=client->server, error=Transport endpoint is not connected (os error 107))
1 Like