How to get the "Fly-Client-IP" with TCP Pass Through

Questions / Help
1

Hi

Context: I am deploying a reverse proxy (Caddy) which handles TLS termination and doing TCP pass-through. My fly.toml looks like this:

[[services]]
  internal_port = 443
  protocol = "tcp"

  [[services.ports]]
    port = "443"

How do I get the “Fly-Client-IP” header without using the “Http” and “Proxy_proto” handler. I need to be able to get the Users’ IP address.

Any help would be appreciated as I’ve been on this for days now.

PS: It is important that I terminate the SSL with caddy

2

With TCP pass-through, you’d get the forwarded packets delivered to your app. You’d need to use the proxy_proto handler to get the client IPs.

TCP Pass Through

If you don’t specify handlers, we just forward TCP to your application as is. This is useful if you want to handle TLS termination yourself, for example.

(in this context as-is means “from fly-proxy”).

what’s your use-case for terminating TLS without the proxy_proto hander? You might find this thread useful if you need more information about our internal network.

2 Likes
3

Thanks, @eli for the quick feedback.

I’m getting an error “ERR_CONNECTION_CLOSED” when using the “proxy_proto” handler.

Log:

Error: while TCP-proxying data to/from app: failed to copy (direction=server->client, error=Connection reset by peer (os error 104))
4

Caddy is dropping the connection. I don’t think it natively support proxy protocol, but this plugin might help: GitHub - mastercactapus/caddy2-proxyprotocol

2 Likes
5

With Caddy, try one of these non-standard proxy_proto listener modules (1, 2), if you aren’t already. Make sure to configure the module for either proxy_proto version v1 or v2 as approp (Fly supports both: tls + proxy_protocol How to set HTTP/2 ALPN? - #14 by jerome).

1 Like
6

@kurt, @ignoramous. Thank you for pointing this out, I will give it a go now

7

@kurt, @ignoramous. It works perfectly! Thank you for your kind support.

1 Like
8

@mankind Im having a similar issue. Are you able to share your caddy file and fly file? Also, how are you setting x-forwarded-for to be fly-client-IP?

9

If you’re ever curious again, here’s how I solved it at least.

  • Installed the proxy_proto package for caddy. Here’s my dockerfile step
RUN xcaddy build \
    --with github.com/mastercactapus/caddy2-proxyprotocol
  • Configured my Caddyfile with a new global setting (to be clear i have no idea what this is doing haha):
servers {
		listener_wrappers {
			proxy_protocol
			tls
		}
	}
  • Updated my fly.toml with these services and the proxy_proto handlers
[[services]]
  internal_port = 80
  protocol = "tcp"

  [services.concurrency]
    hard_limit = 150
    soft_limit = 100
    type = "connections"

  [[services.ports]]
    handlers = ["proxy_proto"] // <---- NEW
    port = "80"

  [[services.tcp_checks]]
    grace_period = "1s"
    interval = "15s"
    restart_limit = 0
    timeout = "2s"

[[services]]
  internal_port = 443
  protocol = "tcp"
  
  [services.concurrency]
    hard_limit = 150
    soft_limit = 100
    type = "connections"

  [[services.ports]]
    handlers = ["proxy_proto"] // <---- NEW
    port = "443"

  [[services.tcp_checks]]
    grace_period = "1s"
    interval = "15s"
    restart_limit = 0
    timeout = "2s"

Hope this helps anyone :slight_smile:

1 Like