Proxy protocol header not sent.

Questions / Help
1

Hi I am trying to use the proxy_proto handler but so far I am unlucky.

My relevant fly.toml has:

  [[services.ports]]
    handlers = ["tls", "proxy_proto", "http"]
    port = "443"

And looking with tcpdump inside the container I see the request coming in without any proxy prefix. Is the order relevant, does this not work for http?

Thanks,
Florian

2

Are you sure you need the proxy protocol? If your application already supports HTTP, you can use Fly-Client-IP or X-Forwarded-For to fetch the client IP without additional config.

3

Need is relative. Interestingly enough it would be “easier” to configure for me (since the webserver supports it ootb).

I cannot really use X-Forwaded-For because fly.io allows spoofing it (I just checked via tcpdump). I could write a middleware that puts fly-client-ip into REMOTE_ADDR though.

Nevertheless I was wondering if I am doing something wrong. I am using fly.io since a day and when something doesn’t work I’d like to understand what I am doing wrong, irrespective of whether it is the best option – simply in the spirit of learning.

4

You’re doing nothing wrong! It’s just not clear from the docs that these handlers are mutually exclusive.

The proxy protocol sits a layer above HTTP, so these handlers can’t work together. Your web server will be the one handling HTTP requests directly. proxy_proto is mostly for scenarios like offering custom TLS certificates for their your own customers.

Does that make sense?

5

Well if you say it is a fly.io limitation then yes it makes sense :slight_smile: We use proxy protocol quite extensively between haproxy and nginx at other locations, so it didn’t occur to me that it would not work via http on fly.

Thanks for clearing that up.

6

It used to work so this sounds like a regression.

Let’s see if I can fix that this morning.

7

Are you using services.concurrency.type = "requests"? Can you show us your full fly.toml?

8

No I am not, have to read up what that is. This is my full fly.toml:

app = "the-app"

kill_signal = "SIGINT"
kill_timeout = 5

[deploy]
  release_command = "django-admin migrate"

[[statics]]
  guest_path = "/app/static"
  url_prefix = "/static"

[[services]]
  internal_port = 8000
  protocol = "tcp"

  [services.concurrency]
    hard_limit = 25
    soft_limit = 20

  [[services.ports]]
    handlers = ["http"]
    port = "80"
    force_https = true

  [[services.ports]]
    handlers = ["tls", "proxy_proto", "http"]
    port = "443"

  [[services.tcp_checks]]
    interval = 10000
    timeout = 2000
9

I cannot find that in the docs at all. Google only shows Load balancer documentation as a hit without much more details.

10

The feature has not been publicized really. I was only wondering because it takes slightly different code paths in our proxy.

@apollo13 I just looked at your app and I don’t see the proxy_proto handler set at all. Are you sure the deploy succeeded?

Edit: Oh I see now you changed it back.

11

@jerome well yes and no, I reverted back from it since it did not work.

12

FWIW, I did not try all handler ordering combinations. So if this setting is order dependent, then I might have missed it; but given that I dumped the whole connection inside the firecracker VM with tcpdump I am pretty sure that it didn’t send it, I could add port 444 (maybe?) with proxy_proto set up and retry. Would that help you for debugging?

13

Yes, that would help.

And no, ordering doesn’t matter.

14

Ok deployed as v23 and port is 8443.

15

Would your server error or does it handle a missing proxy protocol gracefully?

16

It handles it gracefully (ie it just ignores a non-existing PROXY line)

17

Ok, I think I fixed it (for the host where your app currently is).

There was a regression. For now I have to fake the source port because we don’t have that information all the time, at least not until I merge my branch and deploy further out.

I’m seeing internal logs about the proxy protocol being applied, so I’m fairly confident it works now.

18

Yes, I just confirmed via tcpdump. Thank you very much! Is it safe to change the configuration on my side now or will that deploy the app on a new host and it would be better if I wait a week or so? (Would it be possible to tell me when this is rolled out globally?)

19

Not safe yet, I’ll let you know once we roll it out everywhere.

1 Like
20

Should be good now.