proxy_proto + in-app TLS termination.

1

Hi :wave:

Context: I am deploying a reverse proxy (Envoy) which handles TLS termination and doing TCP pass-through. My fly.toml looks like this:

[[services]]
  internal_port = 10000
  protocol = "tcp"

  [[services.ports]]
    port = "443"
…

When reviewing Envoy access logs, I see that the downstream address is Fly’s proxy.

{
    "downstreamRemoteAddress": {
        "socketAddress": {
            "address": "136.144.55.141",
            "portValue": 42428
        }
    },
    "downstreamLocalAddress": {
        "socketAddress": {
            "address": "172.19.1.234",
            "portValue": 10000
        }
    }

I see in the documentation that there is support for a proxy_protocol handler: Public Networking · Fly

When testing locally with HAProxy as a stub Fly proxy, everything works fine. Here’s the relevant part of the HAProxy config:

backend envoy
    mode tcp
    server envoy1 127.0.0.1:10000 send-proxy-v2
    # v1 works too
    # server envoy1 127.0.0.1:10000 send-proxy

but when deployed to Fly with the following fly.toml I get an error:

curl: (35) LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to …

[[services]]
  internal_port = 10000
  protocol = "tcp"

  [[services.ports]]
    handlers = ["proxy_proto"]
    port = "443"

What am I missing? is it possible to do TCP pass-through (for TLS) with proxy_proto?

Thanks!

2

This should indeed work. It will prepend a plaintext proxy protocol V1 line to the TLS Client Hello.

I’m looking into this now.

3

@pims As far as I can tell, we’re sending the correct proxy protocol line.

I created a small golang app that uses a proxyproto package to wrap a net.Listener. I’ve only made it work successfully for a non-TLS listener. the tls.Listener was trying to read the TLS ClientHello right away and wouldn’t let the proxy protocol reader peek at the bytes without triggering an error.

In your setup, Envoy is the accepting/parsing the proxy protocol? Can you try with a non-TLS endpoint?

4

Oh, I wrapped my proxy protocol listener the wrong way, I needed the tls.Listener on the outside. Therefore, this works for my on both non-TLS and TLS listeners.

5

Interesting. I’ll give it another try this evening. I’ve added both configs to this gist: proxy_protocol: envoy + haproxy · GitHub

The above Envoy config works with HAProxy in TLS mode. My goal was to replicate this setup with Fly instead of HAProxy.

Thanks for looking into it.

6

@pims I was able to reproduce locally and “fix” it by setting the proper filter_chain_match.

I know next to nothing about envoy, but, as far as I can tell, you’ll need to specify every hostname you’re expecting to match with your certificates in there:

# ...
    filter_chains:
    - filter_chain_match:
        server_names: ["localhost","localhost:10000"]
      transport_socket:
        name: envoy.transport_sockets.tls
# ...

I set my envoy to the debug log level (with the -l debug CLI flag) and saw these:

[2021-01-25 18:23:30.053][17][debug][filter] [source/extensions/filters/listener/proxy_protocol/proxy_protocol.cc:68] proxy_protocol: New connection accepted
[2021-01-25 18:23:30.056][17][debug][filter] [source/extensions/filters/listener/tls_inspector/tls_inspector.cc:79] tls inspector: new connection accepted
[2021-01-25 18:23:30.056][17][debug][conn_handler] [source/server/connection_handler_impl.cc:469] closing connection: no matching filter chain found

I was testing with envoy v1.17

7

I guess my response was a bit confusing.
I have properly setup Envoy on Fly with tcp pass through, and support for TLS. It all works well.

The part that doesn’t work on Fly, but works locally, is configuring proxy protocol.

The configs I have pasted are from my local test with HAProxy, not what I have deployed to Fly.

I’ll give it another try later this evening. Since you’ve shown that it works for you, I must be doing something wrong somewhere.

Thanks for your help!

8

I confirm that the proxy_proto handler works with Envoy. The issue was with my dynamic Envoy configuration. Thanks again for looking into this @jerome

1 Like
9

Woo! Glad that worked. You should write up an Envoy article that we can promote sometime. :smiley:

10

Is there any chance proxy-proto headers could be fragmented over tcp segments?

Worst case for v1 proxy-proto header is 107 bytes, but still any possibility?