tls + proxy_protocol How to set HTTP/2 ALPN?

[nazar554]

Hi I’m using tls offloading with proxy_protocol and it’s great (I see the corrent client IP), but I still can’t get a HTTP/2 connection negotiated in any way (including HTTP2 prior knowledge).

It seems that there is no way to set TLS ALPN to be h2, it’s always http/1.1. Is there any way to tell the fly app to advertise HTTP2 support on a specific port.

# This is my networking section
[[services]]
  internal_port = 8080
  protocol = "tcp"

  [[services.ports]]
    handlers = ["tls", "proxy_proto"]
    port = "443"

Server hello examples (note it’s not h2):

[nazar554]

Nevermind, found it
Here GitHub - fly-apps/grpc-service: Running gRPC services on Fly.io
And here

Fly Proxy H/2 Details

It’s definitely not passthrough if you have the http handler. If you want to pass through http/2, you can enable only the tls handler and use the following port definition: [[services.ports]] handlers = ["tls"] port = "443" tls_options = { alpn = ["h2", "http/1.1"] } Or just h2 in the list of alpn. Then your app will need to understand http/2 over plaintext.

[kurt]

Nice! What kind of app are you building?

[nazar554]

A .NET 6 gRPC app. Wrote my own connection middleware to process proxy protocol header. It seems that fly is only using plain text PROXY V1, so it could been a bit simpler.

Currently having issues with connection processing where Kestrel complaints about h2/h2c and http/https scheme mismatch, since it’s not aware about TLS offload (endpoint is HTTP2 only). And also it’s not possible to process both h2c and http/1.1 on same port.

[nazar554]

It seems that Kestrel needs to know the ALPN to choose the correct protocol.
I can force it to be h2, but it will break if client used http/1.1.

Probably can hack it to detect HTTP2 client preface as in RFC7540

If fly.io supported PROXY V2 then it would be possible to use PP2_TYPE_ALPN

[jerome]

Will consult with the team and get back to you

Sounds reasonable.

[nazar554]

Nice! For now I managed to make it work with the client preface hack. Now I’m figuring out the how to use OpenTelemetry stuff for metrics (on a isolated endpoint without proxy protocol)

[nazar554]

You can find my current middleware here.

GitHub - nazar554/Scintillating.ProxyProtocol.Middleware: ASP.NET Core...

[ignoramous]

Wait a second: Is Fly’s proxy_proto handler v1 only? If so, it shouldn’t support for IPv6 (which I think it does)?

[nazar554]

No, IPv6 works correctly with even with proxy V1, see the example here

  - TCP/IPv6 :
      "PROXY TCP6 ffff:f...f:ffff ffff:f...f:ffff 65535 65535\r\n"
    => 5 + 1 + 4 + 1 + 39 + 1 + 39 + 1 + 5 + 1 + 5 + 2 = 104 chars

[ignoramous]

^ Thanks.

---

@jerome et al any timeline for support for proxy_proto v2? The ClientHello contents (specifically, ALPN and SNI) sent in v2 is helpful for our use-case.

[ignoramous]

Any update on supporting PROXY v2? Our middleware (request router) needs to peek at TLS ClientHello, too.

[jerome]

Update: We have a working branch for proxy proto v2.

I’m going to test it out a bit before letting you in on it.

[jerome]

Update: This is now available.

If you use something like this for your port handler, it should work:

[[services.ports]]
handlers = ["proxy_proto"]
port = "5000"
proxy_proto_options = { version = "v2" }

[nazar554]

I can confirm I receive the PP2_TYPE_ALPN but no PP2_TYPE_SSL.
I guess I’ll have to change the code to assume that receiving either of these means the connection is using TLS.

[jerome]

We can fix that

[nazar554]

Did you change something in PROXY V2 support?

My app started crashing with a 0x14 TLV (length is 0x3B = 59 bytes), see the 14 00 3b sequence in hexdump
I think when you tried adding PP2_TYPE_SSL you used the wrong TLV value in decimal, since 0x14 HEX = 20 DEC, but it’s defined as hexadecimal

#define PP2_TYPE_SSL 0x20

00000000  0d 0a 0d 0a 00 0d 0a 51  55 49 54 0a 21 11 00 70   .......Q UIT.!..p
00000010  2e 76 78 d1 6d 69 d8 8e  05 40 01 bb 01 00 02 68   .vx.mi.. .@.....h
00000020  32 02 00 1e 73 61 72 64  61 75 6b 61 72 2d 64 31   2...sard aukar-d1
00000030  36 65 65 63 37 38 31 62  34 31 2e 66 6c 79 2e 64   6eec781b 41.fly.d
00000040  65 76 14 00 3b 01 00 00  00 01 15 00 07 54 4c 53   ev..;... .....TLS
00000050  76 31 2e 33 19 00 05 45  43 44 53 41 18 00 06 53   v1.3...E CDSA...S
00000060  48 41 32 35 36 17 00 18  54 4c 53 31 33 5f 41 45   HA256... TLS13_AE
00000070  53 5f 31 32 38 5f 47 43  4d 5f 53 48 41 32 35 36   S_128_GC M_SHA256

[jerome]

Yes we did add the SSL TLV.

It might be slightly wrong. We can take a look in about 2 hours. Until then I recommend switching the v2 option off if it’s not too much trouble. (If it’s crashing your app)

[amos]

Yeah, we added SSL and Authority TLVs last week: https://poppy-rust.fly.dev/

The 0x20/20 mixup seems like a bug in a crate we’re using: model.rs - source

Fairly easy to work around, unfortunately I’m off today.

However, I wanted to recommend making sure your app doesn’t crash on unrecognized TLVs - the spec may evolve later and new TLVs might be added, it’s relatively straightforward to skip over them since they’re prefixed by type+length.

[jerome]

@nazar554 we deployed a fix for this. It should now have the correct value.