h2_backend loses incoming request host information

Questions / Help
1

In Support for HTTP trailers?, I learned that I needed to enable the h2_backend option and configure my app to handle the h2c protocol to get HTTP/2 working.

Unfortunately, enabling the h2_backend seems to cause Fly’s proxy layer to lose incoming host information, which my app needs for correct functionality. Here’s what I’m seeing:

:white_check_mark: Without h2_backend enabled
My test app receives requests with expected Host: httpbingo-test.fly.dev header set

:x: With h2_backend enabled
My test app receives requests with Host: 172.19.137.34:8080 header set.

This makes h2c a non-starter for my use case. Is there any chance this is a bug/oversight that can be fixed?

Thanks!

1 Like
2

Hey @mccutchen

Are you getting Host value from net/http.Request.Host variable? If so, it’s set to :authority: pseudo-header for h2 requests:

// For server requests, Host specifies the host on which the
// URL is sought. For HTTP/1 (per RFC 7230, section 5.4), this
// is either the value of the “Host” header or the host name
// given in the URL itself. For HTTP/2, it is the value of the
// “:authority” pseudo-header field.

:authority: being set to the local IP address of the machine is an artifact of our implementation. I’m planning to fix this, but can’t give you any ETA right now.

For now you can check the Host header, it should contain the valid hostname:

❯ curl --http2 https://pbor-h2c-test.fly.dev
r.Host = "172.19.149.250:8080"
r.Header[Host] = "pbor-h2c-test.fly.dev"
3

Interesting, that’s not what I’m seeing in my app. I’ve got this code, with debug logging added, which tries to re-assemble the incoming request URL:

func getURL(r *http.Request) *url.URL {
	log.Printf("XXX getURL: r.URL=%q", r.URL)
	log.Printf("XXX getURL: r.URL.Host=%q", r.URL.Host)
	log.Printf("XXX getURL: r.Host=%q", r.Host)
	log.Printf("XXX getURL: r.Header.Get(\"Host\")=%q", r.Header.Get("Host"))
	log.Printf("XXX getURL: r.Headers=%#v", r.Header)

	scheme := r.Header.Get("X-Forwarded-Proto")
	if scheme == "" {
		scheme = r.Header.Get("X-Forwarded-Protocol")
	}
	if scheme == "" && r.Header.Get("X-Forwarded-Ssl") == "on" {
		scheme = "https"
	}
	if scheme == "" && r.TLS != nil {
		scheme = "https"
	}
	if scheme == "" {
		scheme = "http"
	}

	host := r.URL.Host
	if host == "" {
		host = r.Host
	}

	return &url.URL{
		Scheme:     scheme,
		Opaque:     r.URL.Opaque,
		User:       r.URL.User,
		Host:       host,
		Path:       r.URL.Path,
		RawPath:    r.URL.RawPath,
		ForceQuery: r.URL.ForceQuery,
		RawQuery:   r.URL.RawQuery,
		Fragment:   r.URL.Fragment,
	}
}

When I make a curl --http2 request, I see this in the logs, which shows r.Host and r.Header.Get("Host") both having a value of 172.19.0.26:8080:

2024-09-24T21:20:58.638 app[5683594a146648] dfw [info] 2024/09/24 21:20:58 XXX getURL: r.URL="/get?v=New+York"
2024-09-24T21:20:58.638 app[5683594a146648] dfw [info] 2024/09/24 21:20:58 XXX getURL: r.URL.Host=""
2024-09-24T21:20:58.638 app[5683594a146648] dfw [info] 2024/09/24 21:20:58 XXX getURL: r.Host="172.19.0.26:8080"
2024-09-24T21:20:58.638 app[5683594a146648] dfw [info] 2024/09/24 21:20:58 XXX getURL: r.Header.Get("Host")="172.19.0.26:8080"
2024-09-24T21:20:58.638 app[5683594a146648] dfw [info] 2024/09/24 21:20:58 XXX getURL: r.Headers=http.Header{"Accept":[]string{"*/*"}, "Fly-Client-Ip":[]string{"2601:19e:8200:e6d0::72f2"}, "Fly-Forwarded-Port":[]string{"443"}, "Fly-Forwarded-Proto":[]string{"https"}, "Fly-Forwarded-Ssl":[]string{"on"}, "Fly-Region":[]string{"bos"}, "Fly-Request-Id":[]string{"01J8JYHB7VT2MQPCDQKSMYDCA5-bos"}, "Fly-Traceparent":[]string{"00-998435d8ef7838aa48ad8f290a50851c-ae4dc48b47d48d67-00"}, "Fly-Tracestate":[]string{""}, "Host":[]string{"172.19.0.26:8080"}, "User-Agent":[]string{"curl/8.7.1"}, "Via":[]string{"2 fly.io"}, "X-Forwarded-For":[]string{"2601:19e:8200:e6d0::72f2, 2a09:8280:1::47:e4e2:0"}, "X-Forwarded-Port":[]string{"443"}, "X-Forwarded-Proto":[]string{"https"}, "X-Forwarded-Ssl":[]string{"on"}, "X-Request-Start":[]string{"t=1727212858619402"}}

A curl --http1.1 request on the other hand works as expected:

2024-09-24T21:25:43.962 app[5683594a146648] dfw [info] 2024/09/24 21:25:43 XXX getURL: r.URL="/get?v=New+York"
2024-09-24T21:25:43.962 app[5683594a146648] dfw [info] 2024/09/24 21:25:43 XXX getURL: r.URL.Host=""
2024-09-24T21:25:43.962 app[5683594a146648] dfw [info] 2024/09/24 21:25:43 XXX getURL: r.Host="httpbingo-test.fly.dev"
2024-09-24T21:25:43.962 app[5683594a146648] dfw [info] 2024/09/24 21:25:43 XXX getURL: r.Header.Get("Host")="httpbingo-test.fly.dev"
2024-09-24T21:25:43.962 app[5683594a146648] dfw [info] 2024/09/24 21:25:43 XXX getURL: r.Headers=http.Header{"Accept":[]string{"*/*"}, "Fly-Client-Ip":[]string{"2601:19e:8200:e6d0::72f2"}, "Fly-Forwarded-Port":[]string{"443"}, "Fly-Forwarded-Proto":[]string{"https"}, "Fly-Forwarded-Ssl":[]string{"on"}, "Fly-Region":[]string{"bos"}, "Fly-Request-Id":[]string{"01J8JYT10KHM700QM7RP7RW5TK-bos"}, "Fly-Traceparent":[]string{"00-ad16c715554689f92edd860222e1a1b4-14da2c2558897bff-00"}, "Fly-Tracestate":[]string{""}, "Host":[]string{"httpbingo-test.fly.dev"}, "User-Agent":[]string{"curl/8.7.1"}, "Via":[]string{"1.1 fly.io"}, "X-Forwarded-For":[]string{"2601:19e:8200:e6d0::72f2, 2a09:8280:1::47:e4e2:0"}, "X-Forwarded-Port":[]string{"443"}, "X-Forwarded-Proto":[]string{"https"}, "X-Forwarded-Ssl":[]string{"on"}, "X-Request-Start":[]string{"t=1727213143060047"}}

Any idea why I’m seeing different behavior here?

1 Like
4

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.