Setting HTTP response headers on Fly proxy breaks deployment

Questions / Help Build debugging
, ,
1

Since recently (~3 days ago or more), setting certain HTTP response headers via fly.toml became effectively impossible. The app won’t become reachable when the headers are set. I’m using latest flyctl v0.1.134 and also tested with some older versions back to v0.1.129, but the issue stayed the same. Thus I think the issue lies on the server-side of the Fly platform.

Multiple of my Fly apps are affected (but with subtle differences in the affected headers). The header’s actual values do not seem to matter, just trying to set them causes Fly Proxy to stumble.

An example of an affected Fly app is rdb-postgrest (fly.toml) with owner zda. Setting the following headers reproducibly[1] breaks deployment (app never becomes reachable; connection gets reset when trying to connect to the app):

[http_service.http_options.response.headers]
  Content-Security-Policy = "default-src 'self'; base-uri 'self'; form-action 'none'; frame-ancestors 'none'; script-src 'self'"
  Cross-Origin-Embedder-Policy = "require-corp"
  Cross-Origin-Opener-Policy = "same-origin"
  Cross-Origin-Resource-Policy = "same-site"
  Permissions-Policy = "camera=(), display-capture=(), microphone=()"
  X-Content-Type-Options = "nosniff"
  X-Frame-Options = "DENY"

I am no networking specialist and do not understand what’s happening behind the curtains. For me, this is an absolutely annoying issue to debug due to the Fly platform being kind of a blackbox regarding setting HTTP response headers. There is no logs or anything I can check, it seems.

I would be glad if someone from Fly could shed some light on what’s going on here. Are you restricting setting security-related HTTP response headers to customers on the pro plan or something (I’m on the hobby plan for more than a year now)?

  1. Today it also became impossible to set the X-Content-Type-Options and Cross-Origin-*-Policy headers. 2 days ago, these headers (but not the others listed above) worked fine. ↩︎

1 Like
2

Hi @salimb

It’s a size issue. The maximum metadata size is 512B. The metadata includes not only the headers you provide, but also port descriptions and some other things that fly-proxy needs to properly route traffic to your app.

There may be a way to shrink the metadata. Because your app uses the [http_service] option it exposes both port 80 and port 443, so the specified headers get passed in twice. Since it also has force_https enabled, requests to port 80 get automatically forwarded to port 443. If you reconfigure your fly.toml to use separate [services] blocks (instead of [http_service] ), you can set the appropriate response headers to only get added on port 443, which might get the metadata size down enough to avoid the error.

You’ll need to add one service block per port. I’ve included an example below:

[[services]]  
  auto_start_machines = true
  auto_stop_machines = true
  force_https = true
  internal_port = 3000
  min_machines_running = 1
  protocol = "tcp"

  [services.concurrency]
    hard_limit = 500
    soft_limit = 400
    type = "connections"

 [[services.ports]]
    handlers = ["http"]
    port = 80
    force_https = true

 [[services.ports]]
    handlers = ["tls", "http"]
    port = 443

   [services.ports.http_options.response.headers]
     Content-Security-Policy = "default-src 'self'; base-uri 'self'; form-action 'none'; frame-ancestors 'none'; script-src 'self'"
     Cross-Origin-Embedder-Policy = "require-corp"
     Cross-Origin-Opener-Policy = "same-origin"
     Cross-Origin-Resource-Policy = "same-site"
     Permissions-Policy = "camera=(), display-capture=(), microphone=()"
     X-Content-Type-Options = "nosniff"
     X-Frame-Options = "DENY"
  
 [[services.http_checks]]
    grace_period = "3s"
    interval = "15s"
    method = "GET"
    timeout = "5s"
    path = "/"
    protocol = "http"
2 Likes
3

Thank you @pavel for the fast response and also for the detailed services block example!

The underlying issue being about max. metadata size clears things up—a bit. Actually, I was using separate [[services.ports]] blocks (source) until I ran into this issue. In my attempts to fiddle around with the config, I decided to switch to the simpler [http_service] block and just left it like that. I’ll switch back.

But my main question would be: Was this metadata limit of 512B introduced (or reduced) only recently? The app used to work not long ago with a lot more headers being set…

And are there plans on providing feedback in any form to the user when the metadata limit is exceeded?

4

Hey @salimb

Sorry for late reply.

The limit was always there, but a few things have changed on our side recently.

fly-proxy doesn’t use Consul directly, but rather our own service discovery which periodically pulls data from Consul. At some point we started writing directly to both Consul and our service discovery system, writes to Consul failed due to metadata size, but succeeded to our own service discovery system. That’s why it used to work.

Recently, the sync process between the two systems became a bit stricter and it now removes the services not present in Consul, so it stopped working.

We are looking at possible ways to fix this.

5

Thanks for the follow-up @pavel.

That would be great. I would really appreciate if you could post an update here once it’s fixed.

In the meantime, couldn’t you let flyctl print some sensible error to the user when the limit is exceeded and writing fails / the service is removed?

1 Like
6

Added wishlist

7

Added proxy

8

@pavel I’m new to fly.io and encountered the same issue, and applied the suggested approach after finding this post. The app still becomes unreachable when attempting to add similar security headers (mine has some ‘sha256-xxxx…’ values). Has there been any fixes? If not can you provide any other workarounds to adding response headers without violating the 512B metadata limit? :pray:

9

Hey @AJEdgarCraft

Not yet. It’s possible that 512 bytes metadata limit will be lifted in the future, but I can’t provide any ETA right now.

Hmm, if you already tried the workaround and your configuration still exceeds 512 bytes I think the only way is to respond with these headers from your app (and remove them from fly.toml). The 512 bytes limit applies only to configuration, not to the actual responses.