Recommended setting for `trust proxy` on express

Questions / Help
1

What are the recommended settings for trust proxy when using Express on node? None of these seem to work: 'loopback', 'linklocal', 'uniquelocal' as the proxy’s address is not in that range. A value of 2 seems to work for me now, but I am wondering if there is a better setting for a subnet that will reliably filter out the load balancers / proxies that fly.io uses.

1 Like
2

Have you tried

app.set('trust proxy', true)

The ExpressJS docs provide this warning:

When setting to true , it is important to ensure that the last reverse proxy trusted is removing/overwriting all of the following HTTP headers: X-Forwarded-For , X-Forwarded-Host , and X-Forwarded-Proto otherwise it may be possible for the client to provide any value.

The Fly.io Runtime Environment docs don’t specify X-Forwarded-Host, but X-Forwarded-Proto should obey that since it’s just a single value rather than a comma-separated list.

The Fly.io HTTP Services docs elaborate a little further.

debug.fly.dev is also quite helpful here.

I do think Fly’s docs would be improved by having a full table including

  • header name
  • description
  • examples
  • what happens if a client sets it (e.g. an attacker attempts IP spoofing)?

I’m considering writing a middleware to copy the Fly-* versions to the X-Forwarded-* ones as they seem to be more tamper-resistant.

1 Like
3

Some testing with debug.fly.dev suggests the X- headers are open to spoofing:

curl -sv\
  -H "Host: injected.host"\
  -H "X-Forwarded-Proto: injected"\
  -H "X-Forwarded-For: 1.2.3.4"\
  -H "X-Forwarded-Port: 1234"\
  -H "X-Forwarded-Ssl: maybe"\
  https://debug.fly.dev/
…
=== Headers ===
Host: injected.host
X-Forwarded-For: 1.2.3.4, (my personal IP address)
X-Request-Start: t=1666727508875392
User-Agent: curl/7.79.1
X-Forwarded-Ssl: maybe
X-Forwarded-Port: 1234
Fly-Client-Ip: (my personal IP address)
Fly-Forwarded-Proto: https
Fly-Forwarded-Ssl: on
Fly-Region: sjc
Fly-Traceparent: 00-c8dd4e09d3d6b8c81ad3861603bef8e1-02b0e9d7fecde099-00
Accept: */*
Fly-Forwarded-Port: 443
Fly-Request-Id: 01GG8B6YWB2KA6XWV19CRXQ31T-sjc
Via: 2 fly.io
Fly-Tracestate:
X-Forwarded-Proto: injected
1 Like
4

@jamesarosen it looks like we shouldn’t “trust proxy” right now!

Did you get round to writing your Fly- → X- express middleware? Sounds like that would be a good hack for now, although careful not to enable it when not running on fly, as presumably that could introduce the same problem in reverse :laughing:

Fly subtly document at Public Network Services · Fly Docs that X-Forwarded-Port cannot be trusted, but don’t mention that the rest of the headers can be spoofed. This looks like a real problem to me. Perhaps more clearly documented by modifying your example like so:

$ curl -sv \
       -H "Host: injected.host" \
       -H "X-Forwarded-Proto: https" \
       -H "X-Forwarded-For: 1.2.3.4" \
       -H "X-Forwarded-Port: 443" \
       -H "X-Forwarded-Ssl: on" \
     http://debug.fly.dev/
...
=== Headers ===
Host: injected.host
Fly-Forwarded-Proto: http
Fly-Forwarded-Port: 80
X-Forwarded-Proto: https
Fly-Client-Ip: (my personal IP address)
Fly-Request-Id: 01GH5YR86ZVDQTGYD6YKB14WQB-sin
Fly-Forwarded-Ssl: off
X-Forwarded-Port: 443
X-Forwarded-Ssl: on
X-Forwarded-For: 1.2.3.4, 188.93.151.254
...
5

Here’s what I have so far:

function preferHeader(request: Request, from: string, to: string): void {
  const preferredValue = request.get(from.toLowerCase());
  if (preferredValue == null) return;

  delete request.headers[to];
  request.headers[to.toLowerCase()] = preferredValue;
}

const flyHeaders: RequestHandler = function flyHeaders(req, res, next) {
  if (process.env.FLY_APP_NAME == null) return next();

  req.app.set("trust proxy", true);

  preferHeader(req, "Fly-Client-IP", "X-Forwarded-For");
  preferHeader(req, "Fly-Forwarded-Port", "X-Forwarded-Port");
  preferHeader(req, "Fly-Forwarded-Proto", "X-Forwarded-Protocol");
  preferHeader(req, "Fly-Forwarded-Ssl", "X-Forwarded-Ssl");

  return next();
};
1 Like
6

@jamesarosen :+1: thanks for sharing! In the meantime, I wrote something very similar:

function apply(router) {
  if(!process.env.FLY_ALLOC_ID) {
    // Not running on fly
    return;
  }

  router.use(middleware);
}

function middleware(req, res, next) {
  remapHeader(req, 'x-forwarded-proto', 'fly-forwarded-proto');
  remapHeader(req, 'x-forwarded-port',  'fly-forwarded-port');
  remapHeader(req, 'x-forwarded-ssl',   'fly-forwarded-ssl');

  next();
}

function remapHeader({ headers }, to, from) {
  headers[to] = headers[from];
  delete headers[from];
}

I’m not convinced that remapping Fly-Client-IP to X-Forwarded-For is correct though. For me, the last entry in that list is a fly.io IP address, and so (I guess) correct. It is also not my IP address (Fly-Client-IP).

Reading the MDN docs on this header, it seems like leftmost values in this header are expected to be untrusted/spoofable, so there’s nothing surprising in Fly’s current handling of this header.