Recommended setting for `trust proxy` on express

[AlexRothberg]

What are the recommended settings for trust proxy when using Express on node? None of these seem to work: 'loopback', 'linklocal', 'uniquelocal' as the proxy’s address is not in that range. A value of 2 seems to work for me now, but I am wondering if there is a better setting for a subnet that will reliably filter out the load balancers / proxies that fly.io uses.

[jamesarosen]

Have you tried

app.set('trust proxy', true)

The ExpressJS docs provide this warning:

When setting to true , it is important to ensure that the last reverse proxy trusted is removing/overwriting all of the following HTTP headers: X-Forwarded-For , X-Forwarded-Host , and X-Forwarded-Proto otherwise it may be possible for the client to provide any value.

The Fly.io Runtime Environment docs don’t specify X-Forwarded-Host, but X-Forwarded-Proto should obey that since it’s just a single value rather than a comma-separated list.

The Fly.io HTTP Services docs elaborate a little further.

debug.fly.dev is also quite helpful here.

I do think Fly’s docs would be improved by having a full table including

• header name
• description
• examples
• what happens if a client sets it (e.g. an attacker attempts IP spoofing)?

I’m considering writing a middleware to copy the Fly-* versions to the X-Forwarded-* ones as they seem to be more tamper-resistant.

[jamesarosen]

Some testing with debug.fly.dev suggests the X- headers are open to spoofing:

curl -sv\
  -H "Host: injected.host"\
  -H "X-Forwarded-Proto: injected"\
  -H "X-Forwarded-For: 1.2.3.4"\
  -H "X-Forwarded-Port: 1234"\
  -H "X-Forwarded-Ssl: maybe"\
  https://debug.fly.dev/
…
=== Headers ===
Host: injected.host
X-Forwarded-For: 1.2.3.4, (my personal IP address)
X-Request-Start: t=1666727508875392
User-Agent: curl/7.79.1
X-Forwarded-Ssl: maybe
X-Forwarded-Port: 1234
Fly-Client-Ip: (my personal IP address)
Fly-Forwarded-Proto: https
Fly-Forwarded-Ssl: on
Fly-Region: sjc
Fly-Traceparent: 00-c8dd4e09d3d6b8c81ad3861603bef8e1-02b0e9d7fecde099-00
Accept: */*
Fly-Forwarded-Port: 443
Fly-Request-Id: 01GG8B6YWB2KA6XWV19CRXQ31T-sjc
Via: 2 fly.io
Fly-Tracestate:
X-Forwarded-Proto: injected

[aln]

@jamesarosen it looks like we shouldn’t “trust proxy” right now!

Did you get round to writing your Fly- → X- express middleware? Sounds like that would be a good hack for now, although careful not to enable it when not running on fly, as presumably that could introduce the same problem in reverse

Fly subtly document at Public Network Services · Fly Docs that X-Forwarded-Port cannot be trusted, but don’t mention that the rest of the headers can be spoofed. This looks like a real problem to me. Perhaps more clearly documented by modifying your example like so:

$ curl -sv \
       -H "Host: injected.host" \
       -H "X-Forwarded-Proto: https" \
       -H "X-Forwarded-For: 1.2.3.4" \
       -H "X-Forwarded-Port: 443" \
       -H "X-Forwarded-Ssl: on" \
     http://debug.fly.dev/
...
=== Headers ===
Host: injected.host
Fly-Forwarded-Proto: http
Fly-Forwarded-Port: 80
X-Forwarded-Proto: https
Fly-Client-Ip: (my personal IP address)
Fly-Request-Id: 01GH5YR86ZVDQTGYD6YKB14WQB-sin
Fly-Forwarded-Ssl: off
X-Forwarded-Port: 443
X-Forwarded-Ssl: on
X-Forwarded-For: 1.2.3.4, 188.93.151.254
...

[jamesarosen]

Here’s what I have so far:

function preferHeader(request: Request, from: string, to: string): void {
  const preferredValue = request.get(from.toLowerCase());
  if (preferredValue == null) return;

  delete request.headers[to];
  request.headers[to.toLowerCase()] = preferredValue;
}

const flyHeaders: RequestHandler = function flyHeaders(req, res, next) {
  if (process.env.FLY_APP_NAME == null) return next();

  req.app.set("trust proxy", true);

  preferHeader(req, "Fly-Client-IP", "X-Forwarded-For");
  preferHeader(req, "Fly-Forwarded-Port", "X-Forwarded-Port");
  preferHeader(req, "Fly-Forwarded-Proto", "X-Forwarded-Protocol");
  preferHeader(req, "Fly-Forwarded-Ssl", "X-Forwarded-Ssl");

  return next();
};

[aln]

@jamesarosen thanks for sharing! In the meantime, I wrote something very similar:

function apply(router) {
  if(!process.env.FLY_ALLOC_ID) {
    // Not running on fly
    return;
  }

  router.use(middleware);
}

function middleware(req, res, next) {
  remapHeader(req, 'x-forwarded-proto', 'fly-forwarded-proto');
  remapHeader(req, 'x-forwarded-port',  'fly-forwarded-port');
  remapHeader(req, 'x-forwarded-ssl',   'fly-forwarded-ssl');

  next();
}

function remapHeader({ headers }, to, from) {
  headers[to] = headers[from];
  delete headers[from];
}

I’m not convinced that remapping Fly-Client-IP to X-Forwarded-For is correct though. For me, the last entry in that list is a fly.io IP address, and so (I guess) correct. It is also not my IP address (Fly-Client-IP).

Reading the MDN docs on this header, it seems like leftmost values in this header are expected to be untrusted/spoofable, so there’s nothing surprising in Fly’s current handling of this header.