Fly-Instance-Id header alternative for websockets

Questions / Help
1

The Fly-Instance-Id: <id> header allows you to route an HTTP request to a specific instance. However, this does not work for client side JavaScript websockets, because (unlike the XHR API) you cannot specify headers.

It’s a bit a stretch, but is there a way to target a specific instance using DNS, or even a magical querystring parameter?

I can make it work by creating an app for each region, but that seems rather wasteful (considering each app gets an IPv4).

The motivation for the question is that I’m building a minimal single process websocket server, where the messages are only passed around to clients connected to the same server.

2

One could create multiple apps (each get their own IP), and plonk the IPs in any number of DNS A/AAAA/HTTPS/SVCB records as you see fit (1).

One could run one server per region, assign them regional IPs, and map those to DNS records (2).

One can run app over multiple Machines (which, for now, each get a fixed public IPv6 address) (3).

Anchor scaling also helps keep public IPv6 stable for regular Fly app instances, but I see references to it have been removed from docs. Anyways, ref: (4).

One could route traffic to a set of front-end app instances that then forward it to appropriate back-end app instance:

  1. Either connect back-end and front-end instances over 6pn (5, 6).
  2. Or, relay traffic with Fly’s HTTP handlers (unlikely to work with websockets, I’d assume) (7).
1 Like
3

Thank you! That’s extremely useful information (I’m sorry I didn’t find those threads myself). Running a single app, with single instances in each region (with an assigned regional IP) seems like the best bet for me.

1 Like
4

Just wanted to update this thread for anyone who finds it:

First you need to add the regions to your app
flyctl regions add sea iad lhr
Then you need to add IPs for each region

flyctl ips allocate-v4 --region sea
flyctl ips allocate-v4 --region lhr
flyctl ips allocate-v4 --region iad

If you get an error with the IP commands, that means the region does not support regional IPs yet (so pick a different region).

Now, (this is what tripped me up), you need to tell fly.io that you want VMs running in each region:

fly scale count --max-per-region 1 3

The max-per-region flag is important, otherwise fly.io might allocate 2 VMs to the same region (which will not work in this situation, because I need exactly 1 in each region).

Create a DNS entry for each region with the regional IP address, configure SSL, and you’re off to the races.

2 Likes
5

One could consider deploying Machine apps instead, if one wants to break free of their Nomad orchestration.

There are rough edges, notably with how traffic is spread over to Machines by Fly proxy and a lack of clarity around health checks, but things can only improve from here on out.

6

As an aside, I have tried this document.cookie method to set a header and it does work, not that I tested extensively. If domain cookies can be set you may be able to use it.

var authToken = 'R3YKZFKBVi';

document.cookie = 'X-Authorization=' + authToken + '; path=/';

var ws = new WebSocket(
    'wss://localhost:9000/wss/'
);
1 Like