Request: force WebSockets requests to a specific machine from client

[stephen_builds]

Because JS WebSocket cannot pass custom headers, it’s not possible to use the fly-force-instance-id header for WebSockets.

I’d argue being able to force a WebSocket connection to a specific machine is actually one of the most useful reasons to want a specific machine (so you can treat that machine’s memory as low latency shared state for clients). I’m aware there is a workaround for now, which is to use fly-replay from the server once a WebSocket request is made to an incorrect machine.

Is it possible to support fly-force-instance-id as a querystring parameter or some other mechanism that makes it useful for WebSockets?

[stephen_builds]

For future web searchers, it does work to respond to the WebSocket request with a fly-replay header instead of doing an upgrade.

Here’s an example of what I’m doing in Bun:

// inside the WebSocket request handler where you'd normally do an upgrade:
const loadFileResult = fileCache.loadFile(fileId);
...
if (loadFileResult.outcome === LOCKED_BY_OTHER_MACHINE) {
    const otherMachineId = loadFileResult.machineIdWithLock;
    return new Response(undefined, { headers: { 'fly-replay': `instance=${otherMachineId}` } });
}

[GinQuin]

Awesome. I will send the desired machine ID in the query URL of the websocket.

Thank you for sharing!

[khuezy]

Hmm how are you able to return new Response inside a node/bun .on('upgrade') event?

[stephen_builds]

So you actually do the replay header response before upgrading.

Bun.serve takes a fetch and a websocket key in its parameter options.

The upgrade is done inside the fetch handler, which can still return a Response.

Here’s a minimal example:

// fake fetch handler, assumes every request is a websocket upgrade request
function handleFetch(req: Request, server: Server) {
  if (youWantToRedirect) {
    // will replay the upgrade request to another machine
    return new Response(undefined, { headers: { 'fly-replay': `instance=replace-me` } });
  } else {
    // keep the websocket on this machine
    server.upgrade(req);
    // check if upgrade was successful, etc
  }  
}

const server = Bun.serve({
  port: 8080,
  fetch: handleFetch,
  websocket: /* .. some other websocket handler after upgrade */
});

[stephen_builds]

I should note: this websocket replay is working well in my testing and staging but I’m not using it in production yet. It’s possible I’m missing a problem.

[khuezy]

Very cool. I thought a web socket connection is done over a socket, so being able to return a Response in Bun is neat.