proxies websocket requests between the container and frontend
The worker container:
has a websocket server for 2-way communication
I got a test version of my “worker” app running on Fly, and I’m able to spin up multiple instances and connect via the custom url (app-name.fly.dev) pointing to the websocket port – but I’m not able to figure out how to direct messages to specific machines.
Since the api service (also running on fly) is proxying the WS messages, private networking seems like the way to go. Thank you for pointing in that direction, I’m a step closer now.
From the docs I’ve managed to verify that I can ping the private_ip from within the org, but the app-name.internal thing doesn’t seem to work. Am I missing something?
I think that’s the only way I’ll be able to connect to the machine via websocket?
What’s wrong when you try and use the .internal address? Additionally, you can avoid having to proxy the websocket connection at all by leveraging fly-replay. Check out this article from a bit ago for an example of the model: Real-Time Collaboration with Replicache and Fly-Replay · Fly
If you have a Debian based image, and have installed inetutils-ping, then ping6 app-name.internal (assuming you substitute app-name with your real app name, of course) should work. If you have dnsutils installed, try:
dig +short -t txt _instances.internal | sed 's/" *//g; s/;/\n/g'
I tried it out – it likely works, just no luck actually connecting to it.
Tried every variation of this with no success in finding an open host:port combo: curl -vs http://{id}.vms.{app-name}.internal
Edit: able to successfully ping the correct host http://{id}.vms.{app-name}.internal now, but still not able to get a response from the webserver / websocket.
Never had so many issues setting up websockets before. Any chance you can see what I’m doing wrong here? This all works locally without issues, so I’m expecting the issue to be with my fly setup.
import express from "express";
import { WebSocketServer } from "ws";
import http from "http";
const app = express();
app.get('/ping', (req, res) => {
res.send('pong');
});
const wss = new WebSocketServer({
noServer: true,
path: "/websocket",
});
wss.on('connection', (ws) => {
console.log('New WebSocket connection');
ws.on('message', (message) => {
console.log(`Received message: ${message}`);
ws.send(`Echo: ${message}`);
});
ws.on('close', () => {
console.log('Connection closed');
});
});
// Integrate WebSocket server with the Express application
const server = http.createServer(app);
server.on('upgrade', (request, socket, head) => {
wss.handleUpgrade(request, socket, head, (ws) => {
wss.emit('connection', ws, request);
});
});
const port = process.env.PORT || 8080;
server.listen(port, "::", undefined, () => {
console.log(`Server is listening on :${port}`);
});
Nothing is obviously wrong with the code to me. I’ve created express apps with websockets before and never had to handle the upgrade myself, but if that is working locally it should be fine.
What catches my eye is the fly.toml has [[services]] instead of [http_service]. If that is intentional, that’s fine; but otherwise it may indicate that you have an old flyctl version and/or are still on apps v1.
Before I look deeper, can we try it the other way? I can provide you with a working app that uses express and websockets that you can deploy to multiple regions. In an empty directory run:
I think you need to include the internal port: <id>.vm.<app-name>.internal:8080.
The [[services.ports]] section configures the routing at the fly-proxy level for external requests. The proxy receives requests and forwards them to your application, which is listening on port 8080. Requests within the private network do not pass through the proxy [unless you are using Flycast - Private Load Balancing].
@rubys I’ve just been copying parts of examples / blogs that I’m running into! I appreciate the demo project, it definitely provides a better foundation for me.
