I drove myself a little nuts getting a DNS feature shipped yesterday, and since I wrote the UDP feature for Fly.io, I figured if I was going nuts, so must everyone else. I tried to write down all the stuff you might need to know to get a UDP/TCP app working on Fly.io here:
Thanks. I want to point out that this “app guide” remains unlinked from the sidebar nav section “Guides and Examples”.
And more annoyingly importantly, https://fly.io/docs/app-guides/ is 4xx. It’d be rad, if fly-docs went unix-y, and listed all sub-guides and guides (subdirs and files).
2 Likes
Yep. We’re working on beating the dox into shape.
1 Like
Has anything changed recently in regards to how UDP works? I just tried deploying a new server in another region (GRU), and I seem to be having trouble getting my UDP connections through.
One thing not mentioned in the docs that at least was true before was that if you enabled one UDP port in your fly.toml, ALL udp ports would get forwarded. I’m taking advantage of this fact because LiveKit (and most WebRTC apps) user random ports in a range, in my case 50,000 - 60,000. Since the fly config doesn’t support port ranges, it’s been beneficial that I can just list one port and get all of them opened. However, for some reason or another, my newest instance is failing to connect on the UDP ports.
After some more testing, I think my suspicions are confirmed. If this is the case, it’s a rather unfortunate change for me.
Is there any way for me to either specify a UDP port range or go back to the previous UDP behavior opening up all UDP ports? It’s not uncommon for apps that use UDP to use TCP for signaling and broad UDP port ranges.
The UDP BPF code hasn’t changed meaningfully in many months, and none of those changes make UDP forwarding port-specific: if you light up UDP at all for your app, we’ll forward all UDP for your Anycast address to your instances.
Odd. Looks like my problem is something different. It looks like every other time I deploy, I get a version that doesn’t work. It just so happened that every time I was trying to list the individual ports, it worked, and every time I went back to a single port in my config it was failing. I finally did a second deploy without making any changes and it worked.
Is there possibly something on one of the host machines in GRU that is causing UDP not to work and it just matters which host my app is landing on during a deploy?
This worried me, so I put a little UDP echo service on each GRU worker, individually, and tested; they all seem fine.
If you want to clone your app onto a count=1 small job and get it into a state where it isn’t working, and then tell me where to find it, I can take a closer look.
Thanks for looking into it.
I just repeated the same tests I was doing last night and I’m not able to reproduce the error today. I guess it was something intermittent or temporary. If it happens again, I’ll leave a server in that state and let you know.
Thanks again!
Thank you for letting us know about it!
FYI: The nc example in the article uses TCP, not UDP like the text claims. I can’t get UDP ncat to work: Trouble getting UDP to work
Probably related to DNS over TCP works, but UDP doesn't - #18 by kurt
Just a quick note that I tried this on another thread and couldn’t repro a problem:
I had not seen anything about this requirement to bind to a special address (and not 0.0.0.0) for UDP before, so that’s very good to know!
Oh! That reminds me. There’s only 3 UDP gotchas now, not 4, since Jerome slayed the Dreaded Port Filter (RIP). I’ve updated the docs accordingly.