UDP changes in some regions?

1

Were there any changes to some regions recently that may have affected UDP traffic? For a while now, I’ve been using LiveKit in multiple regions. Sometime recently, some regions have stopped working.

It requires a broad UDP port range (50000-60000), but I’ve been using the trick that if you open one UDP port, all UDP traffic works. So my config just has:

[[services]]
http_checks = []
internal_port = 10000
protocol = "udp"
script_checks = []

[[services.ports]]
handlers = []
port = 10000

As of some time recently, I’m getting TCP traffic just fine, but my UDP connections are acting like they aren’t being routed.

Working: sea, fra
Failing: iad, ord

I know that I saw all of my ord instances restart some time last night for multiple apps, so I’m wondering if there was a newer change to that and some other regions that has caused this to start failing. If so, is there a change I can make on my end to restore this or is there something that can be done to restore UDP ranges on Fly’s end?

For a test case, this is a guide for bringing up a LiveKit instance using a similar config: GitHub - bekriebel/livekit-flydotio: An example on how to deploy LiveKit on Fly.io

2

What’s your app? We’re working out a kernel problem on a couple of worker machines; if you’re relying on UDP, I’ll add a constraint for your jobs to ensure you don’t get scheduled to those machines.

(Feel free to DM or email).

3

Sent via DM. There’s also a discussion on the LiveKit Slack where @ryanbrwr was seeing this issue. He’s the one who pointed out iad was having the issue, so I tested it and found it is having the same issue I was originally seeing in ord.

4

Thanks! Followed up in DM.

5

I am also unable to launch with livekit on fly.

I have tried:
iad
sea
ord

so far all seem to have the same issue that @bekit is having.

6

@ryanbrwr Sorry about that, we weren’t steering UDP apps to the right places properly on deploy. I’m pretty sure this bug is fixed now and you should be good to go.

7

Amazing thanks, should I redeploy?

8

I applied the fix to your app already so you may not need to, will you check and see if it’s behaving itself?

9

It is working! Thank you, appreciate the help