UDP Services Working on 0.0.0.0 + Rate Limiting

Questions / Help
1

Hi,
I’m running a private UDP service (well, it’s a DNS server on UDP port 53).

I read the docs requiring it to bind to the magic fly-global-services address, but I’ve found out that when I simply bind to 0.0.0.0 the service… just works, at least over IPv4 non-shared address.

Is the documentation outdated? Did you guys figure out how to do UDP without such tricks :exploding_head:?

On a different note - is there a built-in way to limit the outgoing bandwidth of my app? I don’t want the server to be abused.

Thanks!

1 Like
2

On bandwidth abuse, Fly may refund within reason: How concerned should I be about a surprise bill? - #9 by kurt

To limit bandwidth, you can employ tc (I haven’t used it on Fly, so you’ll need to test if it works as expected): Introduction to Linux Traffic Control

Tailscale engs mentioned that they shape traffic on their relay servers hosted on Fly, but I’m not sure what they use. May be someone at Fly does.

Surprising it works. Docs are open, any one can propose edits: https://github.com/superfly/docs/blob/d52feb7fa83c4dbf9589742c4f2ae2fdc6e16338/app-guides/udp-and-tcp.html.md.erb

3

0.0.0.0 works sometimes, it depends entirely on the networking library.

The problem is the return path. Some libraries send UDP responses from the wrong IP when you bind 0.0.0.0. Which is silly. They use the first IP they can discover from the system for all UDP responses.

UDP only works if you reply from the fly-global-services IP. So if it works for you, it will continue to work. But it’s actually somewhat rare that libraries do the right thing.

2 Likes
4

@kurt dang, you’re right! This is so cool!

Looks like DNSMASQ/PiHole works correctly :partying_face: , without fly-global-services. Nice.

On the other hand, a socat echo server (udp port 54, just as an example) doesn’t work out-of-the-box:

# Doesn't work. 
# Incoming packets are from 172.19.65.179, which happens to be the 2nd address assigned to eth0
# Socat naively responds from 172.19.65.178, which is incorrect (and the 1st address of eth0).
socat -v UDP-LISTEN:54,fork,bind=0.0.0.0 exec:'/bin/cat'

# Works:
socat -v UDP-LISTEN:54,fork,bind=fly-global-services exec:'/bin/cat'

Client side:

socat - UDP:app-name.fly.dev:54

Anyhow, maybe the fact that the issue is on the return path and some libs handle it correctly is worth documenting.

5

Regarding the other thing - @kurt - any idea what’s up with tc? Isn’t it supported in the firecracker-microvm setup or something?

$ tc qdisc add dev eth0 root tbf rate 2.5kbit burst 160kbit latency 1s
Error: Specified qdisc not found.

Is there a different way to limit traffic?

6

Looks like we’re missing the sch_netem module. Let me check if I can compile that for you.

2 Likes
7

If you restart your app’s instance, you should get a kernel with that module loaded in. Let us know how that goes! (I only deployed this on the server where your volume / instance lives)

8

Thanks so much!

I was a bit dumb and instead of restarting the app, I re-deployed it (fly deploy) and by that I assume the host changed.

Can you make the change again and I’ll check?

9

If you’re mounting the volume you have in otp, then it should’ve been on the same host.

You can validate the config is there with:

/ # zgrep SCH_NETEM /proc/config.gz
CONFIG_NET_SCH_NETEM=y
10

Looks like I’ll need to enable more configs for this to work. The qdisc kind is not found (new error).

11

Confirmed, I’m running on the modified host.

Now I’m getting

Error: Specified qdisc kind is unknown.

Running lsmod prints nothing; I assume that a module has to be compiled and loaded?

Edit: this :point_down:

12

We don’t provide compiled packages to be added at runtime. We compile them as part of the kernel image only.

I know which ones you probably need to make this work (after some research), but now I’m wondering which extra ones I should enable by default.

13

Ok this works now:

/ # tc qdisc add dev eth0 root tbf rate 2.5kbit burst 160kbit latency 1s
/ # tc qdisc list
qdisc noqueue 0: dev lo root refcnt 2
qdisc tbf 8001: dev eth0 root refcnt 2 rate 2496bit burst 20Kb lat 1s
2 Likes
14

Thanks. Will it be available in the next Kernel update for everyone?

15

Yes.

Right now this also comes with a bump in kernel version (from 5.12.2 to 5.15.93, but I might as well make it 5.15.98 since that’s the latest 5.15). Previous compilations of 5.15 have been deployed to a few servers as a test, but not many.

I’ll try to get this rolled out soon.

2 Likes
16

You are the best, thanks!

I know I’m bumping this thread after it’s been solved, but this is just too awesome not to say an explcit thank you. I definitely didn’t expect this level of service. Appreciated.

Matan

2 Likes
17

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.