Incoming! UDP Support Arriving Soon

Hello, Flyers!

Just a quick-ish note about some stuff we’ve been working on. Sometime in the coming weeks, we’ll be opening a beta for UDP Anycast services — so, you can take a Docker container that serves authoritative DNS for a zone (or a million zillion zones) and quickly deploy it across the globe.

How this’ll work for users is, you’ll open up your trusty fly.toml and add a service whose protocol is UDP — err, "udp". You’ll set up your app to bind to a special address — fly-global-services. When you deploy, we’ll start routing UDP traffic to your IP address to to your instances, on that fly-global-services address. Things will “just work”: you’ll get the actual source addresses of packets, and your responses will (of course) bear your anycast addresses.

This is one of those features that is super straightforward to describe but was pretty complicated to actually implement. I’ll go into more detail in the future, but the short description is: we use XDP/BPF to relay packets arriving at our edge across our WireGuard mesh to the nearest worker, without them ever seeing userland. We quietly slip proxy headers on and off your packets, and rewrite addresses accordingly. It’s pretty neato.

If anyone’s super interested in playing with this, please let us know! Depending on your tolerance for jank, we might be able to get you started relatively soon. My hope is that for most UDP applications, there’s really not much you have to do to make it work other than plugging the ports into fly.toml.

4 Likes

If you’re interested in DNS examples, let us know what you want to see. We have pi-hole running (kind of silly) and an authoritative nameserver setup in the works with CoreDNS.

Aformentioned pi-hole config:

app = "fli-hole"


[[services]]
  internal_port = 80
  protocol = "tcp"

  [services.concurrency]
    hard_limit = 25
    soft_limit = 20

  [[services.ports]]
    handlers = []
    port = "80"

  [[services.ports]]
    handlers = ["tls"]
    port = "443"

  [[services.tcp_checks]]
    interval = 10000
    timeout = 2000

[[services]]
  internal_port = 53
  protocol = "udp"

  [[services.ports]]
    port = "53"
1 Like

Is UDP on IPv6 implemented?

I’m unable to connect to a UDP port exposed on the load balancer IPv6 address.
And I’m also unable to do any UDP connections inside the VM/container to outside IPv6 addresses.

Here is a test done with dig inside a fly VM/container:

bash-5.0# dig @2001:4860:4860::8888 google.com

; <<>> DiG 9.16.6 <<>> @2001:4860:4860::8888 google.com
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached

bash-5.0# dig @2001:4860:4860::8888 google.com +tcp

; <<>> DiG 9.16.6 <<>> @2001:4860:4860::8888 google.com +tcp
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 49548
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;google.com.			IN	A

;; ANSWER SECTION:
google.com.		299	IN	A	216.58.215.46

;; Query time: 12 msec
;; SERVER: 2001:4860:4860::8888#53(2001:4860:4860::8888)
;; WHEN: Thu Feb 04 16:35:35 UTC 2021
;; MSG SIZE  rcvd: 55

As you can see it works perfectly in TCP but not in UDP.

We haven’t enabled IPv6 UDP yet. We were actually working on that this week and then we got stuck in firedrill mode. It should be available soon!

1 Like

Any news about UDP on IPv6 :slight_smile:?

1 Like

@kurt I’m also interested in v6 UDP support.

1 Like

Ahhh! We are more behind on this than I thought. I’ll put an update here next week sometime.

Will you add a dtls handler for UDP at some point?

Tell us more about how you’d like it to work and what you’d do with it? It’s not on our radar, but it can be.

Well, dtls is a stream protocol so would think it would work like normal tls termination where it would chunk up the stream and deliver normal udp datagrams to the internal port. But I’m not aware of any other cloud provider doing this, so unsure how exactly it would work. I know that nginx has experimantal support for it.

As for my use-case, it would be for coap/lwm2m services.

That’s pretty interesting. I don’t think we’ll have dtls fast enough to be useful for you, but it’s possible to run the patched nginx or another dtls proxy as an app on Fly. We have been talking about exposing the certificates we manage to people who want to do their own tls but let us issue certificates.

2 Likes

Is UDP on 53 enabled for everyone? I’m unable to bind to fly-global-services:53 (permission denied) – using port 8080 works just fine however.

1 Like

It is! That error is probably because the port process is running as a lower privilege user. If you add USER root near the end of your Dockerfile it should work.

5 Likes

I didn’t see it mentioned in the docs but I think there’s no service port to internal_port rewrite happening when running UDP services. Rewrite happens only on network layer (ingres: destination addres, egress: source address) and transport layer is untouched.

Like everyone here, I’m deploying a DNS application but I prefer to not run it with any additional capabilities (i.e. CAP_NET_BIND_SERVICE), so it listens on fly-global-services:8053

/ # netstat -l | grep fly-global-services
tcp        0      0 fly-global-services:8053 0.0.0.0:*               LISTEN
udp        0      0 fly-global-services:8053 0.0.0.0:*

Looks like requests come with unmodified destination port:

/ # tcpdump -vi eth0 udp and host fly-global-services
tcpdump: listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
02:37:10.980744 IP (tos 0x28, ttl 44, id 14196, offset 0, flags [DF], proto UDP (17), length 61)
    dialup-3-14-15.ftmeade.md.cybercom.mil.44264 > fly-global-services.53: 45919+ A? acmedns.fly.dev. (33)
                                                                       ^^

For now I surrendered and gave the binary CAP_NET_BIND_SERVICE capability but before I accept this as “the way things work on this platform”, I wanted to ask if this is the intended behavior.

Smuggling in another question:
Is it also fair to assume traffic to fly-global-services address is not packet filtered in any way, and it is by design? If you don’t want to run a globally accessible UDP service, don’t bind your UDP service that address.

Thanks for bringing this up!

I verified this is indeed the case; ports are not rewritten.

The services.internal_port and services.ports.port in fly.toml need to match for udp services.

invalid:

[[services]]
  internal_port = 5000 # this will not work; traffic will come in on port 5005
  protocol = "udp"
  [[services.ports]]
    port = 5005

this would work:

[[services]]
  internal_port = 5005 # matches the services.ports.port below
  protocol = "udp"
  [[services.ports]]
    port = 5005

An app deployed with the above config needs to bind to fly-global-services:5005.

1 Like

Unexpected UDP port exposing behaviour / UDP works in port-forwarding mode only by ignoramous · Pull Request #183 · superfly/docs · GitHub :man_shrugging:

1 Like

Any news about UDP in IPv6?

It’s great to give yet another reason for using IPv6 in We are going to start charging for dedicated IPv4 in January 1st - #2 by thewilkybarkid but if you do not offer the same feature parity as IPv4, people are not going to want to use IPv6.

3 Likes