Incoming! UDP Support Arriving Soon

Hello, Flyers!

Just a quick-ish note about some stuff we’ve been working on. Sometime in the coming weeks, we’ll be opening a beta for UDP Anycast services — so, you can take a Docker container that serves authoritative DNS for a zone (or a million zillion zones) and quickly deploy it across the globe.

How this’ll work for users is, you’ll open up your trusty fly.toml and add a service whose protocol is UDP — err, "udp". You’ll set up your app to bind to a special address — fly-global-services. When you deploy, we’ll start routing UDP traffic to your IP address to to your instances, on that fly-global-services address. Things will “just work”: you’ll get the actual source addresses of packets, and your responses will (of course) bear your anycast addresses.

This is one of those features that is super straightforward to describe but was pretty complicated to actually implement. I’ll go into more detail in the future, but the short description is: we use XDP/BPF to relay packets arriving at our edge across our WireGuard mesh to the nearest worker, without them ever seeing userland. We quietly slip proxy headers on and off your packets, and rewrite addresses accordingly. It’s pretty neato.

If anyone’s super interested in playing with this, please let us know! Depending on your tolerance for jank, we might be able to get you started relatively soon. My hope is that for most UDP applications, there’s really not much you have to do to make it work other than plugging the ports into fly.toml.


If you’re interested in DNS examples, let us know what you want to see. We have pi-hole running (kind of silly) and an authoritative nameserver setup in the works with CoreDNS.

Aformentioned pi-hole config:

app = "fli-hole"

  internal_port = 80
  protocol = "tcp"

    hard_limit = 25
    soft_limit = 20

    handlers = []
    port = "80"

    handlers = ["tls"]
    port = "443"

    interval = 10000
    timeout = 2000

  internal_port = 53
  protocol = "udp"

    port = "53"
1 Like

Is UDP on IPv6 implemented?

I’m unable to connect to a UDP port exposed on the load balancer IPv6 address.
And I’m also unable to do any UDP connections inside the VM/container to outside IPv6 addresses.

Here is a test done with dig inside a fly VM/container:

bash-5.0# dig @2001:4860:4860::8888

; <<>> DiG 9.16.6 <<>> @2001:4860:4860::8888
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached

bash-5.0# dig @2001:4860:4860::8888 +tcp

; <<>> DiG 9.16.6 <<>> @2001:4860:4860::8888 +tcp
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 49548
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

; EDNS: version: 0, flags:; udp: 512
;			IN	A


;; Query time: 12 msec
;; SERVER: 2001:4860:4860::8888#53(2001:4860:4860::8888)
;; WHEN: Thu Feb 04 16:35:35 UTC 2021
;; MSG SIZE  rcvd: 55

As you can see it works perfectly in TCP but not in UDP.

We haven’t enabled IPv6 UDP yet. We were actually working on that this week and then we got stuck in firedrill mode. It should be available soon!

1 Like

Any news about UDP on IPv6 :slight_smile:?

@kurt I’m also interested in v6 UDP support.

1 Like

Ahhh! We are more behind on this than I thought. I’ll put an update here next week sometime.

Will you add a dtls handler for UDP at some point?

Tell us more about how you’d like it to work and what you’d do with it? It’s not on our radar, but it can be.

Well, dtls is a stream protocol so would think it would work like normal tls termination where it would chunk up the stream and deliver normal udp datagrams to the internal port. But I’m not aware of any other cloud provider doing this, so unsure how exactly it would work. I know that nginx has experimantal support for it.

As for my use-case, it would be for coap/lwm2m services.

That’s pretty interesting. I don’t think we’ll have dtls fast enough to be useful for you, but it’s possible to run the patched nginx or another dtls proxy as an app on Fly. We have been talking about exposing the certificates we manage to people who want to do their own tls but let us issue certificates.

1 Like

Is UDP on 53 enabled for everyone? I’m unable to bind to fly-global-services:53 (permission denied) – using port 8080 works just fine however.

It is! That error is probably because the port process is running as a lower privilege user. If you add USER root near the end of your Dockerfile it should work.