Just a quick-ish note about some stuff we’ve been working on. Sometime in the coming weeks, we’ll be opening a beta for UDP Anycast services — so, you can take a Docker container that serves authoritative DNS for a zone (or a million zillion zones) and quickly deploy it across the globe.
How this’ll work for users is, you’ll open up your trusty fly.toml and add a service whose protocol is UDP — err, "udp". You’ll set up your app to bind to a special address — fly-global-services. When you deploy, we’ll start routing UDP traffic to your IP address to to your instances, on that fly-global-services address. Things will “just work”: you’ll get the actual source addresses of packets, and your responses will (of course) bear your anycast addresses.
This is one of those features that is super straightforward to describe but was pretty complicated to actually implement. I’ll go into more detail in the future, but the short description is: we use XDP/BPF to relay packets arriving at our edge across our WireGuard mesh to the nearest worker, without them ever seeing userland. We quietly slip proxy headers on and off your packets, and rewrite addresses accordingly. It’s pretty neato.
If anyone’s super interested in playing with this, please let us know! Depending on your tolerance for jank, we might be able to get you started relatively soon. My hope is that for most UDP applications, there’s really not much you have to do to make it work other than plugging the ports into fly.toml.
If you’re interested in DNS examples, let us know what you want to see. We have pi-hole running (kind of silly) and an authoritative nameserver setup in the works with CoreDNS.
I’m unable to connect to a UDP port exposed on the load balancer IPv6 address.
And I’m also unable to do any UDP connections inside the VM/container to outside IPv6 addresses.
Here is a test done with dig inside a fly VM/container:
bash-5.0# dig @2001:4860:4860::8888 google.com
; <<>> DiG 9.16.6 <<>> @2001:4860:4860::8888 google.com
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached
bash-5.0# dig @2001:4860:4860::8888 google.com +tcp
; <<>> DiG 9.16.6 <<>> @2001:4860:4860::8888 google.com +tcp
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 49548
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;google.com. IN A
;; ANSWER SECTION:
google.com. 299 IN A 216.58.215.46
;; Query time: 12 msec
;; SERVER: 2001:4860:4860::8888#53(2001:4860:4860::8888)
;; WHEN: Thu Feb 04 16:35:35 UTC 2021
;; MSG SIZE rcvd: 55
As you can see it works perfectly in TCP but not in UDP.
Well, dtls is a stream protocol so would think it would work like normal tls termination where it would chunk up the stream and deliver normal udp datagrams to the internal port. But I’m not aware of any other cloud provider doing this, so unsure how exactly it would work. I know that nginx has experimantal support for it.
That’s pretty interesting. I don’t think we’ll have dtls fast enough to be useful for you, but it’s possible to run the patched nginx or another dtls proxy as an app on Fly. We have been talking about exposing the certificates we manage to people who want to do their own tls but let us issue certificates.
It is! That error is probably because the port process is running as a lower privilege user. If you add USER root near the end of your Dockerfile it should work.
I didn’t see it mentioned in the docs but I think there’s no service port to internal_port rewrite happening when running UDP services. Rewrite happens only on network layer (ingres: destination addres, egress: source address) and transport layer is untouched.
Like everyone here, I’m deploying a DNS application but I prefer to not run it with any additional capabilities (i.e. CAP_NET_BIND_SERVICE), so it listens on fly-global-services:8053
Looks like requests come with unmodified destination port:
/ # tcpdump -vi eth0 udp and host fly-global-services
tcpdump: listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
02:37:10.980744 IP (tos 0x28, ttl 44, id 14196, offset 0, flags [DF], proto UDP (17), length 61)
dialup-3-14-15.ftmeade.md.cybercom.mil.44264 > fly-global-services.53: 45919+ A? acmedns.fly.dev. (33)
^^
For now I surrendered and gave the binary CAP_NET_BIND_SERVICE capability but before I accept this as “the way things work on this platform”, I wanted to ask if this is the intended behavior.
Smuggling in another question:
Is it also fair to assume traffic to fly-global-services address is not packet filtered in any way, and it is by design? If you don’t want to run a globally accessible UDP service, don’t bind your UDP service that address.
?