Getting error querying Fly hosted DNS server from another Fly app

1

I’m running an authoritative DNS server in Fly on IP 213.188.197.200. Querying it from outside Fly works (e.g., dig wombatsoftware.com @213.188.197.200).

However, querying the same server from within another Fly app fails with the following error: read udp 172.19.3.226:50977->213.188.197.200:53: read: connection refused. Repeating the same query from the same app to an external recursive DNS server (e.g., 1.1.1.1) succeeds.

Any ideas? FWIW I’m using the Go github.com/miekg/dns DNS library to do the DNS queries.

2

UDP needs a bit of special setup to get right, could you confirm that you’re binding to fly-global-services as described here? Incoming! UDP Support Arriving Soon

There’s also another discussion here on getting UDP to work DNS over TCP works, but UDP doesn't

If you can share the app name I’ll ask someone on the dev team to take a look, but unless the problem is intermittent we should be able to fix it with the correct configuration.

3

Yes, I know about fly-global-services. I’m successfully using it in my DNS server.

However the other Fly app is only making DNS client queries. Querying external DNS servers over UDP works fine, but querying my DNS server running in Fly doesn’t work. So it seems like UDP packets sent exclusively within Fly’s network are getting dropped?

4

Thanks for confirming. I don’t think this is happening intentionally, we do support general internal communication, including DNS over UDP. Will ask one of the specialists on the team to take a look.

5

Are you using the public hostname or private hostname to query it internally? You might need to use <appname>.internal to use it from within Fly, and also make sure it’s listening on ::.

6

I’m using the DNS server’s public IP address to query it. Note that I’m querying it from a separate Fly app so the query is “internal” only in the sense that both apps are Fly hosted.

7

I just checked and using the public IP for UDP from other Fly apps won’t currently work. The way we route public IPs internally means you won’t hit the servers that do all the public UDP work.

This is something we need to fix, but I don’t think we can do it quickly.

Is it possible to use a different resolver internally? You can use UDP over the private network, you’d just need to do the equivalent of dig wombatsoftware.com @<app>.internal.

8

It turns out, this might be a quicker fix than I thought. We may have a fix in the next few days. Or it might turn out to be difficult.

9

Thanks for the clarification. Seems like I’ve triggered a yet another UDP edge condition. :wink:

I can’t really use a different resolver since the querying app is intended to be an uptime checking app for the general public. Oh well, it was just a quick experiment on my part and it’s easy enough for me to deploy the checking agents outside of Fly, so don’t bother about fixing this issue for my use case.

Quite apart from the above issue, I’ve been spoiled by how easy Fly makes it to globally deploy and update apps.

10

Oh we want to fix the issue. It’s actually super valuable you found this!

11

If the issue can be fixed that would be great.

12

@jbarham what are you using to host your own DNS, bind or something else? :eyes: