# DNS UDP responses getting lost in Europe **URL:** \ **Category:** General\ **Created:** [March 7, 2021, 10:23pm UTC](https://community.fly.io/t/dns-udp-responses-getting-lost-in-europe/890 "2021-03-07T22:23:38Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![jbarham](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/jbarham/32/352_2.png) [@jbarham](https://community.fly.io/u/jbarham)\ **Post date:** [March 7, 2021, 10:23pm UTC](https://community.fly.io/t/dns-udp-responses-getting-lost-in-europe/890/1 "2021-03-07T22:23:38Z")
Previously I was seeing [DNS UDP responses being lost](https://community.fly.io/t/london-lhr-region-network-issues/822) between my LHR DNS server and London based monitoring agents. That issue was resolved mid last week. Unfortunately since the 53/tcp port was enabled I’ve been seeing DNS UDP responses from my DNS server running in Fly’s European regions (currently LHR and FRA) being lost when tested by Europe based Pingdom monitoring agents. Note that this is _not_ a London issue, I’m seeing responses frequently not getting through to Pingdom’s monitoring agents in, at least, Zurich, and Falkenberg and Stockholm in Sweden. Strangely though the responses from the same servers get through to Pingdom’s North American monitoring agents 100% of the time. I’m almost certain this is not a Pingdom issue as I have an identical Pingdom monitor checking the same DNS server running in Linode’s London DC and it’s getting 100% uptime. To summarize: - DNS UDP responses from my DNS server running in Fly’s LHR and FRA regions are frequently being lost when queried by Pingdom’s Europe based agents - Responses from the same servers are always getting through to Pingdom’s North American located agents - Responses from the same DNS server software running in Linode’s London DC always get through to Pingdom’s Europe based agents - The start of these failures seems to coincide with the enabling of the 53/tcp port My app name is `slickdns`. You can check my Fly DNS server by running `dig @213.188.193.79 wombatsoftware.com`. ---
**Author:** ![thomas](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/thomas/32/40188_2.png) [@thomas](https://community.fly.io/u/thomas)\ **Post date:** [March 7, 2021, 10:44pm UTC](https://community.fly.io/t/dns-udp-responses-getting-lost-in-europe/890/2 "2021-03-07T22:44:55Z")
I’m digging in on this right now but a quick note that 53/tcp probably couldn’t have impacted 53/udp; they take entirely different data paths (53/tcp has an ordinary socket listener — though we do a port remapping for it — and runs from haproxy into `fly-proxy`, while 53/udp is intercepted by XDP in the kernel, and `fly-proxy` (+ anything else in userland) never sees it.) If there’s not some really dumb misconfiguration we can quickly spot here, my next step is just going to be to replicate Pingdom off-network and continuously monitor 53/udp globally. ---
**Author:** ![thomas](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/thomas/32/40188_2.png) [@thomas](https://community.fly.io/u/thomas)\ **Post date:** [March 8, 2021, 12:20am UTC](https://community.fly.io/t/dns-udp-responses-getting-lost-in-europe/890/3 "2021-03-08T00:20:59Z")
My useless update is that I didn’t see anything obviously misconfigured, I can do eu-\>eu DNS, and we need the monitoring anyways, so I’m standing up something to monitor multi-region DNS into Fly continuously ([github.com/256dpi/newdns](http://github.com/256dpi/newdns) is a strange library, but the thing we needed serverside here sure was its “hello world”). I’ll keep you updated. ---
**Author:** ![jbarham](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/jbarham/32/352_2.png) [@jbarham](https://community.fly.io/u/jbarham)\ **Post date:** [March 8, 2021, 1:00am UTC](https://community.fly.io/t/dns-udp-responses-getting-lost-in-europe/890/4 "2021-03-08T01:00:48Z")
Thanks for the update. So it seems that enabling 53/tcp likely isn’t the culprit. I just mentioned it because I started seeing UDP failures again after I enabled 53/tcp for my server. I’ve disabled 53/tcp for my DNS (i.e., it’s back to UDP only). I’ll report if that makes a difference. ---
**Author:** ![thomas](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/thomas/32/40188_2.png) [@thomas](https://community.fly.io/u/thomas)\ **Post date:** [March 10, 2021, 7:43pm UTC](https://community.fly.io/t/dns-udp-responses-getting-lost-in-europe/890/5 "2021-03-10T19:43:05Z")
Hey, sorry about the delayed response on this, but here’s where we’re at: - We identified an issue that impacts a small subset of our hosts (it’s a dumb iptables configuration thing having to do with how the interfaces are configured). We’d thought we’d fixed it fleetwide, but nope. - The problem was impacting a `fra` edge node, which .EU requests frequently get routed to. - Our provisioning tool now checks for and fixes the problem, fleetwide. - I’ve stood up an off-network monitoring system that hits a small cluster of Fly DNS apps from a bunch of locations around the world (basically: most of Vultr’s regions); we’ve got alerts set on DNS (and thus UDP) error rates worldwide now. Sorry this took so long to resolve! In retrospect I could have wrapped this up faster if I’d started by auditing all our iptables rules. ---
**Author:** ![jbarham](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/jbarham/32/352_2.png) [@jbarham](https://community.fly.io/u/jbarham)\ **Post date:** [March 11, 2021, 9:53pm UTC](https://community.fly.io/t/dns-udp-responses-getting-lost-in-europe/890/6 "2021-03-11T21:53:37Z")
Thanks again for the update. Unfortunately I’m not seeing any improvement at my end (after a restart FWIW), i.e., UDP DNS responses from my DNS server are frequently not getting through to Pindom’s EU based probes whereas the same DNS server running on a VM in Linode’s London DC is at 100% uptime. Don’t know if it makes any difference but I noticed that even though I’ve set LHR and FRA as my app’s regions, the instances are running in LHR and CDG. ---
**Author:** ![jbarham](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/jbarham/32/352_2.png) [@jbarham](https://community.fly.io/u/jbarham)\ **Post date:** [March 19, 2021, 10:00am UTC](https://community.fly.io/t/dns-udp-responses-getting-lost-in-europe/890/7 "2021-03-19T10:00:16Z")
I’ve been getting 100% uptime in Europe and North America with my Fly DNS servers over the past 24 hours. Looks like the fix was to enable tcp/53 and to enable a tcp health check for the same port. So seems like Fly was deeming my app to be unhealthy and dropping UDP responses only when I was running on udp/53 with no health check configured. ---
**Author:** ![kurt](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/kurt/32/43757_2.png) [@kurt](https://community.fly.io/u/kurt)\ **Post date:** [March 19, 2021, 3:39pm UTC](https://community.fly.io/t/dns-udp-responses-getting-lost-in-europe/890/8 "2021-03-19T15:39:11Z")
What! That shouldn’t have had any effect. It is super interesting to know about that, and gives us something else to investigate. ---
**Author:** ![jbarham](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/jbarham/32/352_2.png) [@jbarham](https://community.fly.io/u/jbarham)\ **Post date:** [March 20, 2021, 10:45am UTC](https://community.fly.io/t/dns-udp-responses-getting-lost-in-europe/890/9 "2021-03-20T10:45:05Z")
Hmm, now I’m stumped as to what it is I did that fixed the UDP responses being dropped. Today I tested removing the tcp check from `fly.toml`, got no errors, then removed the tcp service (i.e., back to udp/53 only) and got no errors. So now it’s back to udp/53 and tcp/53 with the tcp health check enabled and it’s all green. FWIW I made the tests above by only changing the app’s `fly.toml` and left the `Dockerfile` untouched. I don’t know if that makes any difference in terms of re-deployment. ---
**Author:** ![kurt](https://sea1.discourse-cdn.com/fly1/user_avatar/community.fly.io/kurt/32/43757_2.png) [@kurt](https://community.fly.io/u/kurt)\ **Post date:** [March 20, 2021, 7:28pm UTC](https://community.fly.io/t/dns-udp-responses-getting-lost-in-europe/890/10 "2021-03-20T19:28:17Z")
That shouldn’t make any difference! Let us know if you see UDP responses get dropped again. There’s a very good chance that just replacing all your instances is what made it start working, sometimes our fixes require app “reboots” to take effect.