Hello Everybody recently I have been trying out fly.io because of the newly announced UDP feature.
I was trying to setup a DNS Server using the Knot Resolver Docker Image cznic/knot-resolver. When listening to 0.0.0.0 i get the following Error on my DNS Client: reply from unexpected source: 185.178.200.138#53, expected 213.188.193.229#53. Is this intended behaviour and a issue on my side or just an error?
That is not intended! That’s an IP from one of our edge nodes, seems like we’re not sending the packet with the right source IP. @thomas will know more.
Hey there! I’m just digging into this right now. Can you share (privately, if you like) the fly.toml you’re using for this service? I’m looking and I don’t see a UDP port registered for it. You’re looking for something like:
[[services]]
internal_port = 53
protocol = "udp"
[[services.ports]]
port = "53"I was testing out a different service in the meantime thats why I removed the udp part in the config. I have now reverted it back to what I had in the first place. To get the error you will need a machine without a nat because the packages will never arrive from the different ip because they are not on the nat table.
`root@ubuntu-s-1vcpu-1gb-fra1-01:~# dig @213.188.193.229 test.com
;; reply from unexpected source: 185.178.200.130#53, expected 213.188.193.229#53
;; reply from unexpected source: 185.178.200.130#53, expected 213.188.193.229#53
;; reply from unexpected source: 185.178.200.130#53, expected 213.188.193.229#53
; <<>> DiG 9.16.1-Ubuntu <<>> @213.188.193.229 test.com
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached`
Thanks! I can tell you what’s happening, but not yet why, without knowing more about your application.
Here’s what I’m seeing:
I send a DNS packet from AWS to 213.188.193.229, the anycast for your application. Our forwarding code translates that IP to 172.19.0.139, the “fly-global-services” address for your instance:
3.237.184.8.38133 > 172.19.0.139.53
And your application responds:
172.19.0.138.53 > 3.237.184.8.38133
Note that the response comes from the instance base address, .138, not fly-global-services, .139, the original destination address.
Our forwarding code maps that .138 IP address differently (you don’t want random DNS requests your own instance makes getting mapped to your anycast address and routed weirdly); that problem is why we have a fly-global-services address for this stuff. But note also that even if we weren’t special-casing that address, the DNS response your app generated still wouldn’t work, because it’s responding to a DNS request to .139 from .138.
(It’s early in the morning and I reserve the right to be wrong about this!)
Got a link to the software you’re running? I’m happy to try to boot it up as an app myself and get it working.
Here is the code to reproduce the problem:
Dockerfile
FROM cznic/knot-resolver
COPY kresd.conf /etc/knot-resolver/kresd.conf
kresd.conf
modules = { 'policy' }
net.listen('0.0.0.0', 53, { kind = 'dns', freebind = true })
-- I have also tried the following settings:
-- net.listen('172.19.0.139', 53, { kind = 'dns', freebind = true })
-- net.listen('127.0.0.1', 53, { kind = 'dns', freebind = true })
-- net.listen(net.interfaces()["eth0"]["addr"][1], 53, { kind = 'dns', freebind = true })
-- I thought this would be most similiar to the PI Hole example
-- net.listen(net.eth0, 53, { kind = 'dns', freebind = true })
policy.add(policy.all(policy.TLS_FORWARD({
{ '1.1.1.1', hostname='cloudflare-dns.com', ca_file='/etc/ssl/certs/ca-certificates.crt' },
})))
fly.toml
[[services]]
internal_port = 53
protocol = "udp"
[[services.ports]]
port = "53"
1 Like
It seems like the listen is ok, it’s just sending responses back from the wrong IP. That is surprising!
We’ll figure out what it takes to get it working. If you have a clever way to send from the fly-global-services IP that might fix you up. That IP is dynamic, but the hostname always points to the right one.
Just a quick update that this is a known problem with Knot (it’s solveable, still tinkering).
To allow a DNS server to listen on 0.0.0.0 (or any other ambiguous address I guess) and respond to packets appropriately, there’s an IP_PKTINFO sockopt that explicitly tells you the address to which a packet was delivered, which you can then use to send the reply. They’ve been debating adding that to Knot for like a year now (the feature is I think 15 years old).
Good news, everyone!
The problem here is super simple, but the Knot build process makes it very annoying to fix.
Knot won’t resolve hostnames in its config file (lame). So, we take the knot-resolver image and replace its ENTRYPOINT with one that sed-edits the config file before running kresd. Here’s my run-knot.sh, referenced in the Dockerfile here: https://gist.github.com/tqbf/9a551848ba5b6d0ced847df6fb3985b6
#!/bin/sh
ip=$(grep fly-global-services /etc/hosts | cut -d ' ' -f 1)
sed "s/fly-global-services/$ip/g" < /etc/knot-resolver/kresd.conf > /tmp/kresd.conf
mv /tmp/kresd.conf /etc/knot-resolver/kresd.conf
/usr/sbin/kresd -c /etc/knot-resolver/kresd.conf
And then the line in kresd.conf is just:
net.listen('fly-global-services', 53, { kind = 'dns', freebind = true })
WIth that, after deploying, I can:
> $ dig www.google.com @213.188.193.197
;; ANSWER SECTION:
www.google.com. 171 IN A 172.217.4.100
It is super annoying that we had to re-roll the Dockerfile to get this far! We’re thinking about what we can do with the image builders to work around this problem, which will surely bite us in the butts frequently in the months to come.
I appreciate you trying to get Knot working! Throw us more problems!
2 Likes
Thanks alot this worked! Luckily knot is not really what im trying to use, it was just to show the error without introducing the posibility that my code is the problem. My configuration hapily accepts this dns = "fly-global-services:53" and will resolve the ip without a problem. Really glad to see such a responsive support, keep it up.
3 Likes