Cannot request external services

[aforsbom]

Hello! Sorry if this is covered somewhere, if that’s the case I haven’t been able to find it.

I’ve been struggling to connect to external services from my docker app (node).

More specifically I’m trying to connect to a SMTP server at mailserver.loopia.se but I’m getting the error

getaddrinfo ENOTFOUND ‘mailcluster.loopia.se’

I’ve accessed the container through flyctl ssh console and verified that the container in fact can access this address. Output from nslookup and ping below:

Server:         fdaa::3
Address:        [fdaa::3]:53

Non-authoritative answer:
Name:   mailcluster.loopia.se
Address: 93.188.3.12
Name:   mailcluster.loopia.se
Address: 93.188.3.13
Name:   mailcluster.loopia.se
Address: 93.188.3.14
Name:   mailcluster.loopia.se
Address: 93.188.3.11

Non-authoritative answer:
Name:   mailcluster.loopia.se
Address: 2a02:250:0:48::14
Name:   mailcluster.loopia.se
Address: 2a02:250:0:48::13
Name:   mailcluster.loopia.se
Address: 2a02:250:0:48::12
Name:   mailcluster.loopia.se
Address: 2a02:250:0:48::11

PING mailcluster.loopia.se (2a02:250:0:48::12): 56 data bytes
64 bytes from 2a02:250:0:48::12: seq=0 ttl=53 time=24.167 ms
64 bytes from 2a02:250:0:48::12: seq=1 ttl=53 time=24.719 ms
64 bytes from 2a02:250:0:48::12: seq=2 ttl=53 time=24.322 ms
64 bytes from 2a02:250:0:48::12: seq=3 ttl=53 time=24.427 ms

But the app will not resolve the address for some reason.
Running out of ideas. Any help would be greatly appreciated

[greg]

Hey,

Are you using a Dockerfile based app? If so, does your base image use Alpine Linux? Only if so, I’ve previously had similar weird errors with DNS when running a Node app. Couldn’t figure it out either! What fixed it for me was switching to a different base image, such as XX-slim. And then the DNS errors went away. Total guess but can’t hurt to try.

e.g

https://christoph.luppri.ch/fixing-dns-resolution-for-ruby-on-alpine-linux

[aforsbom]

Hey and thanks for the tip, greg!

For your question, yes and yes. Both Dockerfile-based and Alpine Linux.

So based on your info, I just switched to bullseye-slim and tried again, but no change unfortunately. The container itself seems to know how to resolve the address, but when I try to resolve it from the app, it doesn’t.

Running the container locally works fine, and it connects to the SMTP properly and sends the e-mail I want. Only when I deploy it and run on fly I get this odd DNS-related error.

It seems like the app is not able to resolve anything outside the private network, I can connect to my other machines on the private network through the application running in the container. It only seems to apply to external services.

E.g. if I do a request towards example-db.internal that works fine, if I request google.com I get ENOTFOUND.

[greg]

Strange! When I switched from Alpine to [another Linux] that fixed my DNS errors with Node. So perhaps there is something else going on … maybe with the host machine itself

Since Fly doesn’t charge for registry storage space, the size of the image is less of a concern (within reason) so maybe try a full Ubuntu/Debian as your base image. Shouldn’t be needed but worth a try while waiting for someone from Fly to debug. That would definitely rule out it being the base image.

I know Fly use Google’s DNS for resolving external domains and so maybe it’s related to that? Again, no idea why it would be. But that was from my fun with DNS a while back:

EAI_AGAIN error (DNS resolution)

The /etc/resolv.conf inside instances is set to our private network DNS server. We need to go through it to resolve private DNS queries without further configuration. If the request is for something outside the private network then it is proxied to Google DNS servers. As a matter of fact, there’s this NetActuate System Status from one of our service provider. I don’t think it should be causing EAI_AGAIN or anything more than increased latency though.

[aforsbom]

Thanks. I’ll have a read through

[aforsbom]

I switched to an image based on full debian buster. This seems to have resolved it, unless something was changed elsewhere that I’m not aware of.

Thanks for the help, I will keep it like this for now and try with a smaller image again later.

[jerome]

@greg

I know Fly use Google’s DNS for resolving external domains and so maybe it’s related to that? Again, no idea why it would be. But that was from my fun with DNS a while back:

FYI: We have since switched to recursively resolving from root DNS servers.

[ignoramous]

@jerome

FYI: We have since switched to recursively resolving from root DNS servers.

OK. Is DNSSEC validation supported?

[bcomnes]

Has anyone managed to figure out a solution to getting requests to ipv6 AAAA records working without switching away from alpine?

[Raboo]

This might be related resolv.conf should be readable by non-root users.