DNS Resolution Failure for Supabase Database Connection from Fly.io

Phoenix
,
1

Our Phoenix application deployed on Fly.io cannot connect to our Supabase
database due to DNS resolution failures, despite the same connection working
perfectly from local machines.

Technical Details:

What Works:

  • :white_check_mark: Local connection via psql using same credentials
  • :white_check_mark: Local DNS resolution: nslookup aws-1-eu-north-1.pooler.supabase.com
    returns valid AWS ELB IPs
  • :white_check_mark: Fly.io secrets are set correctly (verified via flyctl secrets list)

What Fails:

  • :cross_mark: Connection from Fly.io infrastructure gets :nxdomain
  • :cross_mark: Even using direct IP addresses (51.21.18.29, 13.60.102.132) fails with
    same :nxdomain error

Troubleshooting Attempted:

  1. Verified DNS resolution works locally
  2. Tried direct IP addresses instead of domain names
  3. Confirmed secrets are properly set in Fly.io
  4. Added app restarts after setting secrets
  5. Tested multiple Supabase connection formats

Anybody any idea? If not, it was a nice try but after six hours I am sick of such a mess.

2

I wonder if it is considering the port (:5432) part of the domain name.

Getting an :nxdomain even when using the IP directly suggests it’s still trying to use DNS to resolve. nslookup aws-1-eu-north-1.pooler.supabase.com:5432 and nslookup 51.21.18.29:5432 will both give an nxdomain error.

3

Hi Kate,

thanks for your response! You are right, the port is not part of the domain name. The point is that the domain name resolves on my laptop perfectly fine. I can connect from my laptop via psql to the database. But the application on fly.io does not - this is completely out of my control. Or at least, I have no idea how to fix it.

4

This is on a Fly machine:

root@683d56ea106458[yyz]:/app# dig +short aws-1-eu-north-1.pooler.supabase.com
pool-tcp-eun11-6f207a5-0e460494991afd40.elb.eu-north-1.amazonaws.com.
51.21.18.29
13.60.102.132

This suggests DNS resolution is working on Fly machines. You can test it by fly ssh console and then running the same command I ran.

If then the app doesn’t resolve, as you showed, it’s indeed very likely that it’s an app-level configuration issue.

on my laptop perfectly fine. I can connect from my laptop via psql to the database.

So: on Fly.io you have problems with a Phoenix application, but your laptop works with psql - apples and oranges situation!

  1. Try connecting with psql from a Fly machine. This will work.
  2. Try running your Phoenix app on your laptop connecting to Supabase. Does this work?

Let us know what you find!

6

Hey, thank you for pointing me to fly ssh- didn’t know that yet, fly.io rookie :slight_smile:

Summary
The deployment has been failing with database connection errors despite:

  1. Network connectivity working (TCP port 5432 accessible)
  2. Database credentials working (manual `psql` connection succeeds)
  3. Phoenix application failing with DNS resolution errors for both hostnames and IP addresses

Root Cause - IPv4/IPv6 Networking Mismatch in Fly.io Infrastructure**

  1. Fly.io Uses IPv6-first networking by default
  2. Supabase Database IP addresses are IPv4 (`13.60.102.132`, `51.21.18.29`)
  3. Erlang VM On IPv6-first infrastructure, tries IPv6 for IPv4 addresses
  4. Result DNS resolution errors (`:nxdomain`) even for IP addresses

Evidence

# SSH session (system networking - works)
$ psql -h 51.21.18.29 -U postgres.xmdjheppafplzzbwrskn -d postgres
Connected successfully
# Phoenix app (Erlang VM networking - fails)
tcp connect (51.21.18.29:5432): non-existing domain - :nxdomain

Thanks,
Hannes

9

Hi!

tl;dr find where your configuration sets ECTO_IPV6 (fly.toml or env.sh.eex), and try unsetting ECTO_IPv6, I think that should get you correct IPv4-only resolution to your Supabase pooler.

Read on for more explanation :smiley:

  1. Fly.io Uses IPv6-first networking by default

This is not entirely accurate. IPv4 addresses resolve and work fine - asking for ipv6 addresses only, is an application-side behavior. It’s true that this tends to be required for Elixir applications on Fly, and fly launch will configure your application for this, as the most likely scenario is you wanting your machine to be able to talk to other Fly machines over IPv6 and without this configuration, Erlang/Elixir defaults to ipv4 only (which we have also seen and results in exactly the opposite issue: hostnames within the Fly network which always resolve to ipv6 names, result in :nxdomain.

  1. Erlang VM On IPv6-first infrastructure, tries IPv6 for IPv4 addresses

Similar to the above - this is an explicit config at the Elixir/Erlang level when calling name-resolution functions (specifically, :inet.gethostbyname), not a platform particularity. The way the LLM phrased it makes it sound like the BEAM auto-detects the type of network and does the wrong thing, which is not correct.

Here’s an example of using gethostbyname with and without :inet6:

iex(3)> :inet.gethostbyname('aws-1-eu-north-1.pooler.supabase.com')
{:ok,
 {:hostent,
  'pool-tcp-eun11-6f207a5-0e460494991afd40.elb.eu-north-1.amazonaws.com',
  ['aws-1-eu-north-1.pooler.supabase.com'], :inet, 4,
  [{13, 60, 102, 132}, {51, 21, 18, 29}]}}
iex(4)> :inet.gethostbyname('aws-1-eu-north-1.pooler.supabase.com', :inet6)
{:error, :nxdomain}

The component ultimately calling inet.gethostbyname is the Postgrex driver used by Ecto. If you used the recommended configuration for a Phoenix app on Fly, you likely have ECTO_IPV6 set somewhere (fly.toml or env.sh.eex), as needed to talk to Postgres running on Fly machines. If you don’t need Ecto/Postgrex to talk to any ipv6 databases, you can try unsetting ECTO_IPv6, I think that should get you correct IPv4-only resolution to your Supabase pooler.

Give this a try!

  • Daniel
3 Likes
10

Daniel, many thanks for your help, all is good and I learned a lot! Happy that I can use Supabase and Fly.io together :slight_smile: Have a great weekend!

11

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.