Postgres :nxdomain errors

We have a brand new Phoenix app which we ran fly launch on and got the Dockerfile, fly.toml and other config added.

When we run fly deploy --remote-only or trigger a deploy from GitHub Actions during either the migration predeploy phase or after during application boot we are getting

2022-03-30T23:01:23Z   [info]23:01:23.835 [error] Postgrex.Protocol (#PID<0.2364.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (top2.nearest.of.upsell-db.internal:5432): non-existing domain - :nxdomain

errors. But if we try a few more times eventually both the migration and the app connect and the deploy works.

Unrelated to the Postgres nxdomain errors we are also periodically getting the following during the build phase in the GitHub Actions

WARN Remote builder did not start in time. Check remote builder logs with `flyctl logs -a fly-builder-divine-pond-1402`
12

Error failed to fetch an image or build from source: error connecting to docker: remote builder app unavailable

This appears to be a similar error to Deployment fails here and there - #23 by sudhir.j

Destroyed app and created a band new one and that seems to have solved this issue :\

This likely means the Postgres didn’t get registered in DNS properly. We had some issues yesterday in Seattle that caused this in a few places. Thank you for trying again!

It was nice to get it resolved, but it did consume a bunch of time trying to diagnose the issue. Would be nicer if this time of outage/errors were correctly reflected on the status page. At the time of these problems it was reporting everything was working correctly in SEA.

We posted a status page incident for last week’s SEA connectivity problems as soon as we identified the issue (and resolved it shortly after): Fly.io Status - Connectivity failures in the Seattle region

It would indeed be nicer to have a status page that updates itself before we’ve even identified an issue, but for now it’s just managed by us humans doing our best :wave:

@wjordan,
I think I probably missed that one! Thanks for the hard work on updating those!

2 Likes