Connected Postgres DB Throws Domain not found error

I created a new organisation to hold my production database and app so I don’t mess it up accidentally. I recreated the database cluster and the app in the new organisation. But when I created and pushed to the app, it worked for some time after the deploy, but then it started throwing 500 errors. Checking logs throws this message indefinitely

2021-10-02T17:59:57.115035644Z app[8cebf516] ewr [info] 17:59:57.114 [error] Postgrex.Protocol (#PID<0.2248.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (indiepaper-prod-db.internal:5432): non-existing domain - :nxdomain

Both in indiepaper-prod-db and indiepaper-prod are in the same region and in the same organisation.

Checking the graphs the indiepaper-prod-db shows the connection first climbing to 12 and then dropping down to 2 after some time.

I don’t have the SSL certificate generated for indiepaper.me, I think I have exceeded the maximum certificated issued for this week. Would that be the issue ?

Creating a new indiepaper-prod-db and attaching to indiepaper-prod does not resolve the issue.

I have tried creating both of them in the old IndiePaper organisation and they both failed too.

The issue crops up when I issue a certificate for indiepaper.me and then deploy, That might be due to hitting letsencrypt limits.

The fly domain http://indiepaper-prod.fly.dev/ does not have that issue and works fine.

Removing certificates and redeploy fixes this issue.

Seems like this might not be the issue, it fixed itself on the production server and is now happening in my staging server intermittently.

Seems to be fixed by Failing Postgres DB connection from app here and there - #22 by kurt. I restarted my db clusters, pushed a new deploy and the issue hasn’t come up.

Nop the issue seems to be reoccurring.

2021-10-04T12:28:21.503531366Z app[839f9c65] ewr [info] 12:28:21.501 [error] #PID<0.2480.0> running IndiePaperWeb.Endpoint (connection #PID<0.2479.0>, stream id 1) terminated
2021-10-04T12:28:21.503546515Z app[839f9c65] ewr [info] Server: indiepaper.me:80 (http)
2021-10-04T12:28:21.503550142Z app[839f9c65] ewr [info] Request: GET /
2021-10-04T12:28:21.503551955Z app[839f9c65] ewr [info] ** (exit) an exception was raised:
2021-10-04T12:28:21.503555362Z app[839f9c65] ewr [info]     ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 2174ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
2021-10-04T12:28:21.503559750Z app[839f9c65] ewr [info]   1. Ensuring your database is available and that you can connect to it
2021-10-04T12:28:21.503561994Z app[839f9c65] ewr [info]   2. Tracking down slow queries and making sure they are running fast enough
2021-10-04T12:28:21.503564279Z app[839f9c65] ewr [info]   3. Increasing the pool_size (albeit it increases resource consumption)
2021-10-04T12:28:21.503566243Z app[839f9c65] ewr [info]   4. Allowing requests to wait longer by increasing :queue_target and :queue_interval
2021-10-04T12:28:21.503568357Z app[839f9c65] ewr [info] See DBConnection.start_link/2 for more information
2021-10-04T12:28:21.503570441Z app[839f9c65] ewr [info]         (ecto_sql 3.7.0) lib/ecto/adapters/sql.ex:756: Ecto.Adapters.SQL.raise_sql_call_error/1
2021-10-04T12:28:21.503572695Z app[839f9c65] ewr [info]         (ecto_sql 3.7.0) lib/ecto/adapters/sql.ex:689: Ecto.Adapters.SQL.execute/5
2021-10-04T12:28:21.503574338Z app[839f9c65] ewr [info]         (ecto 3.7.1) lib/ecto/repo/queryable.ex:219: Ecto.Repo.Queryable.execute/4
2021-10-04T12:28:21.503576542Z app[839f9c65] ewr [info]         (ecto 3.7.1) lib/ecto/repo/queryable.ex:19: Ecto.Repo.Queryable.all/3
2021-10-04T12:28:21.503578085Z app[839f9c65] ewr [info]         (ecto 3.7.1) lib/ecto/repo/queryable.ex:146: Ecto.Repo.Queryable.one/3
2021-10-04T12:28:21.503579468Z app[839f9c65] ewr [info]         (indie_paper 0.1.0) lib/indie_paper_web/controllers/author_auth.ex:93: IndiePaperWeb.AuthorAuth.fetch_current_author/2
2021-10-04T12:28:21.503581632Z app[839f9c65] ewr [info]         (indie_paper 0.1.0) IndiePaperWeb.Router.browser/2
2021-10-04T12:28:21.503587002Z app[839f9c65] ewr [info]         (indie_paper 0.1.0) lib/indie_paper_web/router.ex:1: IndiePaperWeb.Router.__pipe_through5__/1
2021-10-04T12:28:23.093288578Z app[839f9c65] ewr [info] 12:28:23.092 [error] Postgrex.Protocol (#PID<0.2263.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (indiepaper-prod-db.internal:5432): non-existing domain - :nxdomain
2021-10-04T12:28:24.470066503Z app[839f9c65] ewr [info] 12:28:24.469 [error] Postgrex.Protocol (#PID<0.2255.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (indiepaper-prod-db.internal:5432): non-existing domain - :nxdomain
2021-10-04T12:28:26.221199071Z app[839f9c65] ewr [info] 12:28:26.220 [error] Postgrex.Protocol (#PID<0.2260.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (indiepaper-prod-db.internal:5432): non-existing domain - :nxdomain
2021-10-04T12:28:29.462776120Z app[839f9c65] ewr [info] 12:28:29.462 [error] Postgrex.Protocol (#PID<0.2266.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (indiepaper-prod-db.internal:5432): non-existing domain - :nxdomain
2021-10-04T12:28:30.321793685Z app[839f9c65] ewr [info] 12:28:30.321 [error] Postgrex.Protocol (#PID<0.2259.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (indiepaper-prod-db.internal:5432): non-existing domain - :nxdomain
2021-10-04T12:28:31.338981500Z app[839f9c65] ewr [info] 12:28:31.338 [error] Postgrex.Protocol (#PID<0.2264.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (indiepaper-prod-db.internal:5432): non-existing domain - :nxdomain
2021-10-04T12:28:32.764947179Z app[839f9c65] ewr [info] 12:28:32.764 [error] Postgrex.Protocol (#PID<0.2256.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (indiepaper-prod-db.internal:5432): non-existing domain - :nxdomain
2021-10-04T12:28:36.226013318Z app[839f9c65] ewr [info] 12:28:36.225 [error] Postgrex.Protocol (#PID<0.2260.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (indiepaper-prod-db.internal:5432): non-existing domain - :nxdomain
2021-10-04T12:28:39.695107919Z app[839f9c65] ewr [info] 12:28:39.694 [error] Postgrex.Protocol (#PID<0.2263.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (indiepaper-prod-db.internal:5432): non-existing domain - :nxdomain
2021-10-04T12:28:40.330248558Z app[839f9c65] ewr [info] 12:28:40.329 [error] Postgrex.Protocol (#PID<0.2262.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (indiepaper-prod-db.internal:5432): non-existing domain - :nxdomain
2021-10-04T12:28:42.131174493Z app[839f9c65] ewr [info] 12:28:42.130 [error] Postgrex.Protocol (#PID<0.2255.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (indiepaper-prod-db.internal:5432): non-existing domain - :nxdomain
2021-10-04T12:28:42.529294353Z app[839f9c65] ewr [info] 12:28:42.528 [error] Postgrex.Protocol (#PID<0.2261.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (indiepaper-prod-db.internal:5432): non-existing domain - :nxdomain
2021-10-04T12:28:42.921039276Z app[839f9c65] ewr [info] 12:28:42.920 [error] Postgrex.Protocol (#PID<0.2264.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (indiepaper-prod-db.internal:5432): non-existing domain - :nxdomain
2021-10-04T12:28:46.944843521Z app[839f9c65] ewr [info] 12:28:46.944 [error] Postgrex.Protocol (#PID<0.2265.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (indiepaper-prod-db.internal:5432): non-existing domain - :nxdomain
2021-10-04T12:28:50.380985809Z app[839f9c65] ewr [info] 12:28:50.380 [error] Postgrex.Protocol (#PID<0.2263.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (indiepaper-prod-db.internal:5432): non-existing domain - :nxdomain
2021-10-04T12:28:53.195142012Z app[839f9c65] ewr [info] 12:28:53.194 [error] Postgrex.Protocol (#PID<0.2255.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (indiepaper-prod-db.internal:5432): non-existing domain - :nxdomain
2021-10-04T12:28:53.302069849Z app[839f9c65] ewr [info] 12:28:53.301 [error] Postgrex.Protocol (#PID<0.2259.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (indiepaper-prod-db.internal:5432): non-existing domain - :nxdomain
2021-10-04T12:28:54.251286170Z app[839f9c65] ewr [info] 12:28:54.250 [error] Postgrex.Protocol (#PID<0.2266.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (indiepaper-prod-db.internal:5432): non-existing domain - :nxdomain
2021-10-04T12:28:55.349158582Z app[839f9c65] ewr [info] 12:28:55.348 [error] Postgrex.Protocol (#PID<0.2261.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (indiepaper-prod-db.internal:5432): non-existing domain - :nxdomain
2021-10-04T12:28:55.828119227Z app[839f9c65] ewr [info] 12:28:55.827 [error] Postgrex.Protocol (#PID<0.2260.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (indiepaper-prod-db.internal:5432): non-existing domain - :nxdomain
2021-10-04T12:28:57.046512855Z app[839f9c65] ewr [info] 12:28:57.045 [error] Postgrex.Protocol (#PID<0.2256.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (indiepaper-prod-db.internal:5432): non-existing domain - :nxdomain
2021-10-04T12:29:01.538751420Z app[839f9c65] ewr [info] 12:29:01.538 [error] Postgrex.Protocol (#PID<0.2262.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (indiepaper-prod-db.internal:5432): non-existing domain - :nxdomain
2021-10-04T12:29:02.532055124Z app[839f9c65] ewr [info] 12:29:02.531 [error] Postgrex.Protocol (#PID<0.2264.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (indiepaper-prod-db.internal:5432): non-existing domain - :nxdomain
2021-10-04T12:29:02.591834896Z app[839f9c65] ewr [info] 12:29:02.591 [error] Postgrex.Protocol (#PID<0.2265.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (indiepaper-prod-db.internal:5432): non-existing domain - :nxdomain
2021-10-04T12:29:08.032101151Z app[839f9c65] ewr [info] 12:29:08.031 [error] Postgrex.Protocol (#PID<0.2263.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (indiepaper-prod-db.internal:5432): non-existing domain - :nxdomain
2021-10-04T12:29:09.282299438Z app[839f9c65] ewr [info] 12:29:09.281 [error] Postgrex.Protocol (#PID<0.2260.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (indiepaper-prod-db.internal:5432): non-existing domain - :nxdomain
2021-10-04T12:29:12.697318797Z app[839f9c65] ewr [info] 12:29:12.696 [error] Postgrex.Protocol (#PID<0.2256.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (indiepaper-prod-db.internal:5432): non-existing domain - :nxdomain
2021-10-04T12:29:17.547789949Z app[839f9c65] ewr [info] 12:29:17.547 [error] Postgrex.Protocol (#PID<0.2261.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (indiepaper-prod-db.internal:5432): non-existing domain - :nxdomain
2021-10-04T12:29:17.878030710Z app[839f9c65] ewr [info] 12:29:17.877 [error] Postgrex.Protocol (#PID<0.2265.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (indiepaper-prod-db.internal:5432): non-existing domain - :nxdomain
2021-10-04T12:29:18.671017632Z app[839f9c65] ewr [info] 12:29:18.670 [error] Postgrex.Protocol (#PID<0.2259.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (indiepaper-prod-db.internal:5432): non-existing domain - :nxdomain
2021-10-04T12:29:20.650948599Z app[839f9c65] ewr [info] 12:29:20.650 [error] Postgrex.Protocol (#PID<0.2263.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (indiepaper-prod-db.internal:5432): non-existing domain - :nxdomain
2021-10-04T12:29:20.737098012Z app[839f9c65] ewr [info] 12:29:20.736 [error] Postgrex.Protocol (#PID<0.2264.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (indiepaper-prod-db.internal:5432): non-existing domain - :nxdomain
2021-10-04T12:29:20.837424883Z app[839f9c65] ewr [info] 12:29:20.836 [error] Postgrex.Protocol (#PID<0.2255.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (indiepaper-prod-db.internal:5432): non-existing domain - :nxdomain
V2021-10-04T12:29:22.883078497Z app[839f9c65] ewr [info] 12:29:22.882 [error] Postgrex.Protocol (#PID<0.2266.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (indiepaper-prod-db.internal:5432): non-existing domain - :nxdomain
2021-10-04T12:29:27.222103320Z app[839f9c65] ewr [info] 12:29:27.221 [error] Postgrex.Protocol (#PID<0.2262.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (indiepaper-prod-db.internal:5432): non-existing domain - :nxdomain
2021-10-04T12:29:29.031161955Z app[839f9c65] ewr [info] 12:29:29.030 [error] Postgrex.Protocol (#PID<0.2265.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (indiepaper-prod-db.internal:5432): non-existing domain - :nxdomain
2021-10-04T12:29:30.351188398Z app[839f9c65] ewr [info] 12:29:30.350 [error] Postgrex.Protocol (#PID<0.2260.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (indiepaper-prod-db.internal:5432): non-existing domain - :nxdomain
2021-10-04T12:29:32.890156871Z app[839f9c65] ewr [info] 12:29:32.889 [error] Postgrex.Protocol (#PID<0.2261.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (indiepaper-prod-db.internal:5432): non-existing domain - :nxdomain
2021-10-04T12:29:33.229942478Z app[839f9c65] ewr [info] 12:29:33.229 [error] Postgrex.Protocol (#PID<0.2255.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (indiepaper-prod-db.internal:5432): non-existing domain - :nxdomain
2021-10-04T12:29:38.028062278Z app[839f9c65] ewr [info] 12:29:38.027 [error] Postgrex.Protocol (#PID<0.2256.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (indiepaper-prod-db.internal:5432): non-existing domain - :nxdomain
2021-10-04T12:29:39.619886622Z app[839f9c65] ewr [info] 12:29:39.619 [error] Postgrex.Protocol (#PID<0.2262.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (indiepaper-prod-db.internal:5432): non-existing domain - :nxdomain
2021-10-04T12:29:42.924206209Z app[839f9c65] ewr [info] 12:29:42.923 [error] Postgrex.Protocol (#PID<0.2263.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (indiepaper-prod-db.internal:5432): non-existing domain - :nxdomain
2021-10-04T12:29:43.693248514Z app[839f9c65] ewr [info] 12:29:43.692 [error] Postgrex.Protocol (#PID<0.2260.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (indiepaper-prod-db.internal:5432): non-existing domain - :nxdomain
2021-10-04T12:29:44.863306959Z app[839f9c65] ewr [info] 12:29:44.862 [error] Postgrex.Protocol (#PID<0.2266.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (indiepaper-prod-db.internal:5432): non-existing domain - :nxdomain
2021-10-04T12:29:44.939496035Z app[839f9c65] ewr [info] 12:29:44.938 [error] Postgrex.Protocol (#PID<0.2265.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (indiepaper-prod-db.internal:5432): non-existing domain - :nxdomain
2021-10-04T12:29:46.412202714Z app[839f9c65] ewr [info] 12:29:46.411 [error] Postgrex.Protocol (#PID<0.2264.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (indiepaper-prod-db.internal:5432): non-existing domain - :nxdomain
2021-10-04T12:29:46.834882242Z app[839f9c65] ewr [info] 12:29:46.834 [error] Postgrex.Protocol (#PID<0.2259.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (indiepaper-prod-db.internal:5432): non-existing domain - :nxdomain

It fixes itself when a new deploy is pushed. The database is working properly.

Do you have a connection pool size set? That nxdomain error is suspicious, but I think it might be deceptive.

It’s also worth trying out DNS queries. You can SSH to your instance and run dig (you might need to run apk add bind-tools first). Then run:

dig aaaa indiepaper-prod-db.internal

Dig sends a valid response, I will try again if the db goes down again.

/ # dig aaaa indiepaper-prod-db.internal

; <<>> DiG 9.16.20 <<>> aaaa indiepaper-prod-db.internal
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 38663
;; flags: qr rd; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;indiepaper-prod-db.internal.	IN	AAAA

I’ve started getting this tonight too on one of my DBs. I’ve tried bumping memory up for now so fingers crossed that does it.

I don’t think more memory will necessarily help. Adjusting the connection pool size in the app might! We’re trying to replicate these nxdomain errors, but haven’t seen them ourselves yet.

Should the connection pool size cause the db to fail? This is what I’m seeing with fly status --all

ID       TASK VERSION REGION DESIRED STATUS            HEALTH CHECKS                  RESTARTS CREATED
xxxxxxxx app  2 ⇡     syd    run     running (leader)  3 total, 3 passing             0        15m26s ago
xxxxxxxx app  2 ⇡     syd    run     running (replica) 3 total, 3 passing             0        16m17s ago
xxxxxxxx app  1       syd    stop    complete          3 total, 2 passing, 1 critical 0        17m50s ago
xxxxxxxx app  1       syd    stop    complete          3 total, 3 passing             0        23m30s ago
xxxxxxxx app  1       syd    stop    failed                                           0        26m29s ago
xxxxxxxx app  1       syd    stop    failed                                           0        28m30s ago
xxxxxxxx app  1       syd    stop    failed                                           0        30m26s ago
xxxxxxxx app  1       syd    stop    failed                                           0        30m36s ago
xxxxxxxx app  1       syd    stop    failed                                           0        32m23s ago
xxxxxxxx app  1       syd    stop    failed                                           0        32m23s ago
xxxxxxxx app  1       syd    stop    failed                                           0        33m40s ago
xxxxxxxx app  1       syd    stop    failed            3 total, 3 critical            9        2021-10-03T15:30:12Z
xxxxxxxx app  1       syd    stop    failed            3 total, 3 critical            4        2021-09-20T21:50:32Z
xxxxxxxx app  0       syd    stop    failed            3 total, 3 critical            1        2021-09-20T19:32:08Z

I have set the pool size to 10, and running the db on a dedicated-1x with 2048 mb RAM

No, connection pool size is an app side setting. Those types of failures are generally health checks failing or OOMs, which can indeed be a problem with memory exhaustion.

@aswinmohanme some of these DNS errors were caused by us. When you created indiepaper-prod-db app in one organization, and then removed and recreated it in another, we unintentionally kept the organization ID cached in some parts of our DNS service. So the new app wasn’t able to do a lookup because the DNS service thought it was part of a different org.

This should be fixed soon!

1 Like

That might have been the issue, I faced some errors when I first moved the app, that’s why I recreated it in the new org.

Thanks for the fix, and for supporting these weird workflows :sweat_smile:.

Deploying also fails intermittently

Command: /app/bin/indie_paper eval IndiePaper.Release.migrate
         Starting instance
         Configuring virtual machine
         Pulling container image
         Unpacking image
         Preparing kernel init
         Configuring firecracker
         Starting virtual machine
         Starting init (commit: 50ffe20)...
         Preparing to run: `/app/bin/indie_paper eval IndiePaper.Release.migrate` as nobody
         2021/10/05 04:43:38 listening on [fdaa:0:2d0a:a7b:14bf:1e2d:c443:2]:22 (DNS: [fdaa::3]:53)
         Reaped child process with pid: 561 and signal: SIGUSR1, core dumped? false
         04:43:42.070 [error] Could not create schema migrations table. This error usually happens due to the following:
           * The database does not exist
           * The "schema_migrations" table, which Ecto uses for managing
             migrations, was defined by another library
           * There is a deadlock while migrating (such as using concurrent
             indexes with a migration_lock)
         To fix the first issue, run "mix ecto.create".
         To address the second, you can run "mix ecto.drop" followed by
         "mix ecto.create". Alternatively you may configure Ecto to use
         another table and/or repository for managing migrations:
             config :indie_paper, IndiePaper.Repo,
               migration_source: "some_other_table_for_schema_migrations",
               migration_repo: AnotherRepoForSchemaMigrations
         The full error report is shown below.
         ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 2976ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
           1. Ensuring your database is available and that you can connect to it
           2. Tracking down slow queries and making sure they are running fast enough
           3. Increasing the pool_size (albeit it increases resource consumption)
           4. Allowing requests to wait longer by increasing :queue_target and :queue_interval
         See DBConnection.start_link/2 for more information
             (ecto_sql 3.7.0) lib/ecto/adapters/sql.ex:756: Ecto.Adapters.SQL.raise_sql_call_error/1
             (elixir 1.12.1) lib/enum.ex:1553: Enum."-map/2-lists^map/1-0-"/2
             (ecto_sql 3.7.0) lib/ecto/adapters/sql.ex:844: Ecto.Adapters.SQL.execute_ddl/4
             (ecto_sql 3.7.0) lib/ecto/migrator.ex:645: Ecto.Migrator.verbose_schema_migration/3
             (ecto_sql 3.7.0) lib/ecto/migrator.ex:473: Ecto.Migrator.lock_for_migrations/4
             (ecto_sql 3.7.0) lib/ecto/migrator.ex:388: Ecto.Migrator.run/4
             (ecto_sql 3.7.0) lib/ecto/migrator.ex:146: Ecto.Migrator.with_repo/3
             (indie_paper 0.1.0) lib/indie_paper/release.ex:12: anonymous fn/2 in IndiePaper.Release.migrate/0
         Main child exited normally with code: 1
         Reaped child process with pid: 563 and signal: SIGUSR1, core dumped? false
         Starting clean up.

Here is my runtime.exs

import Config

# config/runtime.exs is executed for all environments, including
# during releases. It is executed after compilation and before the
# system starts, so it is typically used to load production configuration
# and secrets from environment variables or elsewhere. Do not define
# any compile-time configuration in here, as it won't be applied.
# The block below contains prod specific runtime configuration.
if config_env() == :prod do
  database_url =
    System.get_env("DATABASE_URL") ||
      raise """
      environment variable DATABASE_URL is missing.
      For example: ecto://USER:PASS@HOST/DATABASE
      """

  config :indie_paper, IndiePaper.Repo,
    # ssl: true,
    socket_options: [:inet6],
    url: database_url,
    pool_size: String.to_integer(System.get_env("POOL_SIZE") || "10")

  secret_key_base =
    System.get_env("SECRET_KEY_BASE") ||
      raise """
      environment variable SECRET_KEY_BASE is missing.
      You can generate one by calling: mix phx.gen.secret
      """

  app_name =
    System.get_env("FLY_APP_NAME") ||
      raise "FLY_APP_NAME not available"

  indie_paper_deploy_domain =
    System.get_env("INDIEPAPER_DEPLOY_DOMAIN") || raise "INDIEPAPER_DEPLOY_DOMAIN not available"

  config :indie_paper, IndiePaperWeb.Endpoint,
    server: true,
    url: [host: indie_paper_deploy_domain, port: 443, scheme: "https"],
    check_origin: [
      "https://#{app_name}.fly.dev",
      "https://*.#{indie_paper_deploy_domain}"
    ],
    http: [
      # Enable IPv6 and bind on all interfaces.
      # Set it to  {0, 0, 0, 0, 0, 0, 0, 1} for local network only access.
      # See the documentation on https://hexdocs.pm/plug_cowboy/Plug.Cowboy.html
      # for details about using IPv6 vs IPv4 and loopback vs public addresses.
      ip: {0, 0, 0, 0, 0, 0, 0, 0},
      port: String.to_integer(System.get_env("PORT") || "4000")
    ],
    secret_key_base: secret_key_base,
    cache_static_manifest: "priv/static/cache_manifest.json"

and fly.toml

# fly.toml file generated for indiepaper-dev on 2021-09-02T14:43:54+05:30
kill_signal = "SIGTERM"
kill_timeout = 5

[env]

[deploy]
  release_command = "/app/bin/indie_paper eval IndiePaper.Release.migrate"

[[services]]
  internal_port = 4000
  protocol = "tcp"

  [services.concurrency]
    hard_limit = 25
    soft_limit = 20

  [[services.ports]]
    handlers = ["http"]
    port = 80

  [[services.ports]]
    handlers = ["tls", "http"]
    port = 443

  [[services.tcp_checks]]
    grace_period = "30s" # allow some time for startup
    interval = "15s"
    restart_limit = 6
    timeout = "2s"

Output of dig from indiepaper-dev

 # dig aaaa indiepaper-dev-db

; <<>> DiG 9.16.20 <<>> aaaa indiepaper-dev-db
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 4895
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;indiepaper-dev-db.		IN	AAAA

;; AUTHORITY SECTION:
.			86397	IN	SOA	a.root-servers.net. nstld.verisign-grs.com. 2021100401 1800 900 604800 86400

;; Query time: 36 msec
;; SERVER: fdaa::3#53(fdaa::3)
;; WHEN: Tue Oct 05 04:50:06 UTC 2021
;; MSG SIZE  rcvd: 121