MySQL 5.7 - Elixir Phoenix - non-existing domain - :nxdomain

ERIADE · August 4, 2022, 4:51pm

I have a Phoenix Umbrella app that I have got working as expected with the Postgres database also on Flyio. When I try to connect to MySQL (5.7), I get the connection error below.

--> This release will not be available until the release command succeeds.
	 Starting instance
	 Configuring virtual machine
	 Pulling container image
	 Unpacking image
	 Preparing kernel init
	 Configuring firecracker
	 Starting virtual machine
	 Starting init (commit: c86b3dc)...
	 Setting up swapspace version 1, size = 512 MiB (536866816 bytes)
	 2022/08/04 16:44:11 listening on [fdaa:0:7a12:a7b:aa9c:d515:969d:2]:22 (DNS: [fdaa::3]:53)
	 Reaped child process with pid: 571 and signal: SIGUSR1, core dumped? false
	 16:44:13.234 [info] Migrations already up
	 16:44:13.337 [error] MyXQL.Connection (#PID<0.173.0>) failed to connect: ** (DBConnection.ConnectionError) (eriade-db-car2db.internal:3306) non-existing domain - :nxdomain
	 16:44:13.337 [error] MyXQL.Connection (#PID<0.172.0>) failed to connect: ** (DBConnection.ConnectionError) (eriade-db-car2db.internal:3306) non-existing domain - :nxdomain
	 Starting instance
	 Configuring virtual machine
	 Pulling container image
	 Unpacking image
	 Preparing kernel init
	 Configuring firecracker
	 Starting virtual machine
	 Starting init (commit: c86b3dc)...
	 Setting up swapspace version 1, size = 512 MiB (536866816 bytes)
	 no label, UUID=e0b03fca-1c02-4e33-bfff-7ddef4d50a1b
	 Preparing to run: `launcher _build/prod/rel/eriade/bin/eriade eval EriadeWeb.Release.migrate` as heroku
	 2022/08/04 16:44:11 listening on [fdaa:0:7a12:a7b:aa9c:d515:969d:2]:22 (DNS: [fdaa::3]:53)
	 Reaped child process with pid: 571 and signal: SIGUSR1, core dumped? false
	 16:44:13.234 [info] Migrations already up
	 16:44:13.337 [error] MyXQL.Connection (#PID<0.173.0>) failed to connect: ** (DBConnection.ConnectionError) (eriade-db-car2db.internal:3306) non-existing domain - :nxdomain
	 16:44:13.337 [error] MyXQL.Connection (#PID<0.172.0>) failed to connect: ** (DBConnection.ConnectionError) (eriade-db-car2db.internal:3306) non-existing domain - :nxdomain
	 16:44:15.067 [error] MyXQL.Connection (#PID<0.173.0>) failed to connect: ** (DBConnection.ConnectionError) (eriade-db-car2db.internal:3306) non-existing domain - :nxdomain
	 16:44:14.695 [error] MyXQL.Connection (#PID<0.172.0>) failed to connect: ** (DBConnection.ConnectionError) (eriade-db-car2db.internal:3306) non-existing domain - :nxdomain
	 16:44:15.067 [error] MyXQL.Connection (#PID<0.173.0>) failed to connect: ** (DBConnection.ConnectionError) (eriade-db-car2db.internal:3306) non-existing domain - :nxdomain
	     indexes with a migration_lock)
	 To fix the first issue, run "mix ecto.create".
	 "mix ecto.create". Alternatively you may configure Ecto to use
	       migration_repo: AnotherRepoForSchemaMigrations
	 ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 2991ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
	   1. Ensuring your database is available and that you can connect to it
	   2. Tracking down slow queries and making sure they are running fast enough
	   4. Allowing requests to wait longer by increasing :queue_target and :queue_interval
	     (ecto_sql 3.8.3) lib/ecto/adapters/sql.ex:932: Ecto.Adapters.SQL.raise_sql_call_error/1
	     (elixir 1.13.4) lib/enum.ex:1593: Enum."-map/2-lists^map/1-0-"/2
	     (ecto_sql 3.8.3) lib/ecto/migrator.ex:696: Ecto.Migrator.verbose_schema_migration/3
	     (ecto_sql 3.8.3) lib/ecto/migrator.ex:422: Ecto.Migrator.run/4
	     (ecto_sql 3.8.3) lib/ecto/migrator.ex:146: Ecto.Migrator.with_repo/3
	     indexes with a migration_lock)
	 To fix the first issue, run "mix ecto.create".
	 "mix ecto.create". Alternatively you may configure Ecto to use
	       migration_repo: AnotherRepoForSchemaMigrations
	 ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 2991ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
	   1. Ensuring your database is available and that you can connect to it
	   2. Tracking down slow queries and making sure they are running fast enough
	   4. Allowing requests to wait longer by increasing :queue_target and :queue_interval
	     (ecto_sql 3.8.3) lib/ecto/adapters/sql.ex:932: Ecto.Adapters.SQL.raise_sql_call_error/1
	     (elixir 1.13.4) lib/enum.ex:1593: Enum."-map/2-lists^map/1-0-"/2
	     (ecto_sql 3.8.3) lib/ecto/migrator.ex:696: Ecto.Migrator.verbose_schema_migration/3
	     (ecto_sql 3.8.3) lib/ecto/migrator.ex:422: Ecto.Migrator.run/4
	     (ecto_sql 3.8.3) lib/ecto/migrator.ex:146: Ecto.Migrator.with_repo/3
	     (eriade_web 0.1.0) lib/eriade_web/release.ex:14: anonymous fn/2 in EriadeWeb.Release.migrate/0
	     (eriade_web 0.1.0) lib/eriade_web/release.ex:14: anonymous fn/2 in EriadeWeb.Release.migrate/0
	 Main child exited normally with code: 1
	 Reaped child process with pid: 573 and signal: SIGUSR1, core dumped? false
	 Starting clean up.
	 Main child exited normally with code: 1
	 Reaped child process with pid: 573 and signal: SIGUSR1, core dumped? false
	 Starting clean up.
Error release command failed, deployment aborted

I have seen this error before when attempting to connect to my Postgres database, but it was resolved by adding socket_options: [:inet6].

Having followed along with the discussion here I managed to get an instance of MySQL 5.7 deployed and could connect with fly proxy 3306:3306 -a eriade-db-car2db.

My connection string looks something like this:
mysql://car2db:mypassword=@eriade-db-car2db.internal:3306/car2db

When I add socket_options: [:inet6] to my config for MySQL connection like I did for Postgres, I get the below error.

--> This release will not be available until the release command succeeds.
	 Starting instance
	 Configuring virtual machine
	 Pulling container image
	 Unpacking image
	 Starting init (commit: c86b3dc)...
	 Setting up swapspace version 1, size = 512 MiB (536866816 bytes)
	 no label, UUID=f60a5a5f-91ef-4ba0-b98c-e873f85bc97b
	 Preparing to run: `launcher _build/prod/rel/eriade/bin/eriade eval EriadeWeb.Release.migrate` as heroku
	 2022/08/04 17:21:42 listening on [fdaa:0:7a12:a7b:be65:eb1c:9c2c:2]:22 (DNS: [fdaa::3]:53)
	 ** (ArgumentError) expected a keyword list as the second argument, got: [:inet6]
	     (myxql 0.6.2) lib/myxql/client.ex:46: MyXQL.Client.Config.new/1
	     (myxql 0.6.2) lib/myxql/connection.ex:23: MyXQL.Connection.connect/1
	     (stdlib 4.0.1) proc_lib.erl:240: :proc_lib.init_p_do_apply/3
	 Last message: nil
	 17:21:46.175 [error] GenServer #PID<0.173.0> terminating
	 ** (ArgumentError) expected a keyword list as the second argument, got: [:inet6]
	     (db_connection 2.4.2) lib/db_connection/connection.ex:82: DBConnection.Connection.connect/2
	 ** (ArgumentError) expected a keyword list as the second argument, got: [:inet6]
	     (myxql 0.6.2) lib/myxql/connection.ex:23: MyXQL.Connection.connect/1
	     (db_connection 2.4.2) lib/db_connection/connection.ex:82: DBConnection.Connection.connect/2
	     (connection 1.1.0) lib/connection.ex:622: Connection.enter_connect/5
	     (stdlib 4.0.1) proc_lib.erl:240: :proc_lib.init_p_do_apply/3
	 17:21:46.184 [error] GenServer #PID<0.175.0> terminating
	 ** (ArgumentError) expected a keyword list as the second argument, got: [:inet6]
	     (elixir 1.13.4) lib/keyword.ex:922: Keyword.merge/2
	     (myxql 0.6.2) lib/myxql/client.ex:46: MyXQL.Client.Config.new/1
	     (myxql 0.6.2) lib/myxql/connection.ex:23: MyXQL.Connection.connect/1
	 Starting instance
	 Configuring virtual machine
	     (connection 1.1.0) lib/connection.ex:622: Connection.enter_connect/5
	 Last message: nil
Error release command failed, deployment aborted

Any help would be much appreciated.

Mark · August 4, 2022, 8:42pm

Hi @ERIADE! Welcome!

Thank you for reporting the issue and thanks for the helpful information.

Your second error log tells me that myxql doesn’t support the :inet6 socket option. That tells the driver to resolve the IP address using IPv6. That’s probably the issue.

You can use the following command to resolve the IP address and then test your ability to connect to the DB using the already resolved IPv6 address.

fly ips private -a <mysql-app>

Then your connection string might look like this…

mysql://[fdaa...]::3306/car2db

NOTE: Make sure to wrap the IP address in the square brackets.

ERIADE · August 6, 2022, 11:57am

Hi @Mark, thank you for the speedy reply.

I have tried connecting using the suggested connection string with the private ip6 address and am still getting the second error.

When I ssh into my Dev server which is trying to connect to the MySQL server I ran dig -t txt _apps.internal +short and found eriade-db-car2db in the list.

My MySQL Toml file look like this is that helps at all

app = "eriade-db-car2db"
kill_signal = "SIGINT"
kill_timeout = 5
processes = []

[env]
    MYSQL_DATABASE = "car2db"
    MYSQL_USER = "car2db"

[mounts]
  source = "eriade_volume_car2db"
  destination = "/var/lib/mysql"

[build]
  # https://hub.docker.com/_/mysql/
  image = "mysql:5.7"

[experimental]
    cmd = ["--datadir", "/var/lib/mysql/thedata"]

It is running with 256MB of RAM and has a 2GB volume connected. I have bumped up the RAM to 2GB before as I read on the Fly Communinty forum that MySQL might drop connections if there is not enough RAM, but that did not resolve the issue.

One thing worth noting is that my server has these secrets set

ENV ECTO_IPV6 true
ENV ERL_AFLAGS "-proto_dist inet6_tcp"

Might this be the issue? One database(postgres) seems to rely on these to be set for a connection, whereas the MySQL connection does not like that…

Mark · August 11, 2022, 4:26pm

Hi @ERIADE,

Regarding the ENV values, try putting those in the Dockerfile.

That way they are present during the building of your application. Can’t say for sure if that’s will help, but since the problem is IPv6 related, thought I’d mention it.

ERIADE · August 11, 2022, 5:02pm

Appreciate the help @Mark.

I ended up just migrating the data to a Fly Postgres instance with pgloader.

As stated before, I had already set ECTO_IPV6 which I believe was the reason I could not connect to the MySQL server on Fly. Would still however like to know how to deploy a MySQL instance and make it publicly accessible.

Mark · August 11, 2022, 5:08pm

Hey @ERIADE,

I know that the Laravel Bytes section of the Fly.io blog had a post that showed getting a full Laravel setup working that uses MySQL. He ended up going with Planetscale for that.

Perhaps this can be helpful?

ERIADE · August 11, 2022, 5:28pm

I did look into Planetscale but I would have to bring up a MySQL server, make it publically accessible so that Planetscale could connect. I could not get MySQL to be accessible internally to my other apps through Fly DNS or make it publically accessible.

I just gave up, spun up a Postgres database and migrated my MySQL database over.

Would be good to have a tutorial on how to get a MySQL database deployed on Fly and made publically accessible.

Topic		Replies	Views
Failed to connect to database cluster (non-existing domain) Questions / Help	19	5326	June 26, 2024
Errors deploying new pg cluster - :nxdomain Questions / Help elixir , postgres	4	637	December 30, 2021
Postgres :nxdomain errors Questions / Help elixir , postgres	6	846	April 4, 2022
Database connection error deploying phoenix app to fly.io Build debugging elixir , postgres	1	307	July 31, 2024
Error connecting to Heroku Postgres elixir , postgres	1	244	December 13, 2023

MySQL 5.7 - Elixir Phoenix - non-existing domain - :nxdomain

Related topics