Postgres Database Host Name not found by Rails application

Questions / Help
, ,
1

Hello,

I am encountering the following error without changes to my rails application in the logs:

ActiveRecord::DatabaseConnectionError (There is an issue connecting with your hostname: top2.nearest.of.<my-app-db>.internal. Please check your database configuration and ensure there is a valid connection to your database.):

Rails version 7.1.3.4
Postgres version: (on Fly) 14.6

Both my rails application and my database are running without issue in Fly and the rails application redeploys without issue (passing health checks). I have recreated the DATABASE_URL secret for my rails app by running fly postgres attach <my-app-db> --app <my-app> --database-name <my_app_database_name> --database-user <new_app_database_user> which gave me a fresh database user which can connect to the database without issue (verified by running fly postgres connect -a <my-app-db> -d <my_app_database_name> -u <new_app_database_user> -p <password-generated-by-fly-postgres-attach>) and the app still successfully redeploys but I have the same error as above.

Any help would be greatly appreciated.

Thanks,
Chris

2

Hi… What does fly dig top2.nearest.of.<my-app-db>.internal say?

Also, how many Machines do you currently have in my-app-db?

4

Hello - here’s the output from fly dig

fly dig top2.nearest.of.<my-app-db>.internal
;; opcode: QUERY, status: NOERROR, id: 45086
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;top2.nearest.of.<my-app-db>.internal.	IN	 AAAA

;; ANSWER SECTION:
top2.nearest.of.<my-app-db>.internal.	10	IN	AAAA	fdaa:0:b813:a7b:bbfa:bd00:4f4d:

I’m just running the one machine for my-app-db

5

Hm… This might not really be DNS, then. Do you see any authentication failures in fly logs -a my-app-db (the Postgres Machine’s own logs)?

6

No new logs were created good or bad on the db side when attempting to access it from the app side. However I have now noticed that my application is now up and running again without error and without me having changed anything…

1 Like
7

As of now my app is down again with the same error without me changing anything again. While this is an issue I checked the logs on the database app side and no new logs are being created at all despite the host errors on the application side.

I can’t see any issues on the status page https://status.flyio.net/ but it looks like this might be a networking issue of some kind…? If anyone knows any way to troubleshoot this that would be great

8

One thing to try is cloning a Machine from your Rails app into a different region. That would help determine how localized the problem really is. The ams region had a glitch once where servers were reachable from one direction but not another, for example.

You can also SSH into an existing Rails Machine and dig AAAA top2.nearest.of.<my-app-db>.internal from there.

(On Debian, dig is in the dnsutils package.)

9

Thanks for the help - I cloned into the AMS region and opened up a rails console and, sure enough, I can access the DB just fine. Doing so on the main LHR region app I had is having issues so this looks like an LHR region issue specifically on the application side.

1 Like
10

have you looked at the logs of your database app? there are “drop user” as well as “password authentication failed” messages that may explain your issues.

12

Hi :waving_hand: @gideonbrimleaf, do you have insight into what is the underlying error that caused the ActiveRecord::DatabaseConnectionError? The error message only says there is an error, but it seems unclear whether this is a DNS error or some kind of TCP connection error. Checking whether dig AAAA <hostname> works as mentioned above also helps.

13

:wave: again @gideonbrimleaf, sorry for the inconvenience, we recently deployed an update to our DNS stack, and your app seems to have exactly run into a corner case – your main app and the database landed on the same physical host, and that triggered a bug with topN.nearest.of resolution. This has now been fixed, you should not be seeing this consistently failing again.

1 Like
14

Hi @PeterCxy - thanks for your help! My app is back up and running again.

15

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.