Another scary finicky experience that started happening the last hour, where everything seems perfectly fine with our postgres app but none of our other apps will connect.
The only guess is credentials, as I had to hack the DATABASE_URL to include a ?schema=public since no one has helped with this issue: Expanding on ENV variables
I had a ton of issues attaching and detaching so i had to create a new database, which was already a bad experience to begin with, so then I just created a single DATABASE_URL from one app and just applied that same URL to all apps, so not sure what the attach magic does, but if it creates new pg auth username/passwords and “rotates” them, then maybe that is what happen during a deploy to an app server?
Either way, this whole experience with internal network connections and getting apps to be able to talk to each other has been very finicky.
The types of issues like this one could potentially take our entire system down, which can not happen as we are in the ecommerce space where our client’s client’s are trying to checkout 24/7 on our platform.
fly pg attach and fly pg detach simply almost never work, I get “Error An unknown error occured” about 95% of the time I try to use it.
So now I am in the worst possible circumstances, every app (10-15 apps) are down because PG is not accessible to any of them, but there is no issues with the pg instance. We even restarted the instance, nothing. I also have no way of knowing what the issue is, nor can I fix the issue. Luckily we are just getting all the production stuff deployed this week so we don’t currently have any traffic on these servers, but if this happens come Monday, we will have some VERY BAD problems, what happens when this happens at 2am? That just simply can not happen.
Something doesn’t add up here! The pg attach command just creates a db/user in the cluster and then sets the DATABASE_URL secret, it doesn’t do anything else. There’s no credential rotation like on Heroku. The attach command was designed for a one to one app to database configuration, though, it’s buggy when running against an existing database.
The detach command does remove credentials, though. If you used a connection string from a previous attach, then run detach it could break other app connections.
What is breaking on your apps? Are you getting particular DB connection errors?
I have never ran detach on this db since creation.
I only thought to maybe attach the db again to get something working, but that failed too.
Basically I attached all, then used the connection url from the first attach, appended the public schema, then set the DATABASE_URL secret on all the apps to that same url and everything was fine until randomly they all dropped with failed to connect issues.
There is a small chance this happen around the time one of the app servers was deployed, but I wouldn’t think this would effect the db on all the apps?
Do you have the actual error from the apps that are saying failed to connect? They’ll usually include “authorization error” or similar if it’s a credentials issue.
edit: I found one of the logs, it’s saying it can’t reach the postgres hostname. I’m digging a little, it’s probably not related to attach or anything.
The app images. They need to be able to lookup <database>.internal to connect.
There don’t seem to be any good fixes for Node apps that hit this problem. It comes up so much, we might need to put up all kinds of scary warnings when people deploy Alpine based apps though. The bugs just don’t get any attention: EAI_AGAIN using alpine · Issue #1030 · nodejs/docker-node · GitHub
I am wondering if this happen when we switched to the local builders, but the dockerfiles we supply to fly deploy have not changed in the last few days.
I’m no docker expert that is for sure, so I am at the moment confused if this is the docker build image on circle (I doubt it) vs the app docker image (ie the FROM) within the dockerfile supplied to fly.
If so, (FROM node:current-alpine) has been what we have provided since we first deployed.
I doubt this happened because you changed anything. The DNS bugs in Alpine just happen spontaneously on previously working processes. It’s brutal.
Try FROM node:current-slim. If you’re installing any additional packages, you’ll need to change anything that runs apk add to the apt-get equivalent. Feel free to post your Dockerfile here if you want us to look.
A few hours ago one of my Fly apps also stopped being able to connect to PostgreSQL. This is being deployed from a preexisting Docker image that uses python:alpine3.14, and trying to build from python:slim breaks the apk commands. Is there any easy fix here?
Just gave that a try and it looks like the build / release failed. I will be back online shortly, just made that last change from mobile to trigger a deploy haha.
