I’ve developed an app using Remix/prisma/postgres and up until today have had no issues with fly. After deploying to a new ‘live’ version of our app, everything worked as expected without any issues. A couple hours later, my app stopped connecting to my postgres database with the following error and I’ve been stuck since.
2023-06-21T20:17:34.125 app[91857509c76d83] lhr [info] Error: P1001: Can't reach database server at `mailninja-platform-db.flycast`:`5432`
2023-06-21T20:17:34.125 app[91857509c76d83] lhr [info] Please make sure your database server is running at `mailninja-platform-db.flycast`:`5432`.
Since then, I’ve tried the following.
Redeploying (same error).
Restarting the app & database app (same error).
Detaching the app from the db app and reattaching (same error).
Creating a new db and attaching to that (same error).
Creating a new app and attaching that to the original db, and then deploying to the new app (same error).
Changing the DATABASE URL secret to use .internal instead of .flycast with the region at the start of the host on port 5433 (this worked, but I was getting weird read only prisma errors and was unable to use the app).
Giving up trying to save the old database (brand new app so not really a big deal), starting from scratch and launching a new app and creating a new database. (same original error).
I also have a staging version of the app, that literally has zero issues whatsoever, and is deployed using all of the same stuff except the database being scaled up from the simplest option instead of going for the production ready cluster.
I want to reiterate that when I first pushed all this live (those original apps have since been deleted in trying to find a solution to this), it worked absolutely fine. Just randomly decided a couple hours later to not be able to connect to the DB. I’ve not got any failing health checks on the DB. DB logs seem relatively normal from what I can see. It’s almost as if I just can’t create and connect an app to a postgres database anymore.
If you run fly checks list against your db is your cluster failing any healthchecks, or is it otherwise healthy?
I saw you mentioned that switching to .internal instead of flycast worked but was read only. For Fly Postgres port 5433 routes to the closest instance, which isn’t always the primary. If you switch the connection string to use port 5432, all requests will be routed to the primary. More details on that here: High Availability & Global Replication · Fly Docs
I know it doesn’t solve the root issue you’re seeing, but hopefully gets you unblocked in the meantime. Not too sure what’s causing that error, possibly something changed in Remix or Prisma?
I doubt it’s anything in Remix or Prisma - like I say the staging application I have is using the exact same stuff. I did notice the postgres image is 15.2, and in the database that’s not working it’s 15.3. But the biggest thing that suggests it shouldn’t be anything that’s updated is the fact it worked for hours, then just stopped without any prompt/update/deployment.
Success! Updating the host to use lhr.mailninja-platform-db.internal instead of mailninja-platform-db.flycast (without the region - I know there have been issues with regions being included with flycast so I wanna make it clear that’s not what I’ve been doing) and using the 5432 port works!
Thanks a bunch! Happy it’s working again, I guess it’s something to do with this app and the flycast host/proxy then, no idea. Are there any issues or problems I might run into with using the .internal host?
Glad to hear it’s working! We’re taking a look to see if there’s something weird going on with Flycast routing on your host.
The main difference between .internal and .flycast addresses is that requests over .flycast go through the proxy while .internal are direct to the instance. With flycast you get some extra goodies from the proxy (like load balancing) but for the most part you shouldn’t see a big difference. Full details on the difference between the two are here: Flycast for postgres
Any updates on this? I’d like the extra goodies but we have users on our app now so if possible I want to know that it should be solved before changing the connection url back to flycast.