Unable to reach postgres instance

, ,
1

Anyone else having similar issues? I have made no changes ot my app but suddenly i cant connect to postgres from the app.

PG::ConnectionBad: connection to server at “fdaa:3:711f:0:1::8”, port 5432 failed: server closed the connection unexpectedly (PG::ConnectionBad)
This probably means the server terminated abnormally
before or while processing the request.

I’ve never seen this before.

I’ve tried: rebooting instances, destroying machines and re-creating them, deploying the app.

Nothing has changed.

2

I am seeing the same thing, both my apps are having issues connecting to the respective Postgres DBs i.e. I am seeing Postgrex.Protocol (#PID<0.2231.0>) failed to connect: ** (DBConnection.ConnectionError) tcp recv (idle): closed

Seems like one of them will connect off and on. Maybe a fly thing?

3

I’m not sure if i fixed it or if it fixed itself. I did the following:

  • updated my postgres to the latest version
  • destroyed all the existing machines (app - not postgres)
  • scaled the app back up

Then it came online. I did the destroy and and scale up before, but it didnt seem to work…not s sure why it did this time.

4

Thanks, I’ll give that a try, I’m just running some hobby apps at the moment, so at least not a panic haha.

5

Wish i could say the same…ugh nothing worse than customers emailing you about issues you didn’t pickup on.

6

Same issue here I think, my app suddenly cannot find my Postgres instance when it was working fine before. I tried destroying my app and scaling back up but it fails to connect everytime.

8

Can you reach the db through a wireshark tunnel? I could the entire time which is the only reason i knew the db was fine. I’d check that to rule out any other issues.

9

Hi, same here across 4 apps, they all started throwing the same error. It seems it’s only affecting apps in my personal organization, I have two apps in outside organizations and they’re fine.

10

Yep db looks fine internally, can reach it through a Fly Proxy and it responds to queries.

11

same here!

12

Hm… Another thing that could be tried is switching to the .internal address instead of the .flycast one.

(In your web app’s DATABASE_URL—or similar.)

This will no longer auto-start the database or load balance, but it might at least help narrow down the source of the problem…

13

I was suffering from the same issue, the way i resolved it was i rollbacked my postgres instance to a snapshot before the latest snapshot. The new instance from the snapshot didn’t suffer from the same issue. I’m going to assume the daily snapshot was the issue ?

14

From my understanding a daily snapshot shouldn’t change anything on the live database.

But it is a probable cause.

Any additional insight here?

15

Waiting for a reply from the Fly.IO team here.

Should I wait for you guys to actually resolve the issue, or should I try @sonalranjit’s fix?

Seems the issue is on your end.

16

I can confirm this solution worked - I created a new secret “DATABASE_URL_ALT” which replaces “.flycast” with “.internal” and updated my app to use this new var

I am running pretty old versions of flyio/postgres-flex (0.0.39) and haven’t touched the databases forever. Fortunately one instance of my app remained able to connect to the .flycast url, but no other instances or new machines could

18

Same for me, all my apps stopped working…

19

My apps conenct to database via ip address and I have similar issue

20

Yesterday my postgres instance went down the same way, I reprovisioned new with the latest image and did a failover, it seemed to fix the issue.

Now my app can connect to both Redis upstash instances … And as I’m writing this they just added https://status.flyio.net/incidents/ftd07gnytjl4 so it may be networking related :-/