Postgres app no longer connects, logs non-stop proxy + keeper messages

Questions / Help Build debugging
1

Neither my deployed prod Fly app nor my local machine + Wireguard can connect to my Postgres app anymore (previously worked fine), and my Postgres app keeps spitting out the following logs non-stop (address + app names omitted):

2022-02-21T09:28:25.928 app[omitted-app-2] ewr [info] **proxy** | 2022-02-21T09:28:25.928Z INFO cmd/proxy.go:268 master address {"address": "[fdaa:[[omitted]]"}

2022-02-21T09:28:25.957 app[omitted-app-2] ewr [info] **proxy** | 2022-02-21T09:28:25.957Z INFO cmd/proxy.go:286 proxying to master address {"address": "[fdaa:[[omitted]]"}

2022-02-21T09:28:26.147 app[omitted-app-1] ewr [info] **proxy** | 2022-02-21T09:28:26.147Z INFO cmd/proxy.go:268 master address {"address": "[fdaa:[[omitted]]"}

2022-02-21T09:28:26.177 app[omitted-app-1] ewr [info] **proxy** | 2022-02-21T09:28:26.176Z INFO cmd/proxy.go:286 proxying to master address {"address": "[fdaa:[[omitted]]"}

2022-02-21T09:28:29.592 app[omitted-app-2] ewr [info] **keeper** | 2022-02-21T09:28:29.592Z INFO cmd/keeper.go:1505 our db requested role is master

2022-02-21T09:28:29.593 app[omitted-app-2] ewr [info] **keeper** | 2022-02-21T09:28:29.593Z INFO cmd/keeper.go:1543 already master

2022-02-21T09:28:29.608 app[omitted-app-2] ewr [info] **keeper** | 2022-02-21T09:28:29.607Z INFO cmd/keeper.go:1676 postgres parameters not changed

2022-02-21T09:28:29.608 app[omitted-app-2] ewr [info] **keeper** | 2022-02-21T09:28:29.607Z INFO cmd/keeper.go:1703 postgres hba entries not changed

2022-02-21T09:28:30.632 app[omitted-app-1] ewr [info] **keeper** | 2022-02-21T09:28:30.631Z INFO cmd/keeper.go:1557 our db requested role is standby {"followedDB": "4d883312"}

2022-02-21T09:28:30.632 app[omitted-app-1] ewr [info] **keeper** | 2022-02-21T09:28:30.632Z INFO cmd/keeper.go:1576 already standby

2022-02-21T09:28:30.647 app[omitted-app-1] ewr [info] **keeper** | 2022-02-21T09:28:30.646Z INFO cmd/keeper.go:1676 postgres parameters not changed

2022-02-21T09:28:30.647 app[omitted-app-1] ewr [info] **keeper** | 2022-02-21T09:28:30.646Z INFO cmd/keeper.go:1703 postgres hba entries not changed

When I try to connect from my local machine using Wireguard I get a “domain not found” error, and when I run fly deploy from my connected Elixir app I get:

09:28:07.077 [error] Postgrex.Protocol (#PID<0.135.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (omitted-subdomain.internal:5432): non-existing domain - :nxdomain

Memory usage is only 300mb/2gb.

I’m kind of frustrated by this because the Fly dashboard green-lights my DB when it can’t even be connected to (technically it is “running” but technically a database is also “running” when it is on and throwing errors).

Note: I have a second Fly Postgres app I created in December 2021 that connects fine with Wireguard + its app (a different Elixir app); the PSQL app that won’t connect was created sometime around September 2021

2

I see what’s going on. Attempting to bring it back.

1 Like
3

I believe it should be back now. We’re looking into what might have caused this. Looks like our API thought this app shouldn’t be running at all, and yet it was.

1 Like
4

Thanks a ton, Jerome!

When I first ran into this problem, it was right after I terminated a fly deploy build mid-upload from the CLI last Friday (this was my first fly deploy for this app for a little while, and I terminated the first run of the command), but I have not checked if the connected PSQL app was already having this issue prior to terminatingfly deploy.

5

Sure thing!

Are you talking about the app that connects to your postgres cluster or the postgres cluster itself? If the latter, you’re deploying it because you’re using special settings / extensions on your cluster?

1 Like
6

Ah, I meant the Elixir Fly app that connects to my postgres cluster, the postgres cluster itself is just the standard/default created and connected via the Fly CLI.

It is back now, thanks a ton!