It's taking forever to proxy postgres database

Questions / Help Build debugging
1

I am trying to connect to the remote database by proxying with this command:

fly proxy 5432 -a nepalist-db --verbose

Nothing happens and I am not seeing anything even with --verbose. It just doesn’t return anything.

I want to run prisma migration in remote database.

This same command worked few days/week ago.

2

Can you run it with LOG_LEVEL=debug flyctl proxy ...? That will log a lot more. I don’t think --verbose does anything.

3

yeah. i could see bunch of graphql requests returning 200 now.

This response below being the last.

DEBUG {}
DEBUG <-- 200 https://api.fly.io/graphql (158.93ms)

{
  "data": {
    "validateWireGuardPeers": {
      "invalidPeerIps": []
    }
  }
}
4

Do you already have Postgres running locally?

Try this and see if it behaves differently:

fly proxy 15432 -a nepalist-db
5

I had local postgres running in 5432 but I stopped it before I tried to connect to the remote database.

Just tried different port too but same thing is happening. I am waiting for few minutes already! Nothing happens.

Also tried restarting the pg app with no luck.

6

Also, prisma cannot find the postgres database hosted in fly.io. This is the reason why I had to proxy.

Is this a known issue ? @kurt @jerome

7

I don’t know! I don’t use prisma much. Someone else here might know what’s wrong if you start another topic.

For the pg proxy command, try seeing what fly doctor -a nepalist-db says? Then run fly wireguard reset and see if that helps?

No output is weird.

8

This is what i got while running the doctor command.

Testing authentication token... PASSED
Testing flyctl agent... PASSED
Testing local Docker instance... Nope
Pinging WireGuard gateway (give us a sec)... PASSED

App specific checks for nepalist-db:
Checking that app has ip addresses allocated... PASSED
read udp 192.161.0.77:0->9.9.9.9:53: i/o timeout. Can't proceed to check A or AAAA records.

Build checks for nepalist-db:
Checking docker context size (this may take little bit)... PASSED (829 kB)
Checking for .dockerignore... PASSED

However, even after reseting wireguard, proxying is still stuck.

9

Edit: I thought the 9.9.9.9 thing was the issue, but that’s just a fly doctor quirk.

10

Can you do this and see what errors / hangs and paste whatever output you’re comfortable with?

  • fly agent stop
  • LOG_LEVEL=debug fly agent run
  • Then in a different terminal:
    • LOG_LEVEL=debug fly proxy 5432 -a nepalist-db

I am most interested in the agent’s logs.

If you can’t connect to Quad9 as your fly doctor logs show, I think maybe you can’t connect to our DNS servers either.

11

This is what I got:
Few drops it seems.

2023/01/09 15:32:27.658502 #3 -> (  746) "\xe8\x02ok {\"WireGuardState\":{\"org\":\"personal\",\"name\":\"interactive-agent-Roshans-MacBook-Pro-roshangm01-gmail-com-ulidchanged\",\"region\":\"lhr\",\"localPrivateKey\":\"[redacted]\",\"localpublic\":\"bpI+wbmgL+MG/changed+iw=\",\"dns\":\"\",\"peer\":{\"peerip\":\"fdaa:1:a0a:a7b:dc6:0:a:2\",\"endpointip\":\"lhr1.gateway.6pn.dev\",\"pubkey\":\"eCP0xi9xD62HqHSEEa3skpxpFUTxhjvubgDlLfVZyFk=\"}},\"TunnelConfig\":{\"LocalPrivateKey\":\"[redacted]\",\"LocalNetwork\":\"fdaa:1:a0a:a7b:dc6:0:a:0/120\",\"RemotePublicKey\":\"eCP0xi9xD62HqHSEEa3skpxpPublicKeychanged=\",\"RemoteNetwork\":\"fdaa:1:a0a::/48\",\"Endpoint\":\"lhr1.gateway.6pn.dev:51820\",\"DNS\":\"fdaa:1:a0a::3\",\"KeepAlive\":0,\"MTU\":0,\"LogLevel\":0}}\n"
2023/01/09 15:32:27.658581 #3 dropped.
2023/01/09 15:32:27.659753 #4 connected ...
2023/01/09 15:32:27.659801 #4 <- (   14) "probe personal"
2023/01/09 15:32:27.659810 srv probing "personal" ...
2023/01/09 15:32:27.785448 srv "personal" probed: fdaa:1:a0a::3
2023/01/09 15:32:27.785492 #4 -> (    4) "\x02\x00ok"
2023/01/09 15:32:27.785509 #4 dropped.
2023/01/09 15:32:27.970278 #5 connected ...
2023/01/09 15:32:27.970430 srv config change at: 2023-01-09 15:32:27.969737815 +0000 GMT
2023/01/09 15:32:27.970447 #5 <- (    4) "ping"
2023/01/09 15:32:27.970519 #5 -> (   57) "7\x00ok {\"PID\":2834,\"Version\":\"0.0.443\",\"Background\":false}\n"
2023/01/09 15:32:27.970534 #5 dropped.
2023/01/09 15:32:27.971191 #6 connected ...
2023/01/09 15:32:27.971223 #6 <- (   37) "resolve personal nepalist-db.internal"
2023/01/09 15:32:28.069928 #6 -> (   34) " \x00ok fdaa:1:a0a:a7b:8e:74ae:bc08:2"
2023/01/09 15:32:28.069989 #6 dropped.

And at the end, I saw this:
Update:

2023/01/09 15:38:10.314350 srv validated wireguard peers

2023/01/09 15:40:57.464065 srv resetting connection due to error: EOF
2023/01/09 15:40:57.464125 srv resetting connection due to error: EOF
2023/01/09 15:41:02.464294 srv resetting connection due to error: EOF
2023/01/09 15:41:02.503510 srv (re-)connecting to wss://lhr1.gateway.6pn.dev:443/
2023/01/09 15:41:07.464689 srv resetting connection due to error: write tcp 172.20.10.6:52951->176.58.88.43:443: write: broken pipe
2023/01/09 15:41:12.464694 srv resetting connection due to error: EOF
2023/01/09 15:41:26.656813 srv resetting connection due to error: EOF
2023/01/09 15:41:26.656850 srv resetting connection due to error: EOF
2023/01/09 15:41:31.657132 srv resetting connection due to error: EOF
2023/01/09 15:41:32.502515 srv (re-)connecting to wss://lhr1.gateway.6pn.dev:443/
2023/01/09 15:41:36.657462 srv resetting connection due to error: write tcp 172.20.10.6:52951->176.58.88.43:443: write: broken pipe
2023/01/09 15:41:41.658335 srv resetting connection due to error: EOF
2023/01/09 15:41:42.503270 srv (re-)connecting to wss://lhr1.gateway.6pn.dev:443/
12

Alright, this looks like it’s connecting over websockets. Maybe that’s not quite working right.

Can you run fly wireguard websockets disable and then try the same commands again?

13

Now with the websockets disabled, I don’t see the error related to websocket but the first error is still there and proxying is still stuck.

14

@jerome This is something else I am seen.

2023/01/09 16:55:46.531226 srv probing "personal" ...
2023/01/09 16:55:51.532100 #4 -> (   58) "8\x00err failed probing \"personal\": context deadline exceeded"
2023/01/09 16:55:51.532148 #4 dropped.

Edit:
This went away after wireguard reset but still the original problem is still there.

15

I’m seeing all the same logs locally, except fly proxy ... does print:

Proxying local port 5432 to remote [my-app.internal]:5432

Can you try:

LOG_LEVEL=debug fly proxy 5432 fdaa:1:a0a:a7b:8e:74ae:bc08:2 -a nepalist-db
16

Nothing different happening really.

Instead the message looks like this now:

Proxying local port 5432 to remote [fdaa:1:a0a:a7b:8e:74ae:bc08:2]:5432

17

You said you didn’t see anything in your original post. That’s something :slight_smile:

So can you tell us more about the problem you’re facing? That’s proxying port 5432 from your instance to your localhost:5432. Can you now connect to it as if it was local?

It doesn’t print anything after that, I don’t think.

1 Like
18

what :confused: I guess prisma studio got me confused. it wasn’t able to connect to the database at all and migration was failing. And also, that message “Proxying” kept me hanging.

Or maybe, now all those “reset” stuff did the work. Now I am able to run the migration.

Thanks a lot for your help and sorry for wasting your time. :smiley: I feel stupid!

1 Like