I am not able to proxy to my postgres application with flyctl proxy 5432 -a <APPLICATION-NAME>. I’ve tried to proxy over websockets with flyctl wireguard websockets enable and flyctl agent restart. However, no luck for me. Please find the error msg from flyctl doctor
➜ flyctl doctor
Update available 0.0.412 -> v0.0.413.
Run "flyctl version update" to upgrade.
Testing authentication token... PASSED
Testing flyctl agent... PASSED
Testing local Docker instance... PASSED
Pinging WireGuard gateway (give us a sec)... FAILED
(Error: ping gateway: no response from gateway received)
We can't establish connectivity with WireGuard for your personal organization.
WireGuard runs on 51820/udp, which your local network may block.
If this is the first time you've ever used 'flyctl' on this machine, you
can try running 'flyctl doctor' again.
If this was working before, you can ask 'flyctl' to create a new peer for
you by running 'flyctl wireguard reset'.
If your network might be blocking UDP, you can run 'flyctl wireguard websockets enable',
followed by 'flyctl agent restart', and we'll run WireGuard over HTTPS.
It was be really great if anyone can help me. Thank you🙏
Some extra info here, because I just spent a bunch of time trying to debug issues that boiled down to this.
With websockets disabled, I would have issues with hanging connections (the easiest thing was logging into a machine and then trying to run top, but other things like trying to get pg_dump running through a fly proxy would also hang). Spent a while trying older flyctl clients (it would be amazing to have a bisect command for the agents on release version), to no avail.
Enabling websockets resolved my problems, and I’m able to run top properly now. This seems to be related to my network, as tethering to my phone seems to avoid problems. But… well, I was away for a month, a month ago things were fine, now they are not fine. Maybe they were never fine! But I don’t really know how to debug a hanging connection through this wireguard connection.
Some extra details here, the wireguard gateway passed for me, though failure would happen.
For my full repro now, I’m attaching trimmed_top.txt, which is the first couple of lines of top output that I put into a file. When I open a shell and do cat trimmed_top.txt, my shell hangs (no response to Ctrl-C). This problem disappears if I enable wireguard over web packets. But I can for example do head -n 1 trimmed_top.txt. I tried doing some bisecting on this and I think that this is a “size” question, as I seem to be able to print out individual lines without hanging, and it’s more these 5 lines in the text file that cause issue.
So repro:
Spin up an instance of python:3.10-bullseye (the docker image I was using)
with fly sftp shell, put the trimmed_top.txt file onto the machine
within fly ssh console, do head -n 1 trimmed_top.txt to confirm that you are getting stuff over the shell “just fine”
in that same session (or a new one of course) do cat trimmed_top.txt. This should cause the connection to hang.
Some details:
this is all encrypted via wireguard, but wireshark seems to show stuff still going back and forth…
like I said before, I was experiencing this on one network but not by tethering on my phone (though maybe there’s some heuristic that automatically uses websockets? The value is write-only so I can’t tell)
I believe this goes beyond the SSH sessions, because I was trying to do stuff through fly pg connect and would hit things like not being able to list relations (or very long strings?). Clearly something is going on.
I can try diagnosing things deeper, but might need some pointers on how to do things, I know very little about wireguard and debugging networking stuff in general (Ubuntu host though)
I tried this command and the rest suggested by others, no luck. It seems that there is one post talking about the same issues and others are experiencing the same issue…
