fly ssh times out on port 22 while the tunnel and fly doctor are healthy

Questions / Help
1
  • machine in ams, state started, flyctl v0.4.104

fly ssh console hangs and eventually times out on port 22, while the tunnel itself is healthy:

Testing authentication token... PASSED
Testing flyctl agent... PASSED
Pinging WireGuard gateway... PASSED
Testing WireGuard DNS... PASSED
Testing WireGuard Flaps... PASSED

The app serves HTTP normally and users are working on it right now. The machine logs show the SSH server coming up at boot:

INFO SSH listening listen_address=[fdaa:76:752c:a7b:32b:82e4:3848:2]:22

But every attempt ends with:

Connecting to fdaa:76:752c:a7b:32b:82e4:3848:2... complete
Error: error connecting to SSH server: dial: connect tcp [fdaa:76:752c:a7b:32b:82e4:3848:2]:22: operation timed out

Earlier today the same call returned 503 Service Unavailable from the gateway instead of a timeout.

I found this older thread with the same symptom and tried the FLY_AGENT_WG_REGION workaround suggested there, but it did not help:

  • lhr and mad both fail with a 500 from addWireGuardPeer
  • cdg builds the tunnel fine, but SSH still hangs with the machine confirmed started

Everything else I have tried, with no change:

  • flyctl wireguard reset, and removing stale peers (one peer left)
  • flyctl agent restart, and deleting ~/.fly/wireguard*
  • restarting the machine from the dashboard, and fly machine start immediately followed by fly ssh console
  • flyctl ssh issue with fresh credentials
  • flyctl proxy 10022:22 plus plain ssh -v: the TCP connection is accepted but the server never sends its protocol version string, so the handshake never starts

This worked normally until yesterday and nothing changed on the app side; the last deploy was three days ago. Is there anything else I can check, or does this need someone to look at the gateway?

3

Hm… I just tried connecting through a cdg peer to a Machine of my own and didn’t encounter any problems.

Someone else was reporting SSH timeouts earlier today, though, so there might a snag specific to a particular underlying physical host machine, etc.

I moved your thread into the Questions / Help category, which is the place to put things when you’d like input from Fly.io staff, etc.

In the interim, you could try deleting all the WireGuard peers (not just most) after stopping the local agent.

Another thing is to create a temporary Machine in another region, and see if that, too, has problems.

4

Hi, does the same thing happen with machines in another app?

5

No, it does not. I created a throwaway app to test.

A fresh nginx Machine in the same region (ams) under a different app: SSH works immediately.

Only my existing Machine 784397db1d46e8 in ams times out on port 22,
from the same client, same agent, same peers, minutes apart from the
working tests.

That Machine is otherwise healthy: it serves HTTP normally, I`m working on it right now, and fly machine exec works against it over the
Machines API, so I can run commands, just not SSH. Its logs show
“INFO SSH listening …:22” at boot.

It has a volume attached.

6

Thanks, the temporary Machine test was useful.

So it looks like a problem with the host that Machine is on, as you suspected. It is otherwise healthy: HTTP serves fine, users are working on it, and fly machine exec works over the Machines API. It has a volume attached so I cannot easily recreate it elsewhere.

Thanks for moving the thread to Questions / Help. :slight_smile:

7

Hi, it does indeed appear to be the host unfortunately - we’re looking into it, in the meanwhile you could try, if you can afford short downtime:

  • stop the old machine (to prevent unsynced writes to the volume etc)
  • fork the volume
  • clone the machine with --attach-volume to the newly-forked volume
  • try ssh to the newly-cloned machine
  • if successful, can kill the old machine and volume.

This basically gets you another machine in another host that should not be having problems.

8

Ok Thanks for the quick response. In the meanwhile I will follow your suggestions!

9

Thank you. I have deleted all the piers, logged out logged back in, turned off all anti-virus and firewalls, turned on websockets, deleted and reinstalled fly.exe. I can ssh into aws just fine. I don’t know what it is

10

Hi, did you make sure the machine is running when you try to ssh into it? I did find it’s set up to auto-stop, so even if it responds to http requests, it might stop afterwards and be unavailable by the time you try to ssh into it.

11

Hi, The fork and clone procedure worked, thanks.

The forked volume landed in differebt zone of ams, so a different host as intended. SSH into the cloned Machine connected on the first try.

On auto-stop: my app does use it, but it was not the cause here. I had chained fly machine start and fly ssh console in one command and confirmed fly status showed started seconds before each attempt, and it still timed out, while a fresh Machine in the same region under a different app connected fine at the same moment. fly machine exec worked against the stuck Machine throughout too.

The old Machine and its volume are still there, stopped, in case they are useful for your investigation. Let me know.