flyctl ssh console: Waiting for host... and nothing happens

Same here

When you removed the wire_guard_state and tried it again it still didn’t work, right? We’d expect it to put new keys back in, but it’s still not working?

Right
It put new keys and it’s still not working

1 Like

Could this be due to a tools update (flyctl)?

If it was a flyctl problem I’d expect it to break all the orgs you’re connecting to, not just this one. We’re in the process of debugging this, though, will share more info when we have something.

I have the problem for any instances trusseltrust and concordia They both are in region fra

fra I meant in ~/.fly/config.yml but they are in lhr

Thanks, we’re debugging this. In the meanwhile let me try to find a workaround.

@nickolay.loshkarev Could you do a fly status on the trusseltrust and concordia apps? Would help to see how long they’ve been running.

At this point the CLI seems to be connecting to the wrong IPs — could you also run

fly ips private

to get the current list of your VM IPs, and then

fly ssh console -s

— when you choose an app/region combination here it’ll show which IP it’s trying to connect to. If there’s a mismatch there we’ll know that that’s the issue.

The link to short record Monosnap

fly-foodbank-production-web


concordia-production-web


That’s really helpful, thanks. The DNS call to get the IP(s) to connect do seems to be failing, we can look specifically at that.

Wow. That looks network-y (that’s the error you get — it should be a better error! — when flyctl can’t talk to our DNS at all).

A question: does this work sporadically for you, or never?

If it works for you sometimes, does it depend in any way on where you’re working from (home, office, etc)?

@nickolay.loshkarev you can try changing ~/.fly/config.yml with a new peer to get going on this, while we fix the problem:

  • run fly wg create concordia lhr to create a new peer in lhr, you can choose to get the output on stdout. This will print something like
[Peer]
PublicKey = eCP0xXXXXXXXXXXXXpFUTxhjvubgDlLfVZyFk=
Endpoint = lhr1.gateway.6pn.dev:51820
  • run fly wg list, you should see something like
+-----------------------------------------------------+--------+----------------------------+
|                        NAME                         | REGION |          PEER IP           |
+-----------------------------------------------------+--------+----------------------------+
| interactive-Sudhirs-Mac-mini-sudhir-j-gmail-com-785 | maa    | fdaa:0:33b5:a7b:1bfe:0:a:2 |
| interactive-Sudhirs-Mac-mini-sudhir-j-gmail-com-996 | lhr    | fdaa:0:33b5:a7b:dc6:0:a:2  |
+-----------------------------------------------------+--------+----------------------------+
  • kill the agent if it’s running

❯ ps aux | grep "fly agent"                                                                                          22:48:56
sj               50704   0.0  0.2 409255600  35248 s006  S    10:32PM   0:01.73 fly agent daemon-start
sj               50866   0.0  0.0 408103312   1344 s006  S+   10:51PM   0:00.00 grep fly agent
❯ kill 50704   

You can then update the section for wire_guard_state.concordia.peer in config.yml with peerip from the list above, and pubkey, and endpoint (remove the :51820).

You should then be able to run the console for condordia, and you can do the same for the other org as well.

Will update again once we get the issue fixed, but this should route you via lhr instead of fra.

It doesn’t work at all.
I work from Russia.
I tried connect to VPN in London, removed wire_guard_state record and ran flyctl ssh console -a concordia-production-web -r lhr

But for @ matt2 this doesn’t work either, he’s in UK

(((( still can’t connect to concordia Monosnap

Hi @sudhir.j

Do you have any updates?

We don’t have a clear fix for the problem yet — I’ve replicated it and fixed it for myself by removing all my peers using fly wireguard remove, then trying to re-connect (it should create a new peer for you).

You could do fly wireguard list to see which peers are in fra and then remove those selectively.

Could this be related to the stale DNS answers I was seeing in the weird DNS answers thread.

This is a critical issue. Please, send me any script or anything for ssh connection
If necessary I can share my screen to do this together