flyctl ssh console: Waiting for host... and nothing happens

Questions / Help
21

That’s really helpful, thanks. The DNS call to get the IP(s) to connect do seems to be failing, we can look specifically at that.

22

Wow. That looks network-y (that’s the error you get — it should be a better error! — when flyctl can’t talk to our DNS at all).

A question: does this work sporadically for you, or never?

If it works for you sometimes, does it depend in any way on where you’re working from (home, office, etc)?

23

@nickolay.loshkarev you can try changing ~/.fly/config.yml with a new peer to get going on this, while we fix the problem:

  • run fly wg create concordia lhr to create a new peer in lhr, you can choose to get the output on stdout. This will print something like
[Peer]
PublicKey = eCP0xXXXXXXXXXXXXpFUTxhjvubgDlLfVZyFk=
Endpoint = lhr1.gateway.6pn.dev:51820
  • run fly wg list, you should see something like
+-----------------------------------------------------+--------+----------------------------+
|                        NAME                         | REGION |          PEER IP           |
+-----------------------------------------------------+--------+----------------------------+
| interactive-Sudhirs-Mac-mini-sudhir-j-gmail-com-785 | maa    | fdaa:0:33b5:a7b:1bfe:0:a:2 |
| interactive-Sudhirs-Mac-mini-sudhir-j-gmail-com-996 | lhr    | fdaa:0:33b5:a7b:dc6:0:a:2  |
+-----------------------------------------------------+--------+----------------------------+
  • kill the agent if it’s running

❯ ps aux | grep "fly agent"                                                                                          22:48:56
sj               50704   0.0  0.2 409255600  35248 s006  S    10:32PM   0:01.73 fly agent daemon-start
sj               50866   0.0  0.0 408103312   1344 s006  S+   10:51PM   0:00.00 grep fly agent
❯ kill 50704

You can then update the section for wire_guard_state.concordia.peer in config.yml with peerip from the list above, and pubkey, and endpoint (remove the :51820).

You should then be able to run the console for condordia, and you can do the same for the other org as well.

Will update again once we get the issue fixed, but this should route you via lhr instead of fra.

24

It doesn’t work at all.
I work from Russia.
I tried connect to VPN in London, removed wire_guard_state record and ran flyctl ssh console -a concordia-production-web -r lhr

But for @ matt2 this doesn’t work either, he’s in UK

25

(((( still can’t connect to concordia Monosnap

26

Hi @sudhir.j

Do you have any updates?

27

We don’t have a clear fix for the problem yet — I’ve replicated it and fixed it for myself by removing all my peers using fly wireguard remove, then trying to re-connect (it should create a new peer for you).

You could do fly wireguard list to see which peers are in fra and then remove those selectively.

28

Could this be related to the stale DNS answers I was seeing in the weird DNS answers thread.

29

This is a critical issue. Please, send me any script or anything for ssh connection
If necessary I can share my screen to do this together

30

@nickolay.loshkarev can you confirm that the problem is still happening even after removing all peers using fly wireguard remove? That should take out your old peers and give you new ones when you try to connect again.

31

333
3331460×487 82.9 KB

image
image1358×130 12.3 KB

image
image1558×254 23 KB

32

some people from the UK can’t connect too

33

@nickolay.loshkarev you can ssh directly into the machine if the command still isn’t working out:

Using the steps in Private Networking

  • Run fly wireguard create to create a peer. You can save it as concordia.conf.
  • Use your Wireguard VPN client of choice to connect to the VPN withe config saved above.
  • Now run dig concordia-production-web.internal AAAA to get the IPs of the running instances. You should see something like:
> dig slotbooks-api.internal AAAA
; <<>> DiG 9.10.6 <<>> slotbooks-api.internal AAAA
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 33563
;; flags: qr rd ad; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;slotbooks-api.internal.		IN	AAAA
;; ANSWER SECTION:
slotbooks-api.internal.	5	IN	AAAA	fdaa:0:38f8:a7b:232f:fc2a:25a1:2
;; Query time: 191 msec
;; SERVER: fdaa:0:38f8::3#53(fdaa:0:38f8::3)
;; WHEN: Fri Nov 12 22:46:40 IST 2021
;; MSG SIZE rcvd: 79

The fdaa:0:38f8:a7b:232f:fc2a:25a1:2 is one of the IPs of your app.

  • Run fly ssh issue to get an ssh certificate and save it, I’m saving it as sb.
  • Run ssh -i sb root@fdaa:0:38f8:a7b:232f:fc2a:25a1:2 to connect to your app instances directly.

This should get you ssh’ing directly into your applications inside a Fly VPN.

34

In my case the Wireguard macOS app seems to have set up the internal VPN DNS server correctly, in case that doesn’t happen at your end you can change the dig command to explicitly include the DNS server:

dig aaaa <app>.internal @fdaa:<xxx>

You’ll find your DNS server in the .conf created when calling fly wireguard create

35

there’s no fdaa… in answer

image
image1258×630 107 KB

image
image1254×594 94.1 KB

36

image
image2018×704 118 KB

37

Can we call you to share details?

39

Hi @sudhir.j we’re still having issues with this, just tried removing them and getting the same issue, connection just hangs.

1 Like
40

@matthewford will you please follow the troubleshooting steps here and post what errors you’re getting? The DNS errors are very specific, if that’s what you’re seeing too it’ll be helpful to know: flyctl ssh console: Waiting for host... and nothing happens - #33 by sudhir.j

41

Hey @sudhir.j @kurt

I created a new app concordia-prod-web and tried to deploy this one

got an error

12345
123451920×398 77.2 KB

Could you please help me to deploy a new app?