`fly ssh console -s` not showing VMs

1

Whereas before I had issues with fly ssh console -s showing old VMs, it’s now ONLY showing old VMs:

image
image1316×648 152 KB

None of these respond:

image
image1402×284 37 KB

if I look at my VM IDs, they don’t match with the options shown in fly ssh console -s

2

I ran the commands that were recommended in the post that I linked:

➜  salonbase git:(master) fly dig -a salonbase TXT vms.salonbase.internal
;; opcode: QUERY, status: NOERROR, id: 54891
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;vms.salonbase.internal.	IN	 TXT

;; ANSWER SECTION:
vms.salonbase.internal.	5	IN	TXT	"f5c22ea2 ams"

it’s now only showing 1 VM. Re-running fly ssh console -s also now only shows 1 VM:

image
image1476×208 56.4 KB

That one definitely isn’t responding :cry:

3

Now after about 15 mins it’s showing the new VMs…

image
image1920×770 77.6 KB

4

Just did another deploy. Same thing… Soooooo frustraaaating :frowning:

image
image2074×1552 373 KB

EDIT: about 7 mins later the new ones are also showing:

image
image2914×510 70.4 KB

EDIT: about 9 mins after deploying, the old VMs stopped showing in fly dig

image
image1996×532 120 KB

5

One solution is to stop monitoring the VM IDs via flyctl :wink: Move your checks internal to the app?

Btw, flyctl status --all -a <app-name> shows the approp VMs? There’s also flyctl image show -a <app-name>

6

Besides some of the other commands giving wrong information, I believe the overarching problem is that fly ssh console does not work at all, making manual maintenance and inspection of your VMs, which are rather important, impossible.
fly ssh console --select does not work at all because it only allows you to select old, unresponsive VMs. (and not using --select only makes this problem worse of course; then you would be dropped in one of these unresponsive VMs at random, in practice resulting in a timeout.)

7

Just a reminder that this is still happening…

fly ssh console -s often:

  1. Shows old VMs for minutes at a time. In that case you have to kinda guess which one is the live VM.
  2. Doesn’t show the new VM at all! That’s terrible because then it’s impossible to connect to the currently active VM…
8

We have an ongoing incident Fly.io Status - Wireguard gateway connectivity where we’re working on that too.

9

@Yaeger can you let us know how it’s looking the next few times you deploy and ssh? Should be a lot better now.

10

Haven’t been working with Fly for the last two days. But now I use it again and I am still getting that problem. To be honest, it doesn’t seem to be an “incident”. It has been like this since I started using Fly.

VM was created about 3 mins ago. Still unable to connect…

image
image1700×304 52.8 KB

11

Which app is this? Does fly dig aaaa <app>.internal show anything? And can you tell if DNS is working right from within the app?

12

The app is called beerometer.

Output of that command is:

➜  backend git:(updates-from-19-oct) ✗ fly dig aaaa beerometer.internal
Update available 0.0.413 -> v0.0.415.
Run "fly version update" to upgrade.
;; opcode: QUERY, status: NOERROR, id: 34239
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;beerometer.internal.	IN	 AAAA

;; ANSWER SECTION:
beerometer.internal.	5	IN	AAAA	fdaa:0:a116:a7b:c6ef:5ac6:1567:2
13

It works now btw, but it took about ~5 minutes before I could connect

14

fly dig aaa is just not updating :person_shrugging:

image
image4112×1408 445 KB

new VM has been up for almost 2 mins yet it’s still showing the old AAA record.

15

Such a frustrating developer experience. When I deploy I can’t access any machine for several minutes :tired_face:

16

Yeah this is good debugging. We’re going to try and ship a release of flyctl that doesn’t rely on DNS for this. DNS will get faster on gateways, too, but just hitting our API is a good short term fix.

17

Thanks, it would be great to see a sustainable solution for this problem :+1:

18 
➜  backend git:(master) ✗ fly ssh console
Update available 0.0.413 -> v0.0.415.
Run "fly version update" to upgrade.
Error host unavailable at top1.nearest.of.dentalcens.internal: host was not found in DNS

This one is also interesting btw. Also a DNS issue it seems. Getting this one for an app I just created 5 mins ago.