`fly ssh console -s` not showing VMs

Whereas before I had issues with fly ssh console -s showing old VMs, it’s now ONLY showing old VMs:

None of these respond:

if I look at my VM IDs, they don’t match with the options shown in fly ssh console -s

1 Like

I ran the commands that were recommended in the post that I linked:

➜  salonbase git:(master) fly dig -a salonbase TXT vms.salonbase.internal
;; opcode: QUERY, status: NOERROR, id: 54891
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;vms.salonbase.internal.	IN	 TXT

vms.salonbase.internal.	5	IN	TXT	"f5c22ea2 ams"

it’s now only showing 1 VM. Re-running fly ssh console -s also now only shows 1 VM:

That one definitely isn’t responding :cry:

Now after about 15 mins it’s showing the new VMs…

Just did another deploy. Same thing… Soooooo frustraaaating :frowning:

EDIT: about 7 mins later the new ones are also showing:

EDIT: about 9 mins after deploying, the old VMs stopped showing in fly dig

One solution is to stop monitoring the VM IDs via flyctl :wink: Move your checks internal to the app?

Btw, flyctl status --all -a <app-name> shows the approp VMs? There’s also flyctl image show -a <app-name>

Besides some of the other commands giving wrong information, I believe the overarching problem is that fly ssh console does not work at all, making manual maintenance and inspection of your VMs, which are rather important, impossible.
fly ssh console --select does not work at all because it only allows you to select old, unresponsive VMs. (and not using --select only makes this problem worse of course; then you would be dropped in one of these unresponsive VMs at random, in practice resulting in a timeout.)


Just a reminder that this is still happening…

fly ssh console -s often:

  1. Shows old VMs for minutes at a time. In that case you have to kinda guess which one is the live VM.
  2. Doesn’t show the new VM at all! That’s terrible because then it’s impossible to connect to the currently active VM…

We have an ongoing incident Fly.io Status - Wireguard gateway connectivity where we’re working on that too.

@Yaeger can you let us know how it’s looking the next few times you deploy and ssh? Should be a lot better now.

Haven’t been working with Fly for the last two days. But now I use it again and I am still getting that problem. To be honest, it doesn’t seem to be an “incident”. It has been like this since I started using Fly.

VM was created about 3 mins ago. Still unable to connect…

Which app is this? Does fly dig aaaa <app>.internal show anything? And can you tell if DNS is working right from within the app?

The app is called beerometer.

Output of that command is:

➜  backend git:(updates-from-19-oct) ✗ fly dig aaaa beerometer.internal
Update available 0.0.413 -> v0.0.415.
Run "fly version update" to upgrade.
;; opcode: QUERY, status: NOERROR, id: 34239
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;beerometer.internal.	IN	 AAAA

beerometer.internal.	5	IN	AAAA	fdaa:0:a116:a7b:c6ef:5ac6:1567:2

It works now btw, but it took about ~5 minutes before I could connect

fly dig aaa is just not updating :person_shrugging:

new VM has been up for almost 2 mins yet it’s still showing the old AAA record.

Such a frustrating developer experience. When I deploy I can’t access any machine for several minutes :tired_face:

Yeah this is good debugging. We’re going to try and ship a release of flyctl that doesn’t rely on DNS for this. DNS will get faster on gateways, too, but just hitting our API is a good short term fix.


Thanks, it would be great to see a sustainable solution for this problem :+1:

➜  backend git:(master) ✗ fly ssh console
Update available 0.0.413 -> v0.0.415.
Run "fly version update" to upgrade.
Error host unavailable at top1.nearest.of.dentalcens.internal: host was not found in DNS

This one is also interesting btw. Also a DNS issue it seems. Getting this one for an app I just created 5 mins ago.