Can't ssh to app

Just get waiting for ages, never connects:

Connecting to top1.nearest.of.todo-sync.internal... complete
# WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
WWWWWWWWWWWWWWWWWWWWWWW 

In app logs on the dashboard

2022-08-27T09:41:23.509 app[8253087b] cdg [info] 2022/08/27 09:41:23 unexpected error: transient SSH server error: can't resolve _orgcert.internal

2022-08-27T09:41:23.894 app[8253087b] cdg [info] 2022/08/27 09:41:23 unexpected error: [ssh: no auth passed yet, transient SSH server error: can't resolve _orgcert.internal]

If you aren’t deploying a container image without shell/secure-shell support (like distroless), then see if the cmd

flyctl ssh issue

patches it up for you.

Ref: fly ssh: Error establish key failed: key exists and override not set - #2 by kurt

If not, it seems like an internal Fly networking/dns issue and may need an escalation to their eng team, short of recreating / changing regions the app deploys to.

Same issue, loaded the cert into the ssh agent but no luck

? Select Organization: Todo Sync (todo-sync)
? Email address for user to issue cert:  atymicq@gmail.com

!!!! WARNING: We're now prompting you to save an SSH private key and certificate       !!!!
!!!! (the private key in "id_whatever" and the certificate in "id_whatever-cert.pub"). !!!!
!!!! These SSH credentials are time-limited and handling them in files is clunky;      !!!!
!!!! consider running an SSH agent and running this command with --agent. Things       !!!!
!!!! should just sort of work like magic if you do.                                    !!!!
? Path to store private key:  todo_priv
Wrote 24-hour SSH credential to todo_priv, todo_priv-cert.pub

Also tried running the command with --agent which auto loaded the cert with the same issue.

Fails on both the cron and app instances, (cron is the one i’m trying to debug)

Something else seems wonky as well, i have the region set to iad but the instances are in completely different regions?

❯ flyctl regions list                                                                                                                                                                  
Region Pool: 
iad
Backup Region: 

But in the UI, the instances are in different regions?

Strange. I believe some state somewhere (locally on your PC, or remote on Fly’s end) has gone out of sync. Did you ever move this Fly app from another org (like personal org) to its current org?

  1. Does ssh to a different Fly app (but in the same org as this failing app) work? This tell us whether the problem is limited to this one failing app or the entire org.
  2. From a different PC does ssh into this same failing Fly app work? This tells us whether the local state is interfering with flyctl cmds.