Cannot connect to postgres db ssh error

I am new to fly.io.
Context: deploying an elixir liveview app. Postgres db deployed. However, I need to connect to the db to load the correct schema.

I cannot connect to the db due to the error below. I have also included other info which may help you help me resolve this issue. Thank you all for your help.

flyctl postgres connect -a mcdevdb -u postgres -p postgrespassword
Connecting to fdaa:5:c452:a7b:233:93f8:373c:2… complete
Error: error connecting to SSH server: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

flyctl postgres list
NAME OWNER STATUS LATEST DEPLOY
mcdevdb personal deployed

fly status -a mcdevdb
ID STATE ROLE REGION CHECKS IMAGE CREATED
UPDATED
56833042a1d4d8 started primary syd 3 total, 3 passing flyio/postgres-flex:15.3 (v0.0.46) 2024-02-02T13:41:44Z 2024-02-02T13:42:01Z

fly version
fly.exe v0.1.147 windows/amd64 Commit: 4c5c97cd4cf9fba252937825fba858177dff24ea BuildDate: 2024-02-01T19:24:14Z

flyctl doctor
Testing authentication token… PASSED
Testing flyctl agent… PASSED
Testing local Docker instance… Nope
Pinging WireGuard gateway (give us a sec)… PASSED
Testing WireGuard DNS… PASSED
Testing WireGuard Flaps… PASSED

App specific checks for mcdev:
Checking that app has ip addresses allocated… PASSED
Checking A record for mcdev.fly.dev… PASSED
Checking AAAA record for mcdev.fly.dev… PASSED

Build checks for mcdev:
Checking docker context size (this may take little bit)… PASSED (7.3 MB)
Checking for .dockerignore… PASSED

Last view lines from the log
2024-02-02T13:54:31Z app[32871dda169d85] syd [info] INFO Starting clean up.
2024-02-02T13:54:31Z app[32871dda169d85] syd [info] WARN hallpass exited, pid: 315, status: signal: 15 (SIGTERM)
2024-02-02T13:54:31Z app[32871dda169d85] syd [info]2024/02/02 13:54:31 listening on [fdaa:5:c452:a7b:2983:d8d3:8961:2]:22 (DNS: [fdaa::3]:53)
2024-02-02T13:54:32Z app[32871dda169d85] syd [info][ 4.307579] reboot: Restarting system
2024-02-02T13:54:32Z runner[32871dda169d85] syd [info]machine restart policy set to ‘no’, not restarting

Hi… Thanks for all the details… The Postgres machine (56833042a1d4d8) has its own, separately displayable logs, and those should in turn shed more light on the SSH aspect:

fly logs -a mcdevdb

(Transient DNS glitches to the certificate store are one possible cause—as an example.)

Added postgres

Thanks for a quick response.

The db log does show transient SSH server error. Please suggest how I may reslive this. Thanks.

2024-02-03T00:22:04Z app[56833042a1d4d8] syd [info]monitor | Voting member(s): 1, Active: 1, Inactive: 0, Conflicts: 0
2024-02-03T00:22:27Z app[56833042a1d4d8] syd [info]2024/02/03 00:22:27 unexpected error fetching cert: transient SSH server error: can’t resolve _orgcert.internal
2024-02-03T00:22:27Z app[56833042a1d4d8] syd [info]2024/02/03 00:22:27 unexpected error: [ssh: no auth passed yet, transient SSH server error: can’t resolve _orgcert.internal]
2024-02-03T00:23:00Z app[56833042a1d4d8] syd [info]repmgrd | [2024-02-03 00:23:00] [INFO] monitoring primary node “fdaa:5:c452:a7b:233:93f8:373c:2” (ID: 1962847327) in normal state

This part is maintained by Fly, themselves, so fix-it options are limited…

I would just give it a few more hours, trying fly pg connect periodically and also checking your personalized status page.

If the problem persists, you could go the route of creating a new Postgres database—which will typically end up on a different physical server.

Hope it clears up soon!

Thanks. I will keep trying or will create a new db later today. Made the new db—same error.

It’s still the same error. If I cannot solve this, I will have to try a different place to host my application.

fly pg connect -a mcdevdb3
Connecting to fdaa:5:c452:a7b:124:6f0b:90cc:2… complete
Error: Error connecting to SSH server: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

log file for db
2024-02-03T08:30:54Z app[148ed2d5b5e528] sea [info]monitor | Voting member(s): 1, Active: 1, Inactive: 0, Conflicts: 0
2024-02-03T08:31:18Z app[148ed2d5b5e528] sea [info]repmgrd | [2024-02-03 08:31:18] [INFO] monitoring primary node “fdaa:5:c452:a7b:124:6f0b:90cc:2” (ID: 156333374) in normal state
2024-02-03T08:31:22Z app[148ed2d5b5e528] sea [info]2024/02/03 08:31:22 unexpected error fetching cert: transient SSH server error: can’t resolve _orgcert.internal
2024-02-03T08:31:23Z app[148ed2d5b5e528] sea [info]2024/02/03 08:31:23 unexpected error: [ssh: no auth passed yet, transient SSH server error: can’t resolve _orgcert.internal]
2024-02-03T08:32:04Z app[148ed2d5b5e528] sea [info]2024/02/03 08:32:04 unexpected error fetching cert: transient SSH server error: can’t resolve _orgcert.internal
2024-02-03T08:32:04Z app[148ed2d5b5e528] sea [info]2024/02/03 08:32:04 unexpected error: [ssh: no auth passed yet, transient SSH server error: can’t resolve _orgcert.internal]
2024-02-03T08:35:54Z app[148ed2d5b5e528] sea [info]monitor | Voting member(s): 1, Active: 1, Inactive: 0, Conflicts: 0
2024-02-03T08:36:19Z app[148ed2d5b5e528] sea [info]repmgrd | [2024-02-03 08:36:19] [INFO] monitoring primary node “fdaa:5:c452:a7b:124:6f0b:90cc:2” (ID: 156333374) in normal state

It’s very peculiar that you are seeing this in both Sydney and Seattle…

Are you able to SSH into your Elixir machine?

fly m start  # ...choose one from the interactive list.
fly ssh console -C printenv

(You will generally need to issue the second command within ~2 minutes, before the other side auto-stops.)

Thanks. I tried the two command above and got the same error. BTW I installed the hello-fly app this time.

Ran all the below in GIT Bash shell on Windows11.

$ fly m start
? Select machines: 9185e55b724283 quiet-meadow-5117 (stopped, region syd, process group ‘app’)
9185e55b724283 has been started

carl@Carls-2-Surface MINGW64 ~/dev/play/hello-fly (main)
$ fly ssh console -C printenv
Connecting to fdaa:5:c452:a7b:2983:e5cd:fcc4:2… complete
Error: error connecting to SSH server: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

Before I did the above, I ran
$ fly ssh issue
? Select Organization: Carl B Hill (personal)

!!! WARNING: We’re now prompting you to save an SSH private key and certificate !!!
!!! (the private key in “id_whatever” and the certificate in “id_whatever-cert.pub”). !!!
!!! These SSH credentials are time-limited and handling them in files is clunky; !!!
!!! consider running an SSH agent and running this command with --agent. Things !!!
!!! should just sort of work like magic if you do. !!!
? Path to store private key: .ssh
Wrote 24-hour SSH credential to .ssh, .ssh-cert.pub

$ ssh-add ~/.ssh/.ssh
Identity added: /c/Users/carl/.ssh/.ssh (fly.io)
Certificate added: /c/Users/carl/.ssh/.ssh-cert.pub (fly:org:365124:user:339085)

$ printenv | grep SSH
SSH_AUTH_SOCK=/tmp/ssh-MiFvHNGk6JPL/agent.1984
SSH_AGENT_PID=1985

$ fly m stop
? Select machines: 9185e55b724283 quiet-meadow-5117 (stopped, region syd, process group ‘app’)
Sending kill signal to machine 9185e55b724283…
9185e55b724283 has been successfully stopped

I also tried
$ flyctl ssh issue --agent
? Select Organization: Carl B Hill (personal)
Error: can’t connect to SSH agent: dial unix C:/Users/carl/AppData/Local/Temp/ssh-UGtvypdZDpAn/agent.2586: connect: No connection could be made because the target machine actively refused it.

carl@Carls-2-Surface MINGW64 ~/dev/play/hello-fly (main)
$ fly ssh console helloflycarl
Error: host unavailable at helloflycarl: host was not found in DNS

Logs below
2024-02-03T14:43:57Z app[9185e55b724283] syd [info][ 458.808593] reboot: Restarting system
2024-02-04T08:32:19Z app[9185e55b724283] syd [info][ 0.045955] PCI: Fatal: No config space access function found
2024-02-04T08:32:19Z app[9185e55b724283] syd [info] INFO Starting init (commit: bfa79be)…
2024-02-04T08:32:19Z app[9185e55b724283] syd [info] INFO Preparing to run: docker-entrypoint.sh npm start as root
2024-02-04T08:32:19Z app[9185e55b724283] syd [info] INFO [fly api proxy] listening at /.fly/api
2024-02-04T08:32:19Z runner[9185e55b724283] syd [info]Machine started in 459ms
2024-02-04T08:32:19Z app[9185e55b724283] syd [info]2024/02/04 08:32:19 listening on [fdaa:5:c452:a7b:2983:e5cd:fcc4:2]:22 (DNS: [fdaa::3]:53)
2024-02-04T08:32:19Z app[9185e55b724283] syd [info]> hello-fly@0.0.0 start
2024-02-04T08:32:19Z app[9185e55b724283] syd [info]> node ./bin/www
2024-02-04T08:32:26Z app[9185e55b724283] syd [info]2024/02/04 08:32:26 unexpected error fetching cert: transient SSH server error: can’t resolve _orgcert.internal
2024-02-04T08:32:26Z app[9185e55b724283] syd [info]2024/02/04 08:32:26 unexpected error: [ssh: no auth passed yet, transient SSH server error: can’t resolve _orgcert.internal]

1 Like

I am watching this thread. same issue here.

1 Like

I am looking at Can't run ssh console - #5 by kurt which has the same kind of was resolved as follows

from @kurt – which would suggest if I’m reading this right that the problem is scoped to your organization (and not to the machine or the data center).

1 Like

Hi thanks. I did go through @kurt 's recommendations and still could not get it to work. I will try again today. When you say your organization is there a file that I have to check or is my organization setup in fly.io?

I followed these steps and could connect to bitbucked. Followed the same steps and could not connect fly.

Steps.
PS C:\Windows\system32> Start-Service ssh-agent
PS C:\Windows\system32> Get-Service ssh-agent

Status Name DisplayName


Running ssh-agent OpenSSH Authentication Agent

PS C:\users\carl.ssh> ssh-add ./id_rsa
Enter passphrase for ./id_rsa:
Identity added: ./id_rsa (carl@Carls-2-Surface)

Add the id_rsa.pub key to bitbucket

PS C:\users\carl.ssh> ssh -T git@bitbucket.org
You can use git to connect to Bitbucket. Shell access is disabled.

For fly

cd c:\users\carl.ssh
fly ssh issue

? Path to store private key: .ssh
Wrote 24-hour SSH credential to .ssh, .ssh-cert.pub
PS C:\users\carl.ssh> ssh-add ./.ssh
Identity added: ./.ssh (fly.io)
Certificate added: ./.ssh-cert.pub (fly:org:xxxxxx:user:xxxxxx)

fly pg connect -a helloflydb
Connecting to fdaa:5:c452:a7b:232:a26e:aa50:2… complete
Error: error connecting to SSH server: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

2024-02-05T07:07:49Z app[90806145c17e28] syd [info]2024/02/05 07:07:49 unexpected error fetching cert: transient SSH server error: can’t resolve _orgcert.internal
2024-02-05T07:07:49Z app[90806145c17e28] syd [info]2024/02/05 07:07:49 unexpected error: [ssh: no auth passed yet, transient SSH server error: can’t resolve _orgcert.internal]
2024-02-05T07:08:48Z app[90806145c17e28] syd [info]repmgrd | [2024-02-05 07:08:48] [INFO] monitoring primary node “fdaa:5:c452:a7b:232:a26e:aa50:2” (ID: 658137670) in normal state

I do not know if this is a fly server-side issue or an issue on my w11 laptop.

I do not know what to do next…Need to find another hosting provider as I need to connect my machines in order to do well “anything”.

Tried again this time in git bash. The steps shown below show that the ssh-agent is running. Yet fly cannot see the agent.2761 which was just started.

$ eval “$(ssh-agent -s)”
Agent pid 2762

$ printenv | grep SSH
SSH_AUTH_SOCK=/tmp/ssh-X3yseKCylLi5/agent.2761
SSH_AGENT_PID=2762

$ fly ssh issue --agent -d --overwrite -o personal
Error: can’t connect to SSH agent: dial unix C:/Users/carl/AppData/Local/Temp/ssh-X3yseKCylLi5/agent.2761: connect: No connection could be made because the target machine actively refused it.

I even tried
$ fly status -a helloflydb
ID STATE ROLE REGION CHECKS IMAGE CREATED UPDATED
90806145c17e28 started primary syd 3 total, 3 passing flyio/postgres-flex:15.3 (v0.0.46) 2024-02-03T11:14:42Z 2024-02-03T11:14:58Z

carl@Carls-2-Surface MINGW64 ~/dev/play/hello-fly (main)
$ fly ssh console -a helloflydb -s 90806145c17e28
? Select VM: syd: 90806145c17e28 fdaa:5:c452:a7b:232:a26e:aa50:2 falling-night-117 (primary)
Error: host unavailable at 90806145c17e28: host was not found in DNS

now 10:15 PM deleted all apps and machines from dashboard and deployed with fly deploy -ha=false so I get only one machine.
This is what I get when I try to get to the console. I am lost …

Visit your newly deployed app at https://helloflycarl.fly.dev/
PS C:\Users\carl\dev\play\hello-fly> fly ssh issue
? Select Organization: Carl B Hill (personal)

!!! WARNING: We’re now prompting you to save an SSH private key and certificate !!!
!!! (the private key in “id_whatever” and the certificate in “id_whatever-cert.pub”). !!!
!!! These SSH credentials are time-limited and handling them in files is clunky; !!!
!!! consider running an SSH agent and running this command with --agent. Things !!!
!!! should just sort of work like magic if you do. !!!
? Path to store private key: c:\users\carl.ssh.ssh
Wrote 24-hour SSH credential to c:\users\carl.ssh.ssh, c:\users\carl.ssh.ssh-cert.pub
PS C:\Users\carl\dev\play\hello-fly> ssh-add $env:USERPROFILE.ssh.ssh
Identity added: C:\Users\carl.ssh.ssh (fly.io)
Certificate added: C:\Users\carl.ssh.ssh-cert.pub (fly:org:xxxx:user:xxxxxxx)
PS C:\Users\carl\dev\play\hello-fly> fly console -a helloflycarl
Error: failed to launch VM: To create more than 1 machine per app please add a payment method. Sign In · Fly (Request ID: 01HNWFK8PZK6XHAJY1HVHM9WQ2-sin)

It is now 10:51 PM
I joined the $5/m hobby plan and progressed a bit more, but then get to the same error.
$ fly console -a helloflycarl
Created an ephemeral machine 148e432f9d9689 to run the console.
Connecting to fdaa:5:c452:a7b:a755:4e94:557b:2… complete
Waiting for ephemeral machine 148e432f9d9689 to be destroyed … done.
Error: error connecting to SSH server: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

It is now 11:49 PM
$ fly m start
? Select machines: 5683047a66048e damp-river-5753 (started, region syd, process group ‘app’)
5683047a66048e has been started

carl@Carls-2-Surface MINGW64 ~/dev/play/hello-fly (main)
$ flyctl ssh console 5683047a66048e
Error: host unavailable at 5683047a66048e: host was not found in DNS

It is now 12:53 am
I tried @sudir.j [flyctl ssh console: Waiting for host... and nothing happens - #31 by nickolay.loshkarev] tip for using wireguard to VPN directly into the machine. It works. However, I get disconnected by the server immediately.

ssh -i /c/users/carl/.ssh/.ssh root@fdaa:5:c452:a7b:2985:c3bd:e760:2
The authenticity of host ‘xxxxxxxxx’ can’t be established.
ED25519 key fingerprint is SHA256:xxxxxxxxxx.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added ‘fxxxxxxxxxxxxxxxxxxxxxxxxx’ (ED25519) to the list of known hosts.
Received disconnect from xxxxxx port 22:2: too many authentication failures
Disconnected from fdaa:5:c452:a7b:2985:c3bd:e760:2 port 22

When I run the command I get
$ ssh -i /c/users/carl/.ssh/.ssh root@xxxxxxxxxxxxxxxxxxxxxxxxxxx
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ED25519 key sent by the remote host is
SHA256:xxxxxxxxxxxxxxxxxxxxxx.
Please contact your system administrator.
Add correct host key in /c/Users/carl/.ssh/known_hosts to get rid of this message.
Offending ED25519 key in /c/Users/carl/.ssh/known_hosts:5
Host key for fdaaxxxxxxxxxxxxxxxxxxxxx has changed and you have requested strict checking.
Host key verification failed.

This is a problem on the Fly side, not your laptop.

The organization is defined on Fly’s servers—and is a way that they have of structuring billing and grouping people’s Machines.

(Some of their customers have multiple departments that very strongly prefer to be invoiced separately—as an example.)

When you first start out, you just have one organization:

https://fly.io/organizations

@w8emv is pointing out that this might not have been initialized entirely correctly, when your account was first created by Fly.

There was a broader DNS glitch reported in Signapore recently, so this might be related.

https://community.fly.io/t/fly-deploy-fail-server-misbehaving/18040

(The -sin at the end there says that you’re connecting through Singapore.)


In the meantime, fly proxy is an alternative way in which you can access Postgres to create your tables:

https://fly.io/docs/postgres/connecting/connecting-with-flyctl/

This uses WireGuard only—and not SSH.

(You will need psql or such on your local machine.)

Sorry you’re having so much trouble!

1 Like

Thanks @mayailurus I will try the fly proxy today. I tried again this morning and got the same error. I think I will upgrade my account to the $29 / month so that I can get email support. Otherwise, I will have to move to diff hosting provider (probably AWS).

$ fly ssh issue
? Select Organization: Carl B Hill (personal)

!!! WARNING: We’re now prompting you to save an SSH private key and certificate !!!
!!! (the private key in “id_whatever” and the certificate in “id_whatever-cert.pub”). !!!
!!! These SSH credentials are time-limited and handling them in files is clunky; !!!
!!! consider running an SSH agent and running this command with --agent. Things !!!
!!! should just sort of work like magic if you do. !!!
? Path to store private key: c:\users\carl.ssh.ssh
Wrote 24-hour SSH credential to c:\users\carl.ssh.ssh, c:\users\carl.ssh.ssh-cert.pub

carl@Carls-2-Surface MINGW64 ~/dev/play/hello-fly (main)
$ ssh-add /c/users/carl/.ssh/.ssh
Identity added: /c/users/carl/.ssh/.ssh (fly.io)
Certificate added: /c/users/carl/.ssh/.ssh-cert.pub (fly:org:xxxxx:user:xxxxx)

carl@Carls-2-Surface MINGW64 ~/dev/play/hello-fly (main)
$ fly ssh console
Connecting to fdaa:5:xxxxxxxx… complete
Error: error connecting to SSH server: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

Question: the log shows can’t resolve _orgcert.internal. Should this line say helloflycarl.internal and not _orgcert.internal?

Server log below
2024-02-06T22:21:44Z runner[17811322a07338] syd [info]Machine started in 448ms
2024-02-06T22:21:45Z app[17811322a07338] syd [info]> hello-fly@0.0.0 start
2024-02-06T22:21:45Z app[17811322a07338] syd [info]> node ./bin/www
2024-02-06T22:21:52Z app[17811322a07338] syd [info]2024/02/06 22:21:52 unexpected error fetching cert: transient SSH server error: can’t resolve _orgcert.internal
2024-02-06T22:21:52Z app[17811322a07338] syd [info]2024/02/06 22:21:52 unexpected error: [ssh: no auth passed yet, transient SSH server error: can’t resolve _orgcert.internal]

Now I am getting this error
$ fly m start
? Select machines: 17811322a07338 little-leaf-5135 (started, region syd, process group ‘app’)
17811322a07338 has been started

carl@Carls-2-Surface MINGW64 ~/dev/play/hello-fly (main)
$ fly ssh console helloflycarl
Error: host unavailable at helloflycarl: host was not found in DNS

carl@Carls-2-Surface MINGW64 ~/dev/play/hello-fly (main)
$ fly dashboard
Opening Sign In · Fly

carl@Carls-2-Surface MINGW64 ~/dev/play/hello-fly (main)
$ fly ssh console helloflycarl
Error: host unavailable at helloflycarl: host was not found in DNS

Just realized I have to use the .internal
$ fly ssh console helloflycarl.internal
Connecting to helloflycarl.internal… complete
Error: error connecting to SSH server: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

I can still ping the host
carl@Carls-2-Surface MINGW64 ~/dev/play/hello-fly (main)
$ fly ping helloflycarl.internal
35 bytes from fdaa:5:c452:a7b:1b1:8e3d:6a1b:2 (syd.helloflycarl.internal), seq=0 time=46.8ms
35 bytes from fdaa:5:c452:a7b:1b1:8e3d:6a1b:2 (syd.helloflycarl.internal), seq=1 time=44.1ms

Hi @mayailurus thanks for all your help.
I could not resolve the error.

However, I opened a new account with a different email on fly.io, and I could connect to the console for the very first time.

In my old account, there is some setup that is not correct.

Thank you.

1 Like

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.