No worries, our ssh docs don’t cover everything you can do yet.
DM me your app name and I’ll see what backups we have and we can go from there.
I wasn’t able to scp using the instructions above. I tried running flyctl ips private, but I couldn’t ssh to the IP address from my client machine.
Here’s what I did in case it’s helpful for anyone else.
On my client machine, I installed Wireguard:
sudo apt install -y wireguard-tools resolvconf && \
sudo shutdown -r now
And then I added my local dev machine as a Wireguard peer:
FLY_REGION="ewr" # Change to your app's region
FLY_ORG="personal" # Change to your org
WG_PEER_NAME="devbox" # Change to any name
# When prompted, save to a file, like devbox.conf
fly wireguard create "${FLY_ORG}" "${FLY_REGION}" "${WG_PEER_NAME}"
WG_CONF_FILE="devbox.conf" # Or whatever filename you chose above
sudo cp "${WG_CONF_FILE}" /etc/wireguard && \
wg-quick up "${WG_PEER_NAME}"
From here, you’ll be able to SSH into your VM from your dev machine:
APP_NAME="yourappname" # Change to the name of your fly VM
ssh "root@${APP_NAME}.internal"
In my case, I could ssh but not scp. I didn’t realize this, but for scp to work, the scp binary has to be present on both ends of the connection. My app was running a minimal Docker alpine image, so it didn’t have the scp binary. I fixed that with:
ssh "root@${APP_NAME}.internal" # SSH into my alpine image
apk update
apk add openssh
exit
After that, I was able to scp from my client with:
scp "root@${APP_NAME}.internal:/foo/bar" foo/
7 Likes
Instructions have become a lot simpler now that fly proxy exists.
No need to muck about with wireguard any more:
# In a separate terminal:
fly proxy 10022:22
# In the main terminal:
scp -P 10022 root@localhost:/remote_path/to_file local_path/to_file
7 Likes
In my case I wanted to copy a sqlite database from my local to the volume mounted on the app.
This is what worked for me:
-
fly wireguard create, suffix the profile name with.conf, import into wireguard and connect to it, now you can reach to yourappname.internal -
flyctl ssh issue --agentto add the proper ssh key there, otherwise you get permission denied -
ssh root@appname.internaland install openssh-clientapt-get install openssh-client, otherwise you get “scp command not in $PATH”. -
scp foo.db root@appname.internal:/data/foo.db, for some reason it hang - redeploy the app so it loads the latest file
2 Likes
I gave this a go. In a terminal on my local machine:
# fly proxy 10022:22
Proxying local port 10022 to remote [MY_APP_NAME.internal]:22
Then, in a terminal on my server:
# scp -P 10022 root@localhost:/tmp/foo.txt foo.txt
ssh: connect to host localhost port 10022: Connection refused
Also:
# scp -P 10022 root@MY_APP_NAME.internal:/tmp/foo.txt foo.txt
ssh: connect to host localhost port 10022: Connection refused
I just need to pull down the sqlite db file. Any suggestions?
1 Like
I’m trying to send a local file to the VM but this does not work for me =/
scp data.zip root@my_app.internal:/data
ssh: Could not resolve hostname my_app.internal: Name or service not known
scp: Connection closed
I ran before:
1 - flyctl ssh establish
2 - flyctl ssh issue --agent
But fly ssh console works. 
If run fly ips private it show me like this:
fly ips private
ID REGION IP
e123xpto gru fdaa:0:1122:a11...
I still don’t know what I’m doing wrong.
fly ssh console works where scp doesn’t because flyctl does its own DNS lookups; scp is a native app and can only resolve things your native resolver (in /etc/resolve.conf or whatever) can find. One way to work around this is to use fly dig to get the IPv6 address of the host you want to copy from, and use that instead of my-app.internal.
scp is only going to work if you (1) have an explicit WireGuard tunnel set up — the kind you start up with the WireGuard client or wg-quick and (2) are running an ssh-agent, and then ran flyctl ssh issue --agent.
This is, of course, supremely clunky and unpleasant (it’s fine if you’re going to be SSH’ing to things all the time, and do the setup just once). We’re playing with baking sftp directly into flyctl right now.
2 Likes
Not sure if this will help but I had to run fly ssh issue but without --agent which generated keys. Using a config like the following for an app appname scp is now working.
Host appname
User root
HostName appname.internal
Port 22
IdentityFile ~/.ssh/id_appname
StrictHostKeyChecking no
UserKnownHostsFile /dev/null
PubkeyAuthentication yes
IdentitiesOnly yes
I tried too, ssh-ing works fine, scp throws error “Connection to localhost closed by remote host.”. Tried installing scp binary to no effect. I thought maybe I should restart sshd service but systemctl doesn’t seem to work: “System has not been booted with systemd as init system (PID 1). Can’t operate. Failed to connect to bus: Host is down”. Then I tried to do apt install openssh-client in the Docker image with no effect.
Here is my unsuccessful attempt:
$ fly proxy 10022:22
and in another console:
$ fly ssh issue
$ scp -v -i id_windworld -P 10022 root@localhost:/data/windworld.db ww.db
Executing: program /usr/bin/ssh host localhost, user root, command sftp
OpenSSH_9.0p1, OpenSSL 1.1.1q 5 Jul 2022
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Connecting to localhost [::1] port 10022.
debug1: connect to address ::1 port 10022: Connection refused
debug1: Connecting to localhost [127.0.0.1] port 10022.
debug1: Connection established.
debug1: identity file id_windworld type 3
debug1: identity file id_windworld-cert type 7
debug1: Local version string SSH-2.0-OpenSSH_9.0
debug1: Remote protocol version 2.0, remote software version Go
debug1: compat_banner: no match: Go
debug1: Authenticating to localhost:10022 as 'root'
debug1: load_hostkeys: fopen /home/grfork/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host key: ssh-ed25519 SHA256:1XGryNxflX9P4SKI8sp8/C3YPVgR79AnWzAOe98f5pY
debug1: load_hostkeys: fopen /home/grfork/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: Host '[localhost]:10022' is known and matches the ED25519 host key.
debug1: Found key in /home/grfork/.ssh/known_hosts:16
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
debug1: get_agent_identities: agent returned 1 keys
debug1: Will attempt key: (none) RSA SHA256:hSJsG29I25UZ6e8wbA5rpWLHPDbKWRhuH3PJNAGYxPI agent
debug1: Will attempt key: id_windworld ED25519 SHA256:ySJYo2Nop9CIATSKtlig5ew8fxICIZqQfE8g4dsTGSg explicit
debug1: Will attempt key: id_windworld ED25519-CERT SHA256:ySJYo2Nop9CIATSKtlig5ew8fxICIZqQfE8g4dsTGSg explicit
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering public key: (none) RSA SHA256:hSJsG29I25UZ6e8wbA5rpWLHPDbKWRhuH3PJNAGYxPI agent
debug1: send_pubkey_test: no mutual signature algorithm
debug1: Offering public key: id_windworld ED25519 SHA256:ySJYo2Nop9CIATSKtlig5ew8fxICIZqQfE8g4dsTGSg explicit
debug1: Authentications that can continue: publickey
debug1: Offering public key: id_windworld ED25519-CERT SHA256:ySJYo2Nop9CIATSKtlig5ew8fxICIZqQfE8g4dsTGSgexplicit
debug1: Server accepts key: id_windworld ED25519-CERT SHA256:ySJYo2Nop9CIATSKtlig5ew8fxICIZqQfE8g4dsTGSg explicit
Authenticated to localhost ([127.0.0.1]:10022) using "publickey".
debug1: channel 0: new [client-session]
debug1: Entering interactive session.
debug1: pledge: filesystem
debug1: Sending subsystem: sftp
debug1: channel 0: free: client-session, nchannels 1
Connection to localhost closed by remote host.
Transferred: sent 2868, received 1288 bytes, in 0.2 seconds
Bytes per second: sent 17994.5, received 8081.2
debug1: Exit status -1
scp: Connection closed
1 Like
This is really burdensome for developers who are unfamiliar with the intricacies of WireGuard (probably most developers, myself included). +1 to adding sftp-that-just-works to flyctl.
I tried this, but ssh root@appname.internal returns Permission denied (publickey) me.
Thank you so much for this! I had no idea how to make the original answer work. This here works perfect!
The recent versions of fly CLI support SFTP shell out of the box, so there is no need to do it manually anymore. Just run:
> flyctl ssh sftp shell
2 Likes
Hello @qqwy — I’m able to use fly ssh console, but when I try to use your proxy & scp example, I get Permission denied (publickey). Were there other steps that you did before hand that might be necessary? I’m probably missing something super obvious…
Thanks!
Update to the latest flyctl, then try the proxy command after executing flyctl wg reset --org <org-name>?
See also: Deployments not working: error connecting to docker - #60 by mfilej
1 Like
Thanks @ignoramous. I think I’m stuck on ssh key issues, though I’m puzzled why ssh console works and scp doesn’t. When I run with -vvv, it sends my id_rsa public key, but returns receive packet: type 51, which I think means it rejected it. I have checked my permissions, and generated fresh keys, without luck. I’m probably missing something blindingly obvious…
Strange stuff.
Do you happen to run apps in multiple orgs? If so, this post may have some pointers: Fly-Builder can't connect to Docker - #5 by mhostetler
1 Like
Thanks for your help @ignoramous. I do, and I followed the instructions in that post, but no luck. fly doctor gives:
Testing authentication token... PASSED
Testing flyctl agent... PASSED
Testing local Docker instance... Nope
Pinging WireGuard gateway (give us a sec)... PASSED
App specific checks for appname:
Checking that app has ip addresses allocated... PASSED
Checking AAAA record for appname.fly.dev... PASSED
Build checks for appname:
Checking docker context size (this may take little bit)... PASSED (92 MB)
I got it working!
I worked off Using rsync on an app instace, but it had the exact same instructions I’d tried before; however, in the course of things, I ran across two other commands that seemed to have done the trick.
These were the commands that were needed:
# In one terminal
flyctl proxy 10022:22
# In another, SSH in
fly ssh console
# install scp on the remote (alpine linux)
apk add openssh
These commands seemed to be necessary in order for flyctl ssh issue --agent to work:
ssh-agent bash
ssh-keygen -f "/home/USERNAME/.ssh/known_hosts" -R "[localhost]:10022"
flyctl ssh issue --agent
After that, I could use scp to copy files to and from the persistent volume:
# Backup ghost content
scp -r -P 10022 root@localhost:/var/lib/ghost/content/ .
# Restore relevant content
cd content/
scp -r -P 10022 images/ themes/ root@localhost:/var/lib/ghost/content/
Once I better understand why it worked this time, I’ll turn this into a proper tutorial for people who were as baffled as I was…
1 Like