I need to copy the WAL files from postgres vm off the instance.
I have the files copied to a temp directory on the volume (so they don’t get removed). Now trying to figure out how to get them back to a local machine. SCP is not installed, but I assume I can install it, just now sure how that’s going to play with the fly ssh console (how I accessed the VM to start with).
Alternatively if you are looking for someone to test postgres volume snapshots you are doing, I had my own backup but it was a dump which I have restored, but missing about 3 hours of data from the dump
And then I added my local dev machine as a Wireguard peer:
FLY_REGION="ewr" # Change to your app's region
FLY_ORG="personal" # Change to your org
WG_PEER_NAME="devbox" # Change to any name
# When prompted, save to a file, like devbox.conf
fly wireguard create "${FLY_ORG}" "${FLY_REGION}" "${WG_PEER_NAME}"
WG_CONF_FILE="devbox.conf" # Or whatever filename you chose above
sudo cp "${WG_CONF_FILE}" /etc/wireguard && \
wg-quick up "${WG_PEER_NAME}"
From here, you’ll be able to SSH into your VM from your dev machine:
APP_NAME="yourappname" # Change to the name of your fly VM
ssh "root@${APP_NAME}.internal"
In my case, I could ssh but not scp. I didn’t realize this, but for scp to work, the scp binary has to be present on both ends of the connection. My app was running a minimal Docker alpine image, so it didn’t have the scp binary. I fixed that with:
ssh "root@${APP_NAME}.internal" # SSH into my alpine image
apk update
apk add openssh
exit
After that, I was able to scp from my client with:
fly ssh console works where scp doesn’t because flyctl does its own DNS lookups; scp is a native app and can only resolve things your native resolver (in /etc/resolve.conf or whatever) can find. One way to work around this is to use fly dig to get the IPv6 address of the host you want to copy from, and use that instead of my-app.internal.
scp is only going to work if you (1) have an explicit WireGuard tunnel set up — the kind you start up with the WireGuard client or wg-quick and (2) are running an ssh-agent, and then ran flyctl ssh issue --agent.
This is, of course, supremely clunky and unpleasant (it’s fine if you’re going to be SSH’ing to things all the time, and do the setup just once). We’re playing with baking sftp directly into flyctl right now.
Not sure if this will help but I had to run fly ssh issue but without--agent which generated keys. Using a config like the following for an app appname scp is now working.
Host appname
User root
HostName appname.internal
Port 22
IdentityFile ~/.ssh/id_appname
StrictHostKeyChecking no
UserKnownHostsFile /dev/null
PubkeyAuthentication yes
IdentitiesOnly yes
I tried too, ssh-ing works fine, scp throws error “Connection to localhost closed by remote host.”. Tried installing scp binary to no effect. I thought maybe I should restart sshd service but systemctl doesn’t seem to work: “System has not been booted with systemd as init system (PID 1). Can’t operate. Failed to connect to bus: Host is down”. Then I tried to do apt install openssh-client in the Docker image with no effect.
Here is my unsuccessful attempt:
$ fly proxy 10022:22
and in another console:
$ fly ssh issue
$ scp -v -i id_windworld -P 10022 root@localhost:/data/windworld.db ww.db
Executing: program /usr/bin/ssh host localhost, user root, command sftp
OpenSSH_9.0p1, OpenSSL 1.1.1q 5 Jul 2022
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Connecting to localhost [::1] port 10022.
debug1: connect to address ::1 port 10022: Connection refused
debug1: Connecting to localhost [127.0.0.1] port 10022.
debug1: Connection established.
debug1: identity file id_windworld type 3
debug1: identity file id_windworld-cert type 7
debug1: Local version string SSH-2.0-OpenSSH_9.0
debug1: Remote protocol version 2.0, remote software version Go
debug1: compat_banner: no match: Go
debug1: Authenticating to localhost:10022 as 'root'
debug1: load_hostkeys: fopen /home/grfork/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host key: ssh-ed25519 SHA256:1XGryNxflX9P4SKI8sp8/C3YPVgR79AnWzAOe98f5pY
debug1: load_hostkeys: fopen /home/grfork/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: Host '[localhost]:10022' is known and matches the ED25519 host key.
debug1: Found key in /home/grfork/.ssh/known_hosts:16
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
debug1: get_agent_identities: agent returned 1 keys
debug1: Will attempt key: (none) RSA SHA256:hSJsG29I25UZ6e8wbA5rpWLHPDbKWRhuH3PJNAGYxPI agent
debug1: Will attempt key: id_windworld ED25519 SHA256:ySJYo2Nop9CIATSKtlig5ew8fxICIZqQfE8g4dsTGSg explicit
debug1: Will attempt key: id_windworld ED25519-CERT SHA256:ySJYo2Nop9CIATSKtlig5ew8fxICIZqQfE8g4dsTGSg explicit
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering public key: (none) RSA SHA256:hSJsG29I25UZ6e8wbA5rpWLHPDbKWRhuH3PJNAGYxPI agent
debug1: send_pubkey_test: no mutual signature algorithm
debug1: Offering public key: id_windworld ED25519 SHA256:ySJYo2Nop9CIATSKtlig5ew8fxICIZqQfE8g4dsTGSg explicit
debug1: Authentications that can continue: publickey
debug1: Offering public key: id_windworld ED25519-CERT SHA256:ySJYo2Nop9CIATSKtlig5ew8fxICIZqQfE8g4dsTGSgexplicit
debug1: Server accepts key: id_windworld ED25519-CERT SHA256:ySJYo2Nop9CIATSKtlig5ew8fxICIZqQfE8g4dsTGSg explicit
Authenticated to localhost ([127.0.0.1]:10022) using "publickey".
debug1: channel 0: new [client-session]
debug1: Entering interactive session.
debug1: pledge: filesystem
debug1: Sending subsystem: sftp
debug1: channel 0: free: client-session, nchannels 1
Connection to localhost closed by remote host.
Transferred: sent 2868, received 1288 bytes, in 0.2 seconds
Bytes per second: sent 17994.5, received 8081.2
debug1: Exit status -1
scp: Connection closed
This is really burdensome for developers who are unfamiliar with the intricacies of WireGuard (probably most developers, myself included). +1 to adding sftp-that-just-works to flyctl.
Hello @qqwy — I’m able to use fly ssh console, but when I try to use your proxy & scp example, I get Permission denied (publickey). Were there other steps that you did before hand that might be necessary? I’m probably missing something super obvious…
