Uploading entire directories with fly ssh console + rsync

Show & Tell
1

A versatile and sunlit bridge to a bygone era, recursive and incremental file transfer endures long in people’s affections. Well, among other things, it is nice to upload entire directory hierarchies in a single invocation. While looking for easier ways to do this here, it became apparent that there is a variant within the synch-to-machines genre that has not yet been covered in detail in the forums.

rsync over fly ssh console's stdin/stdout

The classic and easist way of doing rsync uses OpenSSH as a subprocess, sending commands and receiving responses over the latter’s stdin and stdout. Since Fly does of course have its own built-in SSH facility, this seems like it should just work as a drop-in subprocess—and hence avoid the auxiliary and fragile structure of a backgrounded fly proxy tunnel, etc.

The catch is that rsync expects the secure-shell subcommand to use the ssh host x y z syntax, whereas fly ssh wants the last three to be concatenated into a single -C string.

The workaround is to create a small Bash script locally:

#!/bin/bash -eup

ADDR=$1;  shift

# the extremely strange syntax at the end handles quoting of spaces
# and other special characters...
fly ssh console --address "$ADDR" -C "${*@Q}"

(Be sure to chmod u+x it, of course.)

Once you have that, plus rsync installed both locally and on your Fly machine, then you can just…

rsync -rplitz -e ~/that-script \
  a-local/dir/ \
  machine-hex-id.vm.your-app-name.internal:/path/on/server/

The remainder of this post is a detailed worked example, geared toward Debian Linux on the local machine.

Local

Rsync is a cooperative protocol, and its binary needs to be installed on both sides:

$ sudo apt-get update
$ sudo apt-get install --no-install-recommends rsync

Copy the shell script at the top of this article into ~/ssh-q and then…

$ chmod u+x ~/ssh-q

And a small directory for transfer testing:

$ cd ~
$ mkdir upload-fodder
$ cd upload-fodder
$ echo a > "tomato's skin"
$ echo b > "rampantly"
$ echo c > "sunlit frolic"
$ mkdir --parents subdir/will-be-recursive
$ echo x > subdir/will-be-recursive/scintillation

Fly Machine

For completeness, I’ll show an entire Dockerfile and fly.toml, but generally you would instead merge its apt-get install clause into your own.

$ mkdir ~/rsync-example
$ cd ~/rsync-example

Create the following Dockerfile:

ARG DEBIAN_VERSION=bullseye-20210902-slim

ARG RUNNER_IMAGE="debian:${DEBIAN_VERSION}"

FROM ${RUNNER_IMAGE}

RUN apt-get update -y \
   && apt-get install -y --no-install-recommends rsync \
   && apt-get clean && rm -f /var/lib/apt/lists/*_*

# note:  there is no rsync server running persistently on your
# server.  instead, a temporary helper process is started by ssh,
# and it terminates on its own once the transfer is complete...
CMD ["sleep", "inf"]

And then a tiny fly.toml…

app = "vibrant-parsnip-rhapsody"
primary_region = "phx"

[mounts]
  source = "garden"
  destination = "/uploaded"

# Note:  do *not* need to open any ports.

And deployment is of course…

$ fly app create vibrant-parsnip-rhapsody
$ fly vol create garden --size 1
$ fly deploy --ha=false

Now we can try uploading a directory:

$ fly m list
$ rsync -rplitz -e ~/ssh-q \
    ~/upload-fodder/ \
    abcd091190fedc.vm.vibrant-parsnip-rhapsody.internal:/uploaded/v/
      # ^ the long hex string is the ID from the list.
      #
      #   (in this case, we could have gotten away with just
      #    vibrant-parsnip-rhapsody.internal, which is unambiguous
      #    when there is only one machine.)

Note: It’s important to remember both the trailing slashes (/) on the two directory paths and the -r (recursive) flag.

Verifying success…

$ fly ssh console -a vibrant-parsnip-rhapsody
# cd /uploaded/v
# find .
.
./tomato's skin
./rampantly
./sunlit frolic
./subdir
./subdir/will-be-recursive
./subdir/will-be-recursive/scintillation

Be sure to read the rsync’s documentation on the --owner and --delete directives, since there isn’t a one-size-fits-all recommendation there.

Hope this helps!

:tomato: :sparkle: :tomato: :sparkle:

7 Likes
2

Thanks, this was really helpful! I was kinda struggling to automate my deploy that depends on a volume using fly sftp… This saved the day :smiley:

1 Like
3

I had a few problems getting this going.

  1. This -C "${*@Q}" does not work on Mac’s shell.
  2. I think the way fly ssh works is outdated. You need to pass the MACHINE_ID not the url.

Here is what works for me:

#!/bin/bash
# Pass this to rysnc as the -e option to use fly ssh.
# You need to pass the machine id as server first argument.
# example:
#   rsync -rltzPi --delete -e ./scripts/fly-rsync-helper ./data/ MACHINE_ID:/data/

set -euo pipefail
MACHINE="$1"
shift
CMD=$(printf " %q" "$@")
fly ssh console --quiet --machine "$MACHINE" -C "$CMD"

So, I end up doing something like this:

  #!/bin/bash
  set -euo pipefail

  # 1) Gather all machine IDs for this app
  MACHINES=$(fly machine list --json | jq -r '.[].id')
  if [ -z "$MACHINES" ]; then
    echo "❌ No machines found for this app"
    exit 1
  fi

  # 2) Start each machine (idempotent if already running)
  echo "🔄 Starting machines..."
  for M in $MACHINES; do
    echo "⏳ Starting $M"
    fly machine start "$M"
  done

  # 3) Give them a moment to come online
  sleep 5

  # 4) Upload via rsync + fly-ssh wrapper
  LOCAL_PATH="./data"
  REMOTE_PATH="/data"
  echo "📡 Uploading to all machines..."
  for M in $MACHINES; do
    echo "➡️  $M"
    rsync -rltzPi --delete \
      -e ./deploy/fly-rsync-helper \
      ./data/ \
      "${M}":/data/
  done

  echo "✅ Upload complete for all machines."
4

Fun hack, I went with running sshd in my fly machines, and then I just have a script that parses the output from tailscale status --json and does the rsync.