Uploading entire directories with fly ssh console + rsync

A versatile and sunlit bridge to a bygone era, recursive and incremental file transfer endures long in people’s affections. Well, among other things, it is nice to upload entire directory hierarchies in a single invocation. While looking for easier ways to do this here, it became apparent that there is a variant within the synch-to-machines genre that has not yet been covered in detail in the forums.

rsync over fly ssh console's stdin/stdout

The classic and easist way of doing rsync uses OpenSSH as a subprocess, sending commands and receiving responses over the latter’s stdin and stdout. Since Fly does of course have its own built-in SSH facility, this seems like it should just work as a drop-in subprocess—and hence avoid the auxiliary and fragile structure of a backgrounded fly proxy tunnel, etc.

The catch is that rsync expects the secure-shell subcommand to use the ssh host x y z syntax, whereas fly ssh wants the last three to be concatenated into a single -C string.

The workaround is to create a small Bash script locally:

#!/bin/bash -eup

ADDR=$1;  shift

# the extremely strange syntax at the end handles quoting of spaces
# and other special characters...
fly ssh console --address "$ADDR" -C "${*@Q}"

(Be sure to chmod u+x it, of course.)

Once you have that, plus rsync installed both locally and on your Fly machine, then you can just…

rsync -rplitz -e ~/that-script \
  a-local/dir/ \
  machine-hex-id.vm.your-app-name.internal:/path/on/server/

The remainder of this post is a detailed worked example, geared toward Debian Linux on the local machine.

Local

Rsync is a cooperative protocol, and its binary needs to be installed on both sides:

$ sudo apt-get update
$ sudo apt-get install --no-install-recommends rsync

Copy the shell script at the top of this article into ~/ssh-q and then…

$ chmod u+x ~/ssh-q

And a small directory for transfer testing:

$ cd ~
$ mkdir upload-fodder
$ cd upload-fodder
$ echo a > "tomato's skin"
$ echo b > "rampantly"
$ echo c > "sunlit frolic"
$ mkdir --parents subdir/will-be-recursive
$ echo x > subdir/will-be-recursive/scintillation

Fly Machine

For completeness, I’ll show an entire Dockerfile and fly.toml, but generally you would instead merge its apt-get install clause into your own.

$ mkdir ~/rsync-example
$ cd ~/rsync-example

Create the following Dockerfile:

ARG DEBIAN_VERSION=bullseye-20210902-slim

ARG RUNNER_IMAGE="debian:${DEBIAN_VERSION}"

FROM ${RUNNER_IMAGE}

RUN apt-get update -y \
   && apt-get install -y --no-install-recommends rsync \
   && apt-get clean && rm -f /var/lib/apt/lists/*_*

# note:  there is no rsync server running persistently on your
# server.  instead, a temporary helper process is started by ssh,
# and it terminates on its own once the transfer is complete...
CMD ["sleep", "inf"]

And then a tiny fly.toml

app = "vibrant-parsnip-rhapsody"
primary_region = "phx"

[mounts]
  source = "garden"
  destination = "/uploaded"

# Note:  do *not* need to open any ports.

And deployment is of course…

$ fly app create vibrant-parsnip-rhapsody
$ fly vol create garden --size 1
$ fly deploy --ha=false

Now we can try uploading a directory:

$ fly m list
$ rsync -rplitz -e ~/ssh-q \
    ~/upload-fodder/ \
    abcd091190fedc.vm.vibrant-parsnip-rhapsody.internal:/uploaded/v/
      # ^ the long hex string is the ID from the list.
      #
      #   (in this case, we could have gotten away with just
      #    vibrant-parsnip-rhapsody.internal, which is unambiguous
      #    when there is only one machine.)

Note: It’s important to remember both the trailing slashes (/) on the two directory paths and the -r (recursive) flag.

Verifying success…

$ fly ssh console -a vibrant-parsnip-rhapsody
# cd /uploaded/v
# find .
.
./tomato's skin
./rampantly
./sunlit frolic
./subdir
./subdir/will-be-recursive
./subdir/will-be-recursive/scintillation

Be sure to read the rsync’s documentation on the --owner and --delete directives, since there isn’t a one-size-fits-all recommendation there.

Hope this helps!

:tomato: :sparkle: :tomato: :sparkle:

3 Likes