A versatile and sunlit bridge to a bygone era, recursive and incremental file transfer endures long in people’s affections. Well, among other things, it is nice to upload entire directory hierarchies in a single invocation. While looking for easier ways to do this here, it became apparent that there is a variant within the synch-to-machines genre that has not yet been covered in detail in the forums.
The classic and easist way of doing rsync uses OpenSSH as a subprocess, sending commands and receiving responses over the latter’s stdin and stdout. Since Fly does of course have its own built-in SSH facility, this seems like it should just work as a drop-in subprocess—and hence avoid the auxiliary and fragile structure of a backgrounded fly proxy tunnel, etc.
The catch is that rsync expects the secure-shell subcommand to use the ssh host x y z syntax, whereas fly ssh wants the last three to be concatenated into a single -C string.
The workaround is to create a small Bash script locally:
#!/bin/bash -eup
ADDR=$1; shift
# the extremely strange syntax at the end handles quoting of spaces
# and other special characters...
fly ssh console --address "$ADDR" -C "${*@Q}"
(Be sure to chmod u+x it, of course.)
Once you have that, plus rsync installed both locally and on your Fly machine, then you can just…
rsync -rplitz -e ~/that-script \
a-local/dir/ \
machine-hex-id.vm.your-app-name.internal:/path/on/server/
The remainder of this post is a detailed worked example, geared toward Debian Linux on the local machine.
Local
Rsync is a cooperative protocol, and its binary needs to be installed on both sides:
$ sudo apt-get update
$ sudo apt-get install --no-install-recommends rsync
Copy the shell script at the top of this article into ~/ssh-q and then…
$ chmod u+x ~/ssh-q
And a small directory for transfer testing:
$ cd ~
$ mkdir upload-fodder
$ cd upload-fodder
$ echo a > "tomato's skin"
$ echo b > "rampantly"
$ echo c > "sunlit frolic"
$ mkdir --parents subdir/will-be-recursive
$ echo x > subdir/will-be-recursive/scintillation
Fly Machine
For completeness, I’ll show an entire Dockerfile and fly.toml, but generally you would instead merge its apt-get install clause into your own.
$ mkdir ~/rsync-example
$ cd ~/rsync-example
Create the following Dockerfile:
ARG DEBIAN_VERSION=bullseye-20210902-slim
ARG RUNNER_IMAGE="debian:${DEBIAN_VERSION}"
FROM ${RUNNER_IMAGE}
RUN apt-get update -y \
&& apt-get install -y --no-install-recommends rsync \
&& apt-get clean && rm -f /var/lib/apt/lists/*_*
# note: there is no rsync server running persistently on your
# server. instead, a temporary helper process is started by ssh,
# and it terminates on its own once the transfer is complete...
CMD ["sleep", "inf"]
And then a tiny fly.toml…
app = "vibrant-parsnip-rhapsody"
primary_region = "phx"
[mounts]
source = "garden"
destination = "/uploaded"
# Note: do *not* need to open any ports.
And deployment is of course…
$ fly app create vibrant-parsnip-rhapsody
$ fly vol create garden --size 1
$ fly deploy --ha=false
Now we can try uploading a directory:
$ fly m list
$ rsync -rplitz -e ~/ssh-q \
~/upload-fodder/ \
abcd091190fedc.vm.vibrant-parsnip-rhapsody.internal:/uploaded/v/
# ^ the long hex string is the ID from the list.
#
# (in this case, we could have gotten away with just
# vibrant-parsnip-rhapsody.internal, which is unambiguous
# when there is only one machine.)
Note: It’s important to remember both the trailing slashes (/) on the two directory paths and the -r (recursive) flag.
Verifying success…
$ fly ssh console -a vibrant-parsnip-rhapsody
# cd /uploaded/v
# find .
.
./tomato's skin
./rampantly
./sunlit frolic
./subdir
./subdir/will-be-recursive
./subdir/will-be-recursive/scintillation
Be sure to read the rsync’s documentation on the --owner and --delete directives, since there isn’t a one-size-fits-all recommendation there.
Hope this helps!
