Try out faster machine creation with zstd

Fresh Produce
1

If you’re creating new machines often (perhaps on demand with users’ requests), and you’re using big docker images, you might be interested in this. When creating a new machine, a majority of the time is spent downloading and unpacking the docker image, especially if it’s a couple of GB or greater. By default, docker uses gzip to compress the image layers when uploading, but there’s a newer, better compression algorithm known as zstd . zstd-compressed files are usually smaller than the equivalent gzip-compressed files, and they can be decompressed faster. We now support running zstd-compressed images, so if you’d like to try this out to be able to create machines faster, follow the instructions below.

Make sure you have docker installed locally, version v20.10.23 or newer.

For the initial set up, you’ll need to authenticate your local docker client with our registry and create a new buildkit builder:

$ fly auth docker
$ docker buildx create --name zstd-builder --driver docker-container
$ docker buildx use zstd-builder

docker builder ls should indicate that you are running at least version v0.10.0 of buildkit.

Update your fly.toml to use the new image we will build:

[build]
  image = "registry.fly.io/$APP:$VERSION"

To build, compress, and upload a new docker image, run:

$ docker buildx build \
  --file Dockerfile \
  --output type=image,name=registry.fly.io/$APP:$VERSION,oci-mediatypes=true,compression=zstd,compression-level=3,force-compression=true,push=true .

You can validate that you’ve successfully pushed a zstd-compressed image with:

$ docker manifest inspect -v registry.fly.io/$APP:$VERSION | grep "application/vnd.oci.image.layer.v1.tar+zstd"

If you see "application/vnd.docker.image.rootfs.diff.tar.gzip" instead, it is using gzip for compression. This can happen if you try to push layers that have already been pushed previously.

Finally, deploy your app:

$ fly deploy

Please drop a comment if you try it out to let me know how it goes! We hope to integrate it into flyctl by default in the future.

6 Likes
2

I’m trying to implement this to improve machine creation on my app but I have some questions.

  1. Are we supposed to populate $APP and $VERSION ourselves manually? O it should be done automatically? This is certainly not required when doing regular deployments.

  2. I understand this is building the image locally and then pushing it to Fly’s registry. Is this strategy compatible with Depot which is now the default building mechanism? If so, how?

  3. If this is not compatible with Depot, can it be implemented in a GitHub Action?

  4. Is this performs better than standard gzip compression, are you planning to make this the default behaviour and integrate it with the Depot-based builders?

3

It looks like Depot does support zstd compression:

Can we have this supported in Fly natively? Maybe as an option in fly.toml config?