[experimental] Speedy machine creation with overlaybd

Fresh Produce
1

Many docker images are able to start doing useful work before the entire image is downloaded. If you have a particularly large image (e.g. >1GB, or if you’re running ML workloads), this can have a real impact on how you decide to architect your application.

We’ve been working on a way to make new machine creation faster, using something called overlaybd. overlaybd is new container image format that allows us to delay downloading parts of the docker image until they are actually needed. When converting a docker image to this new format, each layer is turned into a block device that stores the changed blocks from the previous layer. overlaybd then merges the layers together, and can randomly access individual blocks in the remote container registry. It runs as a userspace daemon, exposing a virtual block device via TCMU, and only downloads blocks when they are read. We pass it along to firecracker as the rootfs for the machine.

To try out lazy-loaded images:

  1. Make sure you are on flyctl >= v0.2.25
$ fly version
fly v0.2.25 darwin/arm64 Commit: 358c5fcbcbf3b9edfab38ba8f5e305c8b786231e BuildDate: 2024-03-26T04:41:20Z
  1. Delete your remote builder app, as we have released a new version.
$ fly apps list | grep builder
fly-builder-bitter-snow-4886    	personal      	suspended	                 	
$ fly apps destroy fly-builder-bitter-snow-4886
  1. Enable lazy-loaded images in your fly.toml
[experimental]
lazy_load_images = true
  1. Run fly deploy
  2. Watch fly logs to see how long it takes to start the machine

This will probably work best for you if you have a large image (>1GB), and you care about machine creation time (e.g. if you maintain a warm pool of machines). The less amount of data you need to start up, the faster it should be.

This is an experimental, alpha-level feature. Once you try it out, we’d really like to know if it works for you, if you find any bugs, and how it performs with your docker images.

13 Likes
2

@ben-io hey, crazy idea but, any chance this would allow support for 60GB docker images?

3

Actually, it probably does already. What’s making your image so large? Note that if you are running a GPU machine we give you a 50GB rootfs instead of 8GB, which is where the image size limitation comes from.

1 Like
4

Just been exploring preloading models into the docker image at build time so that they’re ready to go after machine startup.

I’ve been looking at possibly using ollama with mixtral preloaded and avoiding the need to create volumes for each machine.

1 Like