Large image failing to deploy

1

I’m attempting to deploy a large image (1.1GB) to Fly which consists of files for Linux package repositories + a static webserver to host them (using the “static” builtin). However, I’m receiving the following error when deploying:

flyctl deploy public
Deploying notion-linux
==> Validating App Configuration
--> Validating App Configuration done
Services
TCP 80/443 ⇢ 8080
Deploy source directory '/tmp/cirrus-ci-build/public'
==> Building with Builtin
Using Builtin Builder: static
INFO Local docker unavailable, hooking you up with a remote Docker builder...
INFO Waiting for remote builder to become available...
INFO Remote builder is ready to build!
Step 1/3 : FROM pierrezemb/gostatic
latest: Pulling from pierrezemb/gostatic
63b6fbada7f9: Pulling fs layer
63b6fbada7f9: Verifying Checksum
63b6fbada7f9: Download complete
63b6fbada7f9: Pull complete
Digest: sha256:cedf9cef1cf6452c4f294d1c9c2334075282a7cd34412ca216699e38aa545b17
Status: Downloaded newer image for pierrezemb/gostatic:latest
---> 098af51e79a9
Step 2/3 : COPY . /srv/http/
---> cc23af742873
Step 3/3 : CMD ["-port","8080"]
---> Running in 90265496322b
---> 5b6cebf63d0c
Successfully built 5b6cebf63d0c
Successfully tagged registry.fly.io/notion-linux:deployment-1613181632
--> Building with Builtin done
Image: registry.fly.io/notion-linux:deployment-1613181632
Image size: 1.1 GB
==> Pushing Image
The push refers to repository [registry.fly.io/notion-linux]
9d0bcaae547d: Preparing
ba79b3b261ef: Preparing
ba79b3b261ef: Layer already exists
9d0bcaae547d: Retrying in 5 seconds
9d0bcaae547d: Retrying in 4 seconds
9d0bcaae547d: Retrying in 3 seconds
9d0bcaae547d: Retrying in 2 seconds
9d0bcaae547d: Retrying in 1 second
9d0bcaae547d: Retrying in 10 seconds
9d0bcaae547d: Retrying in 9 seconds
9d0bcaae547d: Retrying in 8 seconds
9d0bcaae547d: Retrying in 7 seconds
9d0bcaae547d: Retrying in 6 seconds
9d0bcaae547d: Retrying in 5 seconds
9d0bcaae547d: Retrying in 4 seconds
9d0bcaae547d: Retrying in 3 seconds
9d0bcaae547d: Retrying in 2 seconds
9d0bcaae547d: Retrying in 1 second
9d0bcaae547d: Retrying in 15 seconds
9d0bcaae547d: Retrying in 14 seconds
9d0bcaae547d: Retrying in 13 seconds
9d0bcaae547d: Retrying in 12 seconds
9d0bcaae547d: Retrying in 11 seconds
9d0bcaae547d: Retrying in 10 seconds
9d0bcaae547d: Retrying in 9 seconds
9d0bcaae547d: Retrying in 8 seconds
9d0bcaae547d: Retrying in 7 seconds
9d0bcaae547d: Retrying in 6 seconds
9d0bcaae547d: Retrying in 5 seconds
9d0bcaae547d: Retrying in 4 seconds
9d0bcaae547d: Retrying in 3 seconds
9d0bcaae547d: Retrying in 2 seconds
9d0bcaae547d: Retrying in 1 second
9d0bcaae547d: Retrying in 20 seconds
9d0bcaae547d: Retrying in 19 seconds
9d0bcaae547d: Retrying in 18 seconds
9d0bcaae547d: Retrying in 17 seconds
9d0bcaae547d: Retrying in 16 seconds
9d0bcaae547d: Retrying in 15 seconds
9d0bcaae547d: Retrying in 14 seconds
9d0bcaae547d: Retrying in 13 seconds
9d0bcaae547d: Retrying in 12 seconds
9d0bcaae547d: Retrying in 11 seconds
9d0bcaae547d: Retrying in 10 seconds
9d0bcaae547d: Retrying in 9 seconds
9d0bcaae547d: Retrying in 8 seconds
9d0bcaae547d: Retrying in 7 seconds
9d0bcaae547d: Retrying in 6 seconds
9d0bcaae547d: Retrying in 5 seconds
9d0bcaae547d: Retrying in 4 seconds
9d0bcaae547d: Retrying in 3 seconds
9d0bcaae547d: Retrying in 2 seconds
9d0bcaae547d: Retrying in 1 second
Error unexpected EOF

I’m deploying from Cirrus CI; you can also see the output I’ve pasted above here: Cirrus CI

My fly.toml is also available here: notion-linux/fly.toml at main · davidbailey00/notion-linux · GitHub

Any ideas what’s going wrong? Presumably the layer upload is failing for some reason?

2

This could just be a temporary network issue. Can you retry the build?

I’m going to investigate on our side of things.

3

Okay, I am retrying the build now: Cirrus CI

4

It does appear our registry struggles with the 1.1GB layer (the base image is only 3-4MB).

I’m looking at various logs to see where the bottleneck is.

5

This might take a little while to fix properly, but I have an idea for a temporary solution.

You might be able to forego the static builtin and use a Dockerfile like this:

FROM pierrezemb/gostatic
COPY ./debs /srv/http/debs
COPY ./rpms /srv/http/rpms
COPY notion-linux.list notion-linux.repo /srv/http/
CMD ["-port","8080", "-https-promote"]

If you put these lines as a Dockerfile in ./public, it might just work. I wrote it without testing, so you might have to tweak it.

What this does is split the huge layer in 2 smaller ones. I think the issue is: a request timeout is reached because no bytes are sent back from our registry until the full upload is done (the last time, it took 125 seconds).

6

Thanks for the suggestion!

As an alternative, I’m considering creating a volume and using a temporary app to prepare it with new files for each deployment, which can be fetched directly from Cirrus CI’s API within the temporary app, instead of including these files in the Docker image itself which seems to be an anti-pattern (since the entire layer would need to be reuploaded each time).

7

That would be even better yes!

Although you can’t really have 2 VMs using the same volume. Perhaps you can have your VM pull the files on boot? Deploy the tiny app every time you build with Cirrus CI’s.

1 Like