Flyctl use existing docker cache, how to make it use the same cache?

Questions / Help
1

Something I’m seeing is that when I run flyctl with --local-only it successfully uses the local docker to build the image and push, however it isn’t using the existing cache.

For example, if I build my image manually with docker build ., and if I repeat that, the layers are all cached. However if I immediately, without making any changes, run fly deploy --local-only, it doesn’t use any of the cached layers. It rebuilds everything. If I then run the fly deploy command a second time, the layers are cached.

Does flyctl use a different cache somehow, or set any particular parameters that cause its builds to be not the same as regular local builds?

Thanks!

2

Odd. I don’t see the call to local Docker doing anything special on that front, and from what I can tell the cache is managed by the Docker daemon itself.

Will check with the team, but if this is blocking anything in the meanwhile it’ll make more sense to raise an issue with example logs on Issues · superfly/flyctl · GitHub

3

I’ve opened an issue at: fly deploy does not use the local docker daemon cached layers, but does seem to use its own cache · Issue #591 · superfly/flyctl · GitHub

4

Thanks! We tried to reproduce it, but couldn’t. The details and logs you’ve posted on the issue should help.

5

@leighmcculloch the only difference I can think of is that docker builds for the local platform by default while flyctl always builds for linux/amd64. If you’re on an M1 Mac for example you’d need to specify the platform like docker build . --platform linux/amd64 to start with the same base images.

However after looking at the logs from your GH issue I don’t think that’s the issue here since flyctl is printing x86_64 for the local docker host. But I also noticed “Step 2” from the first flyctl output includes ---> Using cache with a SHA that matches the previous docker builds. So the cache is getting used, but the COPY in step 3 produces a different SHA from the docker builds.

==> Building image with Docker
--> docker host: 20.10.10 linux x86_64
Sending build context to Docker daemon  1.912MB
Step 1/20 : FROM ruby:2.6.0 AS ruby
 ---> ef8778f370d5
Step 2/20 : WORKDIR /website
 ---> Using cache
 ---> a545949f6e72
Step 3/20 : COPY website ./
 ---> 6fb801cac070

Is something changing the directory contents between runs? Otherwise it might be a difference in how flyctl and docker generate the build context.

6

I’m building on an Intel Mac, so the architecture should be the same.

I think you’re onto something that it is a build context difference.

Nothing is changing on the filesystem between the runs. The four runs that I included logs for on GitHub all occurred one after the other with no changes on disk. I see this happen on multiple repos, not just this one. I only see the rebuilds occur with the flyctl.

I think this might be it.

For what it is worth, when I enable buildkit builds for both as @jsierles suggested, the builds use the same cache. More details are on GitHub: Deploy without Buildkit does not use locally cached layers · Issue #591 · superfly/flyctl · GitHub.