We may have found the issue and a workaround. It looks like sometime earlier today GitHub updated the ubuntu runner which bumped the version of BuildKit from 0.9.1 to 0.10.0. Images built with this version of BuildKit are returning a 404 when our backend attempts to fetch the image manifest from the target registry before a deployment. This is happening for both registry.fly.io and Docker Hub.
Now, I don’t see anything in the BuildKit 0.10.0 release notes that looks related, but something changed and we need to investigate further to figure out and fix our registry client. Until then, reverting to BuildKit 0.9.1 solves the problem.
Workaround
You can either disable BuildKit or specify a working version to get back on track.
Either comment this out:
steps:
-
name: Set up QEMU
uses: docker/setup-qemu-action@v2
-
name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2
or change docker/setup-buildx-action@v2
to pass a version like this:
-
name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2
with:
version: v0.9.1
We’ll keep ya posted.