I’ve had this happen twice in a row. I get this error during my deploy step of my GitHub action:
Searching for image 'registry.fly.io/kcd:main-3545aab8e426caa832abb6b7439a843a20e75ee0' remotely...
Error failed to fetch an image or build from source: Could not find image "registry.fly.io/kcd:main-3545aab8e426caa832abb6b7439a843a20e75ee0"
I’m not sure what caused this error. Normally everything goes off without a hitch. Is there something wrong with the fly registry?
I’m having the exact same issue, with two different apps, one of them:
Searching for image 'registry.fly.io/reference-structuring--staging:main-57ae53e185c798bc7cd98db6b9fed8f926ebbc78' remotely...
Error failed to fetch an image or build from source: Could not find image "registry.fly.io/reference-structuring--staging:main-57ae53e185c798bc7cd98db6b9fed8f926ebbc78"
I have nearly an identical deployment strategy as Kent and I investigated this error for quite awhile and while I don’t have anything concrete, I think the issue involves the Github action building the docker image for the “linux/amd64” platform (correctly) but when the fly cli is called, it’s not specifically looking for that platform and so it reports that the image cannot be found.
@kentcdodds (BTW, Hi ) You could possibly verify what I mentioned above by logging into the FLY registry (using a token) and then running docker pull --platform=linux/amd64 registry.fly.io/.... If an image is successfully pulled, then we know that the image did in fact make it into the registry and that the issue is somwhere in flyctl not knowing how to pull it.
I’m not sure whether I’m doing what you’ve asked correctly, but I have wireguard enabled on my machine and then I run:
$ docker pull --platform=linux/amd64 registry.fly.io/kcd:main-3545aab8e426caa832abb6b7439a843a20e75ee0
Error response from daemon: Head "https://registry.fly.io/v2/kcd/manifests/main-3545aab8e426caa832abb6b7439a843a20e75ee0": no basic auth credentials
We’re investigating this. We don’t know where this error is coming from, it appears to be an error generated by the local Docker Daemon in the Github action. Which doesn’t make a lot of sense.
kind of makes sense cause the build-push action works, so the registry appears to be working…just doesn’t find it when it goes to look it up for deploying.
We’re still working on this. One thing we’re investigating is whether the “current” Ubuntu runner generates images differently. We’ve managed to replicate some of the errors with different buildkit/oci flags, so it’s possible a default changed.
We may have found the issue and a workaround. It looks like sometime earlier today GitHub updated the ubuntu runner which bumped the version of BuildKit from 0.9.1 to 0.10.0. Images built with this version of BuildKit are returning a 404 when our backend attempts to fetch the image manifest from the target registry before a deployment. This is happening for both registry.fly.io and Docker Hub.
Now, I don’t see anything in the BuildKit 0.10.0 release notes that looks related, but something changed and we need to investigate further to figure out and fix our registry client. Until then, reverting to BuildKit 0.9.1 solves the problem.
Workaround
You can either disable BuildKit or specify a working version to get back on track.
Either comment this out:
steps:
-
name: Set up QEMU
uses: docker/setup-qemu-action@v2
-
name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2
or change docker/setup-buildx-action@v2 to pass a version like this:
-
name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2
with:
version: v0.9.1
Oh now that’s interesting. I was testing that our registry client could actually see the images. And I was able to confirm that dockerd could pull them down. We run containerd on workers which might be having a problem. Thanks for letting us know!