Deploying to Fly via GitHub Action failing

I’ve had this happen twice in a row. I get this error during my deploy step of my GitHub action:

Searching for image 'registry.fly.io/kcd:main-3545aab8e426caa832abb6b7439a843a20e75ee0' remotely...
Error failed to fetch an image or build from source: Could not find image "registry.fly.io/kcd:main-3545aab8e426caa832abb6b7439a843a20e75ee0"

I’m not sure what caused this error. Normally everything goes off without a hitch. Is there something wrong with the fly registry?

6 Likes

I’m having the exact same issue, with two different apps, one of them:

Searching for image 'registry.fly.io/reference-structuring--staging:main-57ae53e185c798bc7cd98db6b9fed8f926ebbc78' remotely...
Error failed to fetch an image or build from source: Could not find image "registry.fly.io/reference-structuring--staging:main-57ae53e185c798bc7cd98db6b9fed8f926ebbc78"
1 Like

We are also having this issue. It is mentioned in at least 2-3 other posts in this forum.

I have nearly an identical deployment strategy as Kent and I investigated this error for quite awhile and while I don’t have anything concrete, I think the issue involves the Github action building the docker image for the “linux/amd64” platform (correctly) but when the fly cli is called, it’s not specifically looking for that platform and so it reports that the image cannot be found.

This is likely a Fly error.

Our deployment pipeline has worked perfectly for over half a year. No changes have been made to the pipeline.

@kentcdodds (BTW, Hi :wave:) You could possibly verify what I mentioned above by logging into the FLY registry (using a token) and then running docker pull --platform=linux/amd64 registry.fly.io/.... If an image is successfully pulled, then we know that the image did in fact make it into the registry and that the issue is somwhere in flyctl not knowing how to pull it.

FWIW, I was able to successfully deploy by running fly deploy --local-only from my local computer.

I’m not sure whether I’m doing what you’ve asked correctly, but I have wireguard enabled on my machine and then I run:

$ docker pull --platform=linux/amd64 registry.fly.io/kcd:main-3545aab8e426caa832abb6b7439a843a20e75ee0
Error response from daemon: Head "https://registry.fly.io/v2/kcd/manifests/main-3545aab8e426caa832abb6b7439a843a20e75ee0": no basic auth credentials

Something definitely wrong. Having this issue on two separate apps. Any idea when the registry will be fixed?

We’re investigating this. We don’t know where this error is coming from, it appears to be an error generated by the local Docker Daemon in the Github action. Which doesn’t make a lot of sense.

2 Likes

kind of makes sense cause the build-push action works, so the registry appears to be working…just doesn’t find it when it goes to look it up for deploying.

There’s an outside chance this incident is related: GitHub Status - Incident with Actions, Git Operations and Pages

Will you all see if your actions are working again? We still haven’t been able to replicate or pin down the source of the error.

I kicked off my deploy action again and got the same result: fix login issues when out of primary region · kentcdodds/kentcdodds.com@f49314a · GitHub

I was able to get a needed deploy out using --local-only as @nick-brevity suggested so that’s good :slight_smile: Back to the slopes :snowboarder: God speed!

1 Like

Just tried again, and still getting the same issue. Not able to find the image from the fly registry.

For me it really appears like the image is not published after being uploaded.

It appears like the image is correctly being pushed.

Screenshot 2023-01-19 at 23.39.30

But it can not be resolver thereafter.

I did my best to compare names, tags, and IDs, and everything looked to be correctly written out.

We’re still working on this. One thing we’re investigating is whether the “current” Ubuntu runner generates images differently. We’ve managed to replicate some of the errors with different buildkit/oci flags, so it’s possible a default changed.

1 Like

We are using the GitHub runners to build and push the image. Ie. the image should to fully addressable from it’s URL regardless of fly runners.

We may have found the issue and a workaround. It looks like sometime earlier today GitHub updated the ubuntu runner which bumped the version of BuildKit from 0.9.1 to 0.10.0. Images built with this version of BuildKit are returning a 404 when our backend attempts to fetch the image manifest from the target registry before a deployment. This is happening for both registry.fly.io and Docker Hub.

Now, I don’t see anything in the BuildKit 0.10.0 release notes that looks related, but something changed and we need to investigate further to figure out and fix our registry client. Until then, reverting to BuildKit 0.9.1 solves the problem.

Workaround

You can either disable BuildKit or specify a working version to get back on track.

Either comment this out:

    steps:
      -
        name: Set up QEMU
        uses: docker/setup-qemu-action@v2
      -
        name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2

or change docker/setup-buildx-action@v2 to pass a version like this:

      -
        name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2
        with:
          version: v0.9.1

We’ll keep ya posted.

7 Likes

I can confirm this workaround worked for me.

Thanks for the investigation! Will stay tuned for when we can remove the workaround :+1:

EDIT: Unfortunately I don’t think it did work. So I was able to start the deploy, but the logs of the actual deploy gave me:

2023-01-20T01:00:49Z runner[bb5a374b] den [info]Pull failed, retrying (attempt #1)
2023-01-20T01:00:50Z runner[bb5a374b] den [info]Pull failed, retrying (attempt #2)
2023-01-20T01:00:50Z runner[bb5a374b] den [info]Pulling image failed

This resulted in a rollback.

Oh now that’s interesting. I was testing that our registry client could actually see the images. And I was able to confirm that dockerd could pull them down. We run containerd on workers which might be having a problem. Thanks for letting us know!

1 Like