flyctl machine run hitting MANIFEST_UNKNOWN for a tag that exists (possible control-plane cache issue)

Questions / Help Build debugging
1

I can push update to an existing container tag and inspect an image in Fly’s private registry, but flyctl machine run fails with a 404 MANIFEST_UNKNOWN for a digest that appears to be stale.

docker manifest inspect registry.fly.io/<app-name>:<tag> returns a valid schema v2 manifest.

What fails

flyctl machine run --image registry.fly.io/<app-name>:<tag>  --debug --org myorg --region sin -a myapp

# Error: could not launch machine: failed to get manifest registry.fly.io/<app-name>:<tag>@sha256:<old-digest>:
# request failed: not found [http 404]: {"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown"}]}
# (Request ID: <redacted>)

It looks like flyctl is resolving the tag to an old digest. The registry currently points the tag to a different digest.

What I’ve tried

  • Re-login to the registry.

  • Re-push the same tag.

  • Change the region.

  • Recreated the Fly app (<app-name) and repushed.

  • Specified platform amd64 during docker buildx

  • Changed docker buildx compression to zstd

Environment

  • Host: Linux (rootful Docker).

  • Image platform: linux/amd64.

  • flyctl v0.3.191 linux/amd64 Commit: 54468f932f1a5750e33be235b37d1dfaa66cf5e2 BuildDate: 2025-10-02T16:37:49Z

Expectation
flyctl machine run --image ``registry.fly.io/``<app-name>:<tag> --debug --org myorg --region sin -a myapp should resolve the tag to the current digest and launch successfully.

Is there a known issue with tag to digest resolution caching on the control plane? Is there any way on my side to force a refresh/purge of the cached resolution for this tag?

2

Could be related to this issue: New code not reflected in deploys

3

Did u update flyctl?

4

Yes, pushing to a new tag is okay. this is when reusing a tag.

5

reusing a tag is not supported, we do indeed cache on it.

6

Not sure. I was using this tag for few years with no issue and reuse it all the time with new content. this should be a recent issue.

I just created a new tag, pushed new image, reused it a few time and it works (so far!).

7

Ah! I think I might be bumping into this right now. Special tags for me are small and large, and I’m getting unpredictable behaviour as to how they resolve; the resulting machine seems to be a day or two out of date. I wonder if I should switch to an always-new tagging system.

1 Like