Hey folks, we are just setting up our first app. We are experiencing intermittent failures on deploys where the release_command fails without being able to create a VM. I think it is a race condition between deploy and running the release_command.
Failed: error running release_command machine: error creating a release_command machine: failed to launch VM: failed to get manifest registry.fly.io/xxx: request failed: not found [http 404]: {“errors”:[{“code”:“MANIFEST_UNKNOWN”,“message”:“manifest unknown”,“detail”:“unknown tag=deployment-xxx”}]}
Error: release command failed - aborting deployment. error running release_command machine: error creating a release_command machine: failed to launch VM: failed to get manifest registry.fly.io/xxx: request failed: not found [http 404]: {“errors”:[{“code”:“MANIFEST_UNKNOWN”,“message”:“manifest unknown”,“detail”:“unknown tag=deployment-xxx”}]}
```
@jtsmills Are you doing a two stage deploy? IE building + pushing your image in one fly deploy command, and then running the actual deploy in another right after?
It does look like it’s hitting a race, where the image hasn’t been pushed to our upstream registry by the time you start the deploy, causing the missing manifest error. We’re making some changes to better support that type of workflow.
There’s a couple options to work around it in the meantime.
A retry of the deploy step should succeed after ~1m for normal sized images. Similar with waiting between the initial push and running the deploy.
Switching your CI workflow to do a single step fly deploy instead of doing the separate build + push, then deploy steps. This avoids the error I mentioned as it can pull the image from the host’s local registry instead of needing to pull from the upstream
You can use --depot=false flag on the build step to use fly-hosted builder. This will also avoid the upstream registry issue. This can be used with the two step deployment flow.
I’m getting the same error. I tried to deploy an app that has successfully been deployed in the past (gave it a new name and let Fly generate it’s own toml file automatically again) but even after 24hrs and numerous retries it’s still failing with: (app name redacted)
This deployment will:
* create 2 "app" machines
> Launching new machine
No machines in group app, launching a new machine
✖ Failed: error creating a new machine: failed to launch VM: failed to get manifest registry.fly.io/zzzzz-proxy:deployment-8697292e370f8dd3c19fd7e918078db8: request failed: not found [http 404]: {"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown","detail":"unknown tag=deployment-8697292e370f8dd3c19fd7e918078db8"}]}
Error: error creating a new machine: failed to launch VM: failed to get manifest registry.fly.io/zzzzz-proxy:deployment-8697292e370f8dd3c19fd7e918078db8: request failed: not found [http 404]: {"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown","detail":"unknown tag=deployment-8697292e370f8dd3c19fd7e918078db8"}]}
(Request ID: 01K76R5MNVS7HHTYQ6CDNXF7XR-iad) (Trace ID: f68457cd2e0cd7d59867ea7590fb4692)
unsuccessful command 'flyctl deploy -a zzzzz-proxy --image registry.fly.io/zzzzz-proxy:deployment-8697292e370f8dd3c19fd7e918078db8 --depot-scope=app '
Hi @Jonnie, sorry, we accidentally ended up rolling back this fix while applying some newer updates, causing this bug to resurface. We’ve re-applied this fix to the UI deployer (more permanently this time) so that should be working properly again, sorry for the inconvenience.
I’m just using flyctl (latest version) and I’m still experiencing this race condition. Glad the workaround is available, but this still isn’t fully fixed.
Failed: error running release_command machine: error creating a release_command machine: failed to launch VM: failed to get manifest registry.fly.io/xxx: request failed: not found [http 404]: {“errors”:[{“code”:“MANIFEST_UNKNOWN”,“message”:“manifest unknown”,“detail”:“unknown tag=deployment-xxx”}]}
