My service is using blue-green deployment strategy and recently I’ve got next deployment failure:
Marking green machines as ready
Machine e823970ae77658 [app] now ready
Machine 78102d5a417538 [app] now ready
Checkpointing deployment, this may take a few seconds…
WARN failed to release lease: request returned non-2xx status: 408:
Error: failed to set metadata for 683497ea679d98: request returned non-2xx status: 408:
failed to mark as safe for deletion (Request ID: 01KMHN3Y2SE04SNKWZDF1RPW1H-ord) (Trace ID: 33b15980d7059e23f9dda16a75e5794c)
On next deployment I’ve got next error:
Verifying if app can be safely deployed
Found 2 different images in your app (for bluegreen to work, all machines need to run a single image)
[x] my-novonotes:deployment-01KMH6SC839092ST3FEHKBFZF5 - 2 machine(s) (185e26db3de128,683497ea679d98)
[x] my-novonotes:deployment-01KMHMQ0F97Y6SJS8R8TCFPHAZ - 2 machine(s) (78102d5a417538,e823970ae77658)
Here’s how to fix your app so deployments can go through:
- Find all the unwanted image versions from the list above.
Use ‘fly machines list’ and ‘fly releases --image’ to help determine unwanted images.- For each unwanted image version, run ‘fly machines destroy --force --image= --app ’
3. Retry the deployment with ‘fly deploy’
Error: found multiple image versions
Apparently the first deployment finally got up and running that’s why the second deployment failed. However, this is far from normal to have two different versions of backend running (at least while it’s not because of my actions:)).