"vXXX is being deployed" freeze

Questions / Help Build debugging
1

It’s been a while since I last tried to deploy and I have realized that it now freezes right at the end on “vXXX is being deployed”.

The monitoring tab shows that the new version is being deployed on my 3 instances with

Deployment is running pending automatic promotion.

3 desired, 0 placed, 0 healthy, 0 unhealthy

It did this 3 days ago, I let it be for hours, and the app crashed at some point (probably because of this). After restarting the app the new version was live.

It is doing it again now. I hope it won’t crash in a few hours!

EDIT: I just tried restarting the app and the new version did get deployed. I would like to avoid having to restart the app every time though

1 Like
2

2 days later, this is still an issue :confused:

3

There is likely something blocking this deploy. If it’s hung, try running this:

fly status --all

See if there are any failed instances, or instances that don’t have health check information.

Also, if the app has volumes, it’s possible it got hung trying to do a canary deploy. Canary deploys will wait like this until a previous VM stops when you have volumes. This generally shouldn’t happen because we try to default to rolling deploys, but there are edge cases where it crops up.

You can potentially unstick your deploy by running fly vm stop <id>. Use any old instance ID and see if it helps.

4

fly vm stop did nothing, desired status is now stop tho

5

We’re running into this issue as well — fly vm stop puts the VM into a stopping state, but it hasn’t stopped.

6

fly status --all shows old instances but no new ones being spun up.

I do run a blue green deployment strategy, if that helps figuring this out.
I don’t use volumes though.

7

Did you set a --max-per-region when you scaled by chance? Blue green and canary won’t work with that setting (because Nomad helpfully applies the restriction to new deploys).

fly deploy --strategy rolling is your best bet, just to see if you can get unstuck.