This problem started yesterday. Deploys gets stuck and never ends.
We used the syd region where I see that some problems were reported.
I already removed the region temporarily and did the deploy again but the error continues.
What could I try to deploy?
Does your release command end? I assume it does, but maybe it’s stuck doing something forever like a database transaction that wasn’t cleanly halted (and thus is holding a lock).
You can check
fly status in a different terminal if you want to see what’s happening with your instances. You can also
fly logs and you should see new instances’ logs coming up.
The status command show me:
2022-03-31T12:40:43.802 runner[98bdcb21] iad [info] Configuring firecracker
2022-03-31T12:40:43.803 runner[98bdcb21] iad [info] Starting virtual machine
2022-03-31T12:40:44.007 app[98bdcb21] iad [info] Starting init (commit: 6f9865f)...
2022-03-31T12:40:44.033 app[98bdcb21] iad [info] 2022/03/31 12:40:44 listening on [fdaa:0:37d5:a7b:775a:98bd:cb21:2]:22 (DNS: [fdaa::3]:53)
2022-03-31T12:40:46.033 app[98bdcb21] iad [info] Main child exited normally with code: 0
-- Last deploy logs --
2022-03-31T12:54:04.145 runner[fa165133] iad [info] Starting instance
2022-03-31T12:54:04.565 runner[fa165133] iad [info] Configuring virtual machine
2022-03-31T12:54:04.566 runner[fa165133] iad [info] Pulling container image
2022-03-31T12:54:07.635 runner[fa165133] iad [info] Configuring firecracker
I’m going to try the latest version of the app that worked.
This usually happens because there’s a conflicting scale/regions config.
Did you set
fly scale count 3 --max-per-region 1 by chance? If you only have two regions, and want three VMs, it can never fulfill that. Assuming that’s the case, you can either add a region, or run
fly scale count 2 --max-per-region=1.
This is exactly what happened!
I forget to rescale when you remove the Syd region.
Thanks for your help!
Is there a way for fly to detect this case automatically on deploy and give a good warning?
We do detect it on deploy and scale. I think you can only get into this state if you’re already running successfully and remove a region.