fly deploy never ends

Hi! :smiley:

This problem started yesterday. Deploys gets stuck and never ends.

We used the syd region where I see that some problems were reported.
I already removed the region temporarily and did the deploy again but the error continues.

What could I try to deploy?

Thanks!

Hello!

Does your release command end? I assume it does, but maybe it’s stuck doing something forever like a database transaction that wasn’t cleanly halted (and thus is holding a lock).

You can check fly status in a different terminal if you want to see what’s happening with your instances. You can also fly logs and you should see new instances’ logs coming up.

Hi @jerome!

The status command show me:

Logs:

2022-03-31T12:40:43.802 runner[98bdcb21] iad [info] Configuring firecracker

2022-03-31T12:40:43.803 runner[98bdcb21] iad [info] Starting virtual machine

2022-03-31T12:40:44.007 app[98bdcb21] iad [info] Starting init (commit: 6f9865f)...

2022-03-31T12:40:44.033 app[98bdcb21] iad [info] 2022/03/31 12:40:44 listening on [fdaa:0:37d5:a7b:775a:98bd:cb21:2]:22 (DNS: [fdaa::3]:53)

2022-03-31T12:40:46.033 app[98bdcb21] iad [info] Main child exited normally with code: 0

-- Last deploy logs --

2022-03-31T12:54:04.145 runner[fa165133] iad [info] Starting instance

2022-03-31T12:54:04.565 runner[fa165133] iad [info] Configuring virtual machine

2022-03-31T12:54:04.566 runner[fa165133] iad [info] Pulling container image

2022-03-31T12:54:07.635 runner[fa165133] iad [info] Configuring firecracker

I’m going to try the latest version of the app that worked.

Got the same result

This usually happens because there’s a conflicting scale/regions config.

Did you set fly scale count 3 --max-per-region 1 by chance? If you only have two regions, and want three VMs, it can never fulfill that. Assuming that’s the case, you can either add a region, or run fly scale count 2 --max-per-region=1.

2 Likes

Hi @kurt!

This is exactly what happened!
I forget to rescale when you remove the Syd region.

image

Thank you!

Hi, Kurt.
Thanks for your help!
Is there a way for fly to detect this case automatically on deploy and give a good warning?

We do detect it on deploy and scale. I think you can only get into this state if you’re already running successfully and remove a region.

1 Like