Fly clustering and scaling no longer working

Hi, when I first started clustering via the guides (cookie, clustering your application) everything worked as intended.

However, after upgrading the server cpu/memory and db cpu/memory, clustering no longer works. It went from 2 instances in ewr and sea to 2 instances in sea. I’ve been able to scale down to 0 to remove them now but can no longer scale up any instances at all. flyctl scale count 2 fly scale count 2 fly scale any command doesn’t seem to have any effect.

Running flyctl v0.0.396 linux/amd64 Commit: edf8f512 BuildDate: 2022-09-24T19:06:43Z

I also thought maybe this was an issue of being on a hobby plan, so I also upgrade to the launch plan but no change either.

Deploying also hangs and never finishes at ewr [info] Starting clean up.

A shot in the dark:

Is it a multi process app? Or, did you at any point switch from multi-process to a single process? If so, the app may have hit a (new) bug, similar to this one: can't deploy to multiple regions anymore - #9 by catflydotio

In the past when our app was bit by a similar issue (a year or so ago) which was also using the tricky-to-tame autoscale setup, only creating a new app with both those traits (autoscale and multi-process) removed fixed it for us, then.

You can also see if setting --max-per-region has any effect (1, 2) when deployed with rolling strat.

Or, unstick the deployment with fly vm stop and friends: "vXXX is being deployed" freeze - #3 by kurt

Thanks for the thoughts! It’s an Elixir/Phoenix app so I’m not running wih the multi process app settings. So far no luck with any of these attempts. Currently now there are no instances running and it can’t get anything deployed either. Tried the the rolling start suggestion and the fly vm stop errors since there are now no instances running at all.

In my dashboard it’s stuck on the pending button and the monitoring dash still stuck on this:

On the nomad platform.

I don’t think Fly would in anyway gate such a thing behind plans, given how much Kurt dislikes companies that do so, like Cloudflare (?).

Btw, this means you’re entitled to email support with guaranteed SLAs (?); and do let us know what the issue really is, if they figure it out.

You may have probably tried this, but fly status --all -a <app-name> & fly ips private & fly info -a <app-name> might show some helpful additional information.


I believe something in the release_command might be getting in the way. Is there a way to remove or avoid running it for a test deploy?

1 Like

Thanks! I was able to track down the support email (I hadn’t noticed it in the billing settings before). :blush: Will definitely let you know when we figure out what’s going on, it was definitely amazing when it was working at first.

Okay, so it’s working now. I think what happened is that I had tried switching things before deploys finished and so it was trying to finish deploying to sea when I had taken the region down and just had ewr.

As soon as I added sea back, the deploy finished. And when I ran fly scale count 2 again everything worked as it should! Really appreciate the time and thank you. :blush:

1 Like

Hi @mmark!

From your initial post, it wasn’t clear if the problem was that 2 instances were in 1 region instead of being equally distributed across 2 regions? I don’t think that was it though because even if they were in the wrong region, they’d still be clustered.

If it was the case that they were in the same region and you wanted them spread out, then you should check, out this setting…

fly scale count 2 --max-per-region 1

Otherwise it’s still valid to put two instances in the same region. :slight_smile:

1 Like

Hey @Mark! Yea, I didn’t express it clearly but it had began with having it not equally clustered. Then I think I fiddled around so much with commands before deploys finished (like taking sea down while it was trying to deploy to sea). Once, I set it so it would finish deploying to sea again, I was able to follow those commands to have it spread out equally across ewr and sea. :blush: So… just more learning in public over here. Fly’s support has been incredible from Kurt to @ignoramous and you. It’s seriously awesome, and the guides made it so easy. It has me very excited and feeling like I found another “super power” for being a solo team. Thank you.

2 Likes

Awesome! Thanks for sharing your discoveries!

Best of luck and have fun!

1 Like

Thanks. I am mostly shooting in the dark as I don’t know / use more than 75% of Fly features / services.

Btw, I am not employed by Fly (I must get off of this site! :wink:

2 Likes