Hi, when I first started clustering via the guides (cookie, clustering your application) everything worked as intended.
However, after upgrading the server cpu/memory and db cpu/memory, clustering no longer works. It went from 2 instances in ewr and sea to 2 instances in sea. I’ve been able to scale down to 0 to remove them now but can no longer scale up any instances at all. flyctl scale count 2fly scale count 2fly scale any command doesn’t seem to have any effect.
In the past when our app was bit by a similar issue (a year or so ago) which was also using the tricky-to-tame autoscale setup, only creating a new app with both those traits (autoscale and multi-process) removed fixed it for us, then.
Thanks for the thoughts! It’s an Elixir/Phoenix app so I’m not running wih the multi process app settings. So far no luck with any of these attempts. Currently now there are no instances running and it can’t get anything deployed either. Tried the the rolling start suggestion and the fly vm stop errors since there are now no instances running at all.
In my dashboard it’s stuck on the pending button and the monitoring dash still stuck on this:
I don’t think Fly would in anyway gate such a thing behind plans, given how much Kurt dislikes companies that do so, like Cloudflare (?).
Btw, this means you’re entitled to email support with guaranteed SLAs (?); and do let us know what the issue really is, if they figure it out.
You may have probably tried this, but fly status --all -a <app-name> & fly ips private & fly info -a <app-name> might show some helpful additional information.
I believe something in the release_command might be getting in the way. Is there a way to remove or avoid running it for a test deploy?
Thanks! I was able to track down the support email (I hadn’t noticed it in the billing settings before). Will definitely let you know when we figure out what’s going on, it was definitely amazing when it was working at first.
Okay, so it’s working now. I think what happened is that I had tried switching things before deploys finished and so it was trying to finish deploying to sea when I had taken the region down and just had ewr.
As soon as I added sea back, the deploy finished. And when I ran fly scale count 2 again everything worked as it should! Really appreciate the time and thank you.
From your initial post, it wasn’t clear if the problem was that 2 instances were in 1 region instead of being equally distributed across 2 regions? I don’t think that was it though because even if they were in the wrong region, they’d still be clustered.
If it was the case that they were in the same region and you wanted them spread out, then you should check, out this setting…
fly scale count 2 --max-per-region 1
Otherwise it’s still valid to put two instances in the same region.
Hey @Mark! Yea, I didn’t express it clearly but it had began with having it not equally clustered. Then I think I fiddled around so much with commands before deploys finished (like taking sea down while it was trying to deploy to sea). Once, I set it so it would finish deploying to sea again, I was able to follow those commands to have it spread out equally across ewr and sea. So… just more learning in public over here. Fly’s support has been incredible from Kurt to @ignoramous and you. It’s seriously awesome, and the guides made it so easy. It has me very excited and feeling like I found another “super power” for being a solo team. Thank you.
Will definitely let you know when we figure out what’s going on, it was definitely amazing when it was working at first.
