can't deploy to multiple regions anymore

Questions / Help
1

I had a web app that was running in two regions. I ended up deleting it and recreate with pretty much the same configuration:

❯ fly autoscale standard -a sumiu-web
     Scale Mode: Standard
      Min Count: 1
      Max Count: 10
...
❯ fly regions list -a sumiu-web
Region Pool:
fra
gru
Backup Region:

In the past, I would run fly scale count 2 -a sumiu-web and it would spin up a new instance in the second region. Now, when I do it, I get this message:

❯ fly scale count 2 -a sumiu-web
Warning: Autoscaling is now disabled for sumiu-web

Count changed to 2

and a new app spins up on the same primary region

image
image1514×518 55.4 KB

Did something change?

I’m not entirely sure if this is related but I was playing around with the processes key and I ended up changing the Scale tab to this now:

image
image1656×638 71.6 KB

I changed it back to run the app with CMD ${SERVER_COMMAND} so there is no web, worker or agent anymore running.

2

Hi @luizkowalski, have you tried fly scale count 2 --max-per-region=1?

3

it does nothing
it says Count changed to 2 but I only see one instance in fra. nothing on gru

edit: weird… I set --max-per-region=2 and scaled it to 4 and I only see two on fra, nothing on gru yet

4

Long shot: does your app use volumes?

5

no, it doesn’t

6

Another one: does fly.toml have a [processes] section in it still? If so, can you try removing it and redeploy?

7

it has but it is commented out, I mentioned before that I’m using the CMD on dockerfile then passing $SERVER_COMMAND to it

8

hey @catflydotio, do you have more ideas? is this related to this (Autoscaling not distributing properly - #3 by jerome) maybe?

things are getting strange by the minute, now when I scale to 2, or even 3, absolutely nothing happens.

9

We did have a recent bugfix for autoscaling, as in that thread. The problem you’re seeing still looks like it could be issue with switching to using processes and back, though.

Do you mind trying this and letting us know the result?

fly scale count app=4 --max-per-region=2

1 Like
10

I got this message

❯ fly scale count app=4 --max-per-region=2 -a sumiu-web
Warning: Autoscaling is now disabled for sumiu-web

Count changed to 4

And there are only two VMs

image
image2994×618 108 KB

11

OK, so, it looks like it was an edge case to do with switching to multiple processes and back, and the app process only had one region: fra. We’ve gone in and cleaned that up on your app, so you should see the four instances, two in each region, now.

12

not sure if you did something already but now fly scale count app=2 -a sumiu-web spins up one more instance in gru, as expected.

edit: ah, there you go! hahahaha thanks a lot, @catflydotio

1 Like
13

:tada: I can’t take credit for the fixing. Had to escalate that one pretty far.

The processes feature is in preview and has a lot of edge cases. Your app had configuration left over from trying that. I think you might have been able to sort it with something like fly regions set fra gru --group app, but there might have been more leftovers to find.

2 Likes
14

I’m having the same issue after scaling. Now fly scale count 2 just keeps scaling in the same region (e.g. sea) when I have two regions (sea and ewr).

Hasn’t even been able to scale. It’s stuck on one instance in sea.

15

I think we’re hitting a similar issue @catflydotio - we were using multi processes, but then stopped and just used multiple apps instead. I’ve tried all the combinations of setting regions for a process, scaling up and down and using max-per-region → is there a canonical way to fix that that doesn’t require deleting the app and starting again?

Thanks!

edit - I’ve deleted all mentions of processes from our fly.toml, redeployed and no matter what I can’t set the max per region. fly-scale-show always returns:

VM Resources for app name
        VM Size: shared-cpu-1x
      VM Memory: 2 GB
          Count: 6
 Max Per Region: app=0 web=0 worker=0
16

flyctl scale show is broken for multi-process apps: Incorrect flyctl scale show output for multi-process apps · Issue #1294 · superfly/flyctl · GitHub

Check if LOG_LEVEL=debug fly scale show -a <app-name> shows the correct info in the large json that it prints?

nb, multi-process apps do have some rough edges (but as I understand it, multi-process apps and Machine apps is where Fly’s focus is at).

17

Thanks! I’ve run that now:

        {
          "name": "app",
          "count": 4
        }
      ],
      "processGroups": [
        {
          "name": "abc",
          "maxPerRegion": 0
        },
        {
          "name": "app",
          "maxPerRegion": 0
        },
        {
          "name": "defeag",
          "maxPerRegion": 0
        },
        {
          "name": "web",
          "maxPerRegion": 0
        },
        {
          "name": "worker",
          "maxPerRegion": 0
        }
      ]

So, I can’t set maxPerRegion anymore - since I can’t set it per process group. And there doesn’t seem to be any way for me to remove process groups.

18

@captainb out of curiosity have you tried doing something like below? I was experimenting with this on an app of mine that’s in a similar state and was able to get a max set by doing:

fly scale app=2 --max-per-region 4

The max is applied to only the process groups you specify in the command. That also won’t change the scale count of process groups you don’t specify.

It might help to know that apps without any process groups specified will use the app process group.

19

@jphenow Sorry for the very slow reply - nope, this doesn’t work either I’m afraid :frowning:

There’s no way for me to remove process groups entirely? I don’t really need them…