Different Region on Each Deploy

1

I’m experiencing two issues, likely related:

  1. Each time I deploy, my app is placed in a different (seemingly random) region that is not in my region list.
  2. My region list is only ewr, however my single app instance is in a different region. Running flyctl -app [my app] regions set ewr returns a correct region pool, but does not move my app. The only thing that does move my app is scaling my vm up and down. Then it settles in ewr.

FYI, this only happens on one fly app, the rest are working as expected.

Any ideas?

2

@conor That might happen if the desired region is either full or having an outage. I don’t see any issues in ewr right now so something else might be happening. Could you post or DM your app’s name and we can take a look?

3

@conor just for some context here, we let apps run on backup regions when something goes wrong. We can tweak this for you if you’d rather have a different behavior, by default it picks the two closest regions as backups. Some people run with no backup regions because their workloads don’t make any sense if their preferred regions are unavailable.

In this case, it seems like our scheduling algorithm preferred a backup region over the ewr region you specified. This is something we’re working to improve, it happens when it shouldn’t in some scenarios. We should have better scheduling controls for you in the next couple of months.

4

Something I experienced a while ago was that my app was being deployed to 4 regions when I had defined only 3 regions.

I haven’t deployed any multi-region apps since then though. If I see this again I will write back.

5

@pier that’s a quirk of our “backup region” settings, probably. If you have 3 regions set, we’ll also allow the app to deploy to 3 other nearby regions. So if it does get more instances, it could put them in “backup regions”.

It’s pretty obvious our backup region strategy is surprising in all the wrong ways, so we’re working on some alternatives.

6

It doesn’t sound like it’s a bad strategy tbh, it’s just that you’re not communicating it properly in the CLI.

7

Yeah, this is a feature not a bug, it’s just not documented well enough. Just a confirmation after setting the regions that x,y,z have also been set as backups should be enough.

1 Like
8

I think what we’re going to do is expose backup region configuration, use a reasonable default when an app is first created, and then echo the whole config when you make any kind of change.

It’s nice to hear you all like that feature. :smiley:

1 Like
9

I just submitted a PR to make backup region behavior more obvious: https://github.com/superfly/flyctl/pull/254

Listing regions now shows the backup regions that are in effect for an app. Here’s an app with the default backup region set:

➜ flyctl regions list -a debug
Region Pool:
fra
gru
lax
syd
Backup Region:
ams
mrs
scl
sea
sin
sjc

These can be overridden:

➜  flyctl regions backup dfw ams sin -a debug
Region Pool:
fra
gru
lax
syd
Backup Region:
ams
dfw
sin

It’s not possible disable backup regions (yet) but we’re close!

10

This looks good… I don’t actually see a reason why backups regions should be disabled at all — the only compliance reason I could think of was data residency, but it seems like in all other cases one would prefer a lower latency to not having a server running at all.