Detect when app moves to different region

Hoping the community chimes in on this one :pray:

Our app must always run in 10 different regions, including in LHR and IAD.
After a deploy, the app is correctly running in the 10 specified regions but within a few days, the app in e.g. IAD or LHR (or some other region, but often those two) is ‘moved’ to a different region.

  1. How ensure the app does not move to different region (or less likely to be moved)?
  2. How can I detect a region changed?

All tips/ideas are welcome

1 Like

For Apps v1, I guess you can accomplish it with the --max-per-region directive. It only works with rolling deploys.

Disable autoscale at any rate. Also, with multi-process Fly apps, things are more tricky, so don’t use those either, if you can avoid it.

Apps v2 platform will (?) have an “events” API (and apparently all Fly apps will be migrated over to v2 seamlessly some time in the future). Until then, you can periodically TXT query (vms.<app-name>.internal) the dns resolver running at [fdaa::3]:53 from within any Fly VM, and do a pseudo service discovery to find just where existing VMs are running and alert in case something’s happened that you didn’t expect.

Another alternative might be to read the FLY_REGION env variable on process startup, publish custom metrics and alert (unfortunately, Fly-native alerts are defunct; you may also consider using any other form of notification mechanism, like Queues or webhooks or emails) on them, for example say there’s an unexpected number of VMs in a single region.

Thanks!
I’ve used this in the past and disabled that. Will check notes on why and maybe try again.

Your idea for querying the TXT record in DNS is great :clap: :clap:

Hmm, can’t get the max-per-region working

What I did, in order:

  • fly scale count 10 --max-per-region=1 to enable the max per region
  • fly scale show still shows Max Per Region: Not set :thinking:

Read some community post and seems you need to scale down and then up to get the max per region appear to be set in the CLI … not very clear. So:

  • fly scale count 9 --max-per-region=1
  • fly regions remove syd and this triggers Fly to do a (rolling) deploy to have the VM no longer in Sydney
  • fly scale count 10 --max-per-region=1
  • fly regions add syd and Fly deploys again and Sydney is back
  • fly scale show still shows Max Per Region: Not set :disappointed:

I’m stuck and have no idea now how to get Max Per Region working and know it’s working, with value set to 1

I have Autoscaling: Disabled

On more FYI: when I removed syd from regions pool, the CLI showed a backup region:

Region Pool: 
cdg
gru
hkg
iad
lhr
maa
mia
nrt
sjc
Backup Region: 
iad

The backup region also showed later when adding syd to the pool.
However, fly regions list does not list any backup region, just Backup Region:

1 Like