Detect when app moves to different region

1

Hoping the community chimes in on this one :pray:

Our app must always run in 10 different regions, including in LHR and IAD.
After a deploy, the app is correctly running in the 10 specified regions but within a few days, the app in e.g. IAD or LHR (or some other region, but often those two) is ā€˜movedā€™ to a different region.

  1. How ensure the app does not move to different region (or less likely to be moved)?
  2. How can I detect a region changed?

All tips/ideas are welcome

1 Like
2

For Apps v1, I guess you can accomplish it with the --max-per-region directive. It only works with rolling deploys.

Disable autoscale at any rate. Also, with multi-process Fly apps, things are more tricky, so donā€™t use those either, if you can avoid it.

Apps v2 platform will (?) have an ā€œeventsā€ API (and apparently all Fly apps will be migrated over to v2 seamlessly some time in the future). Until then, you can periodically TXT query (vms.<app-name>.internal) the dns resolver running at [fdaa::3]:53 from within any Fly VM, and do a pseudo service discovery to find just where existing VMs are running and alert in case somethingā€™s happened that you didnā€™t expect.

Another alternative might be to read the FLY_REGION env variable on process startup, publish custom metrics and alert (unfortunately, Fly-native alerts are defunct; you may also consider using any other form of notification mechanism, like Queues or webhooks or emails) on them, for example say thereā€™s an unexpected number of VMs in a single region.

3

Thanks!
Iā€™ve used this in the past and disabled that. Will check notes on why and maybe try again.

Your idea for querying the TXT record in DNS is great :clap: :clap:

4

Hmm, canā€™t get the max-per-region working

What I did, in order:

  • fly scale count 10 --max-per-region=1 to enable the max per region
  • fly scale show still shows Max Per Region: Not set :thinking:

Read some community post and seems you need to scale down and then up to get the max per region appear to be set in the CLI ā€¦ not very clear. So:

  • fly scale count 9 --max-per-region=1
  • fly regions remove syd and this triggers Fly to do a (rolling) deploy to have the VM no longer in Sydney
  • fly scale count 10 --max-per-region=1
  • fly regions add syd and Fly deploys again and Sydney is back
  • fly scale show still shows Max Per Region: Not set :disappointed:

Iā€™m stuck and have no idea now how to get Max Per Region working and know itā€™s working, with value set to 1

I have Autoscaling: Disabled

On more FYI: when I removed syd from regions pool, the CLI showed a backup region:

Region Pool: 
cdg
gru
hkg
iad
lhr
maa
mia
nrt
sjc
Backup Region: 
iad

The backup region also showed later when adding syd to the pool.
However, fly regions list does not list any backup region, just Backup Region:

1 Like