Yesterday, a backup region maa was deployed. However, this morning upon pushing a new build, I had issues with this maa failing and my deployment constantly failing. I went and adjusted the backup region pool to remove maa with:
flyctl regions backup fra hkg iad lax lhr sea vin
However, this completely wiped out all the backup regions. Upon querying flyctl regions list I am getting:
Region Pool:
ams
ewr
sin
sjc
Backup Region:
That is, no backup regions at all. How do I fix this?
Ooof that sucks. Let me have a look at your app and see what the issue is.
Do you need that many backup regions? Backup regions are really meant for apps that have a single primary region. We’re actually considering dropping backup regions altogether because the UX is disgusting and, in practice, the types of outages we deal with aren’t usually solved with backup regions.
Do you need that many backup regions? Backup regions are really meant for apps that have a single primary region. We’re actually considering dropping backup regions altogether because the UX is disgusting and, in practice, the types of outages we deal with aren’t usually solved with backup regions.
Nope. I only need a few. The backup region list automatically grew as time went on for some reason… I ideally only need 4 backup regions.
I do support the idea of dropping backup regions altogether. When I push a new build, sometimes a region will fail to deploy the app. Then the next time I deploy the app, it’ll use a backup region. I then have to deploy the build once again to get all VMs to be using my primary regions… less than ideal.
This appears to be the same issue I reported on previously: Can’t define backup regions - Questions / Help - Fly.io. That’s still the case for all of my apps that I have, I am unable to add backup regions (or at least they don’t show in the UI).
@bekit This is exactly the same issue as you wrote about on Jul 22. Have you had any VM deploy out of dfw/atl to confirm that it’s only flyctl’s list command that’s broken?
I think it might be more than than the displayed UI because the “Different region on each deploy” issue doesn’t happen even if I set backup regions. This is fine for what I need for now since I’m trying to avoid that other bug, but will become an issue if my main environment is ever unavailable.
@kurt , I’ve gone and added in 4 backup regions with flyctl regions list showing no backup regions. What happens if one of my four primary regions go down? Will my app break as there’s no backup regions now specified, or will my other 3 primary regions still continue to take traffic?
Not sure if this is what you are looking for, but if I really just want to issue a restart and don’t care about downtime, I scale to zero and back up to one.
Yeah, I now see that I can take out a region, and add it back in – that will redo a deployment. I guess it might be easier just toggling between adding in and deleting a bogus secret to get things to redeploy.
Btw, @bekit, I had a VM deploy in one of my chosen backup regions (i.e. flyctl regions backup sea)
