Standby Machines in secondary region

[danwetherald]

Is it possible to have a replica of all our machines meaning 1:1 clones as “standby machines” in a different region?

We want all our infra to only run on 4-8 machines for our app process group and 2 workers to run in our worker process group in IAD.

We have switched from Fly Managed PG to PlanetScale as it failed us within the first few weeks, as there was upstream IAD networking maintenance which completely took the clusters down, this is not HA as it is sold.

We would like to primarily run in IAD, but in the case that there is another issue like what happen on Sunday night, we want all our app and worker apps to spin up in ORD while there are region issues.

Can we clone machines as standby machines cross region?

We want to avoid always having these in the mix, as there are increased response times from ORD to our PlanetScale DB running in us-east-1.

We are aware you can flag the clone as a standby in a new region, more so want to be confident in how this will work in the case there are issues or maintenance again in IAD.

[mayailurus]

@danwetherald

more so want to be confident in how this will work in the case there are issues or maintenance again in IAD.

In that same vein, it would be nice if there was a way for us ordinary users to test the standby feature in general and verify that its behavior matched our possibly incomplete interpretation of the docs, etc.

As I understand it, it only kicks in when the underlying physical host machine fails—not on network glitches, fly m destroy --force, etc.—so there is no real way to trigger it intentionally for experiments.