VMs hanging during deployment

I’m seeing vms hanging during deployment. In this case, three of the 4 minimum VMs are booting and serving requests successfully, but one never gets past the “Starting instance” stage. This has the effect of rolling back the deploy, even though the other vms are up and serving requests just fine.

Update: I’m seeing this behavior happen consistently, where the last of a 4 vm minimum autoscaling set doesn’t come up. Even if I try to kill the one that’s not coming up with fly vm stop ID, the new vm fly brings up remains in the “Starting instance” state.

Assuming this is probably related to the issue with volumes in IAD, which is where these vms are located: Fly.io Status - Emergency volume array maintenance in IAD