Stranded volume in GRU, cannot attach to new machine

[dkuhnen]

Hi,

The old machine failed and cannot be restarted.

Error: failed to restart machine 7811439a555618: could not stop machine 7811439a555618: failed to restart VM 7811439a555618: unknown: could not reserve resource for machine: governor policy blocked start: hard gate failed: mem_util_exceeds_target (Memory utilization exceeds target + buffer for existing) (Request ID: 01K8JRFS8TJQ7D4Y3TC2K8F8WE-ams)

I tried to clone and attach the existing volume with:

fly machine clone 7811439a555618 -a fiwell-production --attach-volume vol_4ml0lzlg956601dv:/dat

But I always get:

"failed to launch VM: insufficient resources to create new machine with existing volume 'vol_4ml0lzlg956601dv'"

I believe this means the volume is stranded on a host with no capacity.

Could you please migrate this volume to a host in GRU that has resources, so I can attach it to a new machine?

This is production data and currently I can’t even boot a rescue machine to copy it out.

Thank you very much!

App: fiwell-production
Primary region: GRU
Critical volume: vol_4ml0lzlg956601dv
Old machine (failed and destroyed): 7811439a555618
New machine (empty volume): 6e827947bd1d87

[mayailurus]

Hi… You should be able to resolve this in a self-service way using volume forking:

By default, we place the new volume on a separate physical host in the same region.

For larger volumes, it will spend some time in a “hydrating” state, where the data is accessible but with slower/worse performance. It will catch up on its own eventually, though.

Hope this helps!

---

Aside: If you have a Fly Support plan, then it’s generally best to contact them directly, rather than here in the community forum.

[dkuhnen]

Thank you mayailurus.

I fixed it by restoring from a recent snapshot instead:

1. Listed snapshots:

   fly volumes snapshots list <volume_id>

2. Created a new volume from the latest snapshot:

   fly volumes create actual_data --snapshot-id <snapshot_id> --size 1 -a fiwell-production

3. Cloned a healthy machine and attached this new volume:

   fly machine clone <machine_id> -a fiwell-production --attach-volume <new_volume_id>:/data

That new machine started fine and had all my data.