VM won’t launch - volume stuck after host incident in EZE region

[anaibol]

Hi team,

I’ve been unable to relaunch a VM for my app acanomas in region eze due to this error:
Error: failed to launch VM: insufficient resources to create new machine with existing volume ‘vol_vlp08we6g7o2dwp4’ (Request ID: 01K1B3428SX3VCJWD78D1FB2BT-eze)
This started after a host incident more than two weeks ago that affected the machine previously attached to this volume.

Current situation:

• 1 unattached volume in eze: vol_vlp08we6g7o2dwp4
• Trying to scale a performance-1x machine in the same region (group: web)
• Scaling fails with “insufficient resources” error
• I still see this warning:
  WARNING: There are active host issues affecting your app. Please check fly incidents hosts list
  It seems like there’s no available capacity to reattach the volume. What are my options to get the app running again? Can the volume be moved or force-attached to a new host manually?

Any help would be greatly appreciated!

Thanks!

[lillian]

ah, that error message isn’t very good. volumes are tied to hosts; since this host in eze is dead (as the status message says), this volume is not attachable.
you’ll need to create a new volume, either from snapshot or empty, and then attach that to your machine.

[anaibol]

@lillian

snapshot or empty, and then attach that to your machine.

yes, but i cant destroty the volume to create a new one.

fly vol destroy vol_vlp08we6g7o2dwp4
! WARNING: There are active host issues affecting your app. Please check `fly incidents hosts list` or visit your app in https://fly.io/dashboard

Error: failed to get volume vol_vlp08we6g7o2dwp4: request returned non-2xx status: 408:  (Request ID: 01K1B5EKDXJKPD1TCFY69FXJHP-eze)

[lillian]

ok we should fix that, but you can have multiple volumes in the app, it shouldn’t be a problem.

[anaibol]

ok, can you let me know please? I am currently stuck on deploying vms on that app on that region, that is my main region

[dangra]

hi @anaibol,

We just rolled a fix in flyctl v0.3.163 to unblock scaling the app while it has unavailable volumes. Please update your flyctl version with fly version upgrade and give it a try.

thanks!

[anaibol]

Cool! When running fly version upgrade I get:
Already running latest flyctl v0.3.162

Is v163 already deployed?

[dangra]

Yes, but It may have delays depending on the flyctl installation method you use.

In the meantime, I have flagged the volume as destroyed so it won’t give you troubles scaling the app even with an older flyctl version.

[anaibol]

@dangra

destroyed

Thanks Daniel!

[halfer]

(I’ve amended the title so as not to mislead future readers; I think the length of time since the host incident is not relevant to this bug. I suspect that it would have needed a nudge regardless of whether it was reported last week or several weeks hence ).