fly volumes delete failing

Howdy;
We’re using anchor volumes to set deployment regions, and in trying to migrate from one region to another I’m attempting to do fly volumes delete <volume_id> to leave a particular region and getting this error:

Error failed deleting volume: unhandled upstream service error: error closing encrypted device: error creating volume: device-mapper: remove ioctl on data_0-app_24744_ephemera_cache_38389-plain  failed: Device or resource busy
device-mapper: remove ioctl on data_0-app_24744_ephemera_cache_38389-plain  failed: Device or resource busy
...
device-mapper: remove ioctl on data_0-app_24744_ephemera_cache_38389-plain  failed: Device or resource busy

Device /dev/mapper/data_0-app_24744_ephemera_cache_38389-plain is still in use.
: exit status 5

I tried this for a few of the other volumes I’m wanting to remove, and getting similar errors.

Any help is appreciated!

Hi @eyepulp,

Just to check the obvious – is the volume still mounted on the most recently-deployed version of your app? i.e. does your fly.toml have a mounts section like this (Volumes) if your volume is called “myapp_data”:

[mounts]
source="myapp_data"
destination="/data"

@catflydotio – Yep, the Volume is still listed in the toml (because the app still depends on volumes).

I’m just trying to exit a region that has a volume already spun up. I’ve set up new volumes in other regions, but I don’t know how (in this case) to make the instance runnning in mia go away. I’m fine losing the volume data stored under the mia instance, which is why I tried to delete it.

What’s the proper way to migrate instances from one region to another if you’re using anchor scaling? Scale down the app instances, and hope it leaves the region you want to shut down, delete that volume, then scale back up?

We’re looking at this. That error means the VM using the volume wasn’t stopped successfully.

This volume is not in miami, though. It’s running in sjc. Are you sure that’s the one you want deleted?

Howdy @kurt

Checking my bash history, you’re 100% right, I was hitting the sjc volume ID – I did that after encountering the failure in mia trying to see if it was transient or general in behavior. My cut & paste above was the sjc results, but identical to what was happening in mia.

Both vol_1g67340gjj2rydxw and vol_53q80vd8njx4gzy6 were erroring when attempting to delete them.

Generally speaking, I’m wondering if there’s a proper path to migrating regions when anchor scaling.

This is a problem deleting encrypted volumes when they have an attached VM. You’re doing it right, we just need to fix the bug.

The whole anchor volumes concept is a little flawed. Do you actually need persistent volumes? If not, your life is probably going to be simpler if you remove volumes, set the regions you want, then run:

fly scale count 14 --max-per-region=1

It’s almost never worthwhile to run multiple VMs in a single region. Load shifts to the next nearest during bursts, so one VM per region (surprisingly) usually works best.

Ironically, the move to consolidate multiple instances into regions came from a closer read of the 5hr CDN code. We started with that as a base, and the distributed cache is tied to instances in the same region (which makes sense for speed and locality).

ips=$(dig aaaa $FLY_REGION.$FLY_APP_NAME.internal @fdaa::3 +short | awk '{ printf "[%s]\n", $1 }')

Anyhoo – the volumes are there for cached assets, so we definitely want them, but their contents aren’t sacred and will get rebuilt over time as we sort out our regional instance mix.

We have a test bugfix out in mia. Will you try deleting your miami volume again?

@kurt I apologize for the delayed response - just tried the mia deletion and it worked.

Thank you for the saturday fix!

@kurt - this can wait until next week (please enjoy the IL weather today) but I’m getting the error in lax:

❯ fly volumes delete vol_okgj545pjjvy2wzp
Deleting a volume is not reversible.
? Are you sure you want to delete this volume? Yes
Error failed deleting volume: unhandled upstream service error: error closing encrypted device: error creating volume: device-mapper: remove ioctl on data_0-app_24744_ephemera_cache_10092-plain  failed: Device or resource busy
device-mapper: remove ioctl on data_0-app_24744_ephemera_cache_10092-plain  failed: Device or resource busy
...
Device /dev/mapper/data_0-app_24744_ephemera_cache_10092-plain is still in use.
: exit status 5