Howdy;
We’re using anchor volumes to set deployment regions, and in trying to migrate from one region to another I’m attempting to do fly volumes delete <volume_id> to leave a particular region and getting this error:
Error failed deleting volume: unhandled upstream service error: error closing encrypted device: error creating volume: device-mapper: remove ioctl on data_0-app_24744_ephemera_cache_38389-plain failed: Device or resource busy
device-mapper: remove ioctl on data_0-app_24744_ephemera_cache_38389-plain failed: Device or resource busy
...
device-mapper: remove ioctl on data_0-app_24744_ephemera_cache_38389-plain failed: Device or resource busy
Device /dev/mapper/data_0-app_24744_ephemera_cache_38389-plain is still in use.
: exit status 5
I tried this for a few of the other volumes I’m wanting to remove, and getting similar errors.
Just to check the obvious – is the volume still mounted on the most recently-deployed version of your app? i.e. does your fly.toml have a mounts section like this (Volumes) if your volume is called “myapp_data”:
@catflydotio – Yep, the Volume is still listed in the toml (because the app still depends on volumes).
I’m just trying to exit a region that has a volume already spun up. I’ve set up new volumes in other regions, but I don’t know how (in this case) to make the instance runnning in mia go away. I’m fine losing the volume data stored under the mia instance, which is why I tried to delete it.
What’s the proper way to migrate instances from one region to another if you’re using anchor scaling? Scale down the app instances, and hope it leaves the region you want to shut down, delete that volume, then scale back up?
Checking my bash history, you’re 100% right, I was hitting the sjc volume ID – I did that after encountering the failure in mia trying to see if it was transient or general in behavior. My cut & paste above was the sjc results, but identical to what was happening in mia.
Both vol_1g67340gjj2rydxw and vol_53q80vd8njx4gzy6 were erroring when attempting to delete them.
Generally speaking, I’m wondering if there’s a proper path to migrating regions when anchor scaling.
This is a problem deleting encrypted volumes when they have an attached VM. You’re doing it right, we just need to fix the bug.
The whole anchor volumes concept is a little flawed. Do you actually need persistent volumes? If not, your life is probably going to be simpler if you remove volumes, set the regions you want, then run:
fly scale count 14 --max-per-region=1
It’s almost never worthwhile to run multiple VMs in a single region. Load shifts to the next nearest during bursts, so one VM per region (surprisingly) usually works best.
Ironically, the move to consolidate multiple instances into regions came from a closer read of the 5hr CDN code. We started with that as a base, and the distributed cache is tied to instances in the same region (which makes sense for speed and locality).
Anyhoo – the volumes are there for cached assets, so we definitely want them, but their contents aren’t sacred and will get rebuilt over time as we sort out our regional instance mix.
@kurt - this can wait until next week (please enjoy the IL weather today) but I’m getting the error in lax:
❯ fly volumes delete vol_okgj545pjjvy2wzp
Deleting a volume is not reversible.
? Are you sure you want to delete this volume? Yes
Error failed deleting volume: unhandled upstream service error: error closing encrypted device: error creating volume: device-mapper: remove ioctl on data_0-app_24744_ephemera_cache_10092-plain failed: Device or resource busy
device-mapper: remove ioctl on data_0-app_24744_ephemera_cache_10092-plain failed: Device or resource busy
...
Device /dev/mapper/data_0-app_24744_ephemera_cache_10092-plain is still in use.
: exit status 5
# fly volumes destroy vol_gez1nvx0o5zvmxl7
Deleting a volume is not reversible.
? Are you sure you want to destroy this volume? Yes
Error failed destroying volume: You hit a Fly API error with request ID: 01GPZGWRYJVJW9QZKC574KQR07-lhr
# fly volumes delete vol_gez1nvx0o5zvmxl7
Deleting a volume is not reversible.
? Are you sure you want to destroy this volume? Yes
Error failed destroying volume: You hit a Fly API error with request ID: 01GPZGVMPT4PCJ0TA2FDPZTPT1-lhr
