Over the last ~45 minutes (starting around 17:50 UTC) I’ve been having strange issues with my instances and volumes.
My (bootspoon-yb-masters) app could no longer scale to count=3, stopping at count=2 with no indication that Fly was still working on scaling.
I decided to destroy the app (I needed to rename it anyway), and began deleting the volumes in prep for the app destroy. One of the volumes refuses to delete:
Error failed deleting volume: unhandled upstream service error: error renaming lv: error creating volume: Existing logical volume "app_141831_bootspoon_yb_master_58593" not found in volume group "data_0"
: exit status 5
That volume is still visible in fly volumes list.
I left that alone and recreated the app under the new name I wanted (bootspoon-yugabyte). The first few deploys failed “due to unhealthy allocations”, with logs for some instances stopping at Uninitialized volume 'bootspoon_yugabyte', initializing....
This new app had some kind of ghost instance running too. fly status and fly ips private showed the expected number of instances (either three or zero, varying), but fly proxy -s allowed me to select an extra, secret instance to proxy to. I checked fly status --all and only the expected number of instances were marked as running. I couldn’t identify the status of the secret instance, since fly proxy -s only gave me its IP and not its VM ID.
I think the new app may have had scaling problems too? I’m losing track.
I deleted the app and recreated it, but some volume creations are erroring out with Error failed creating volume: Timeout on CreateVolume.app.
I’m still unable to delete that one volume, and I now have a second volume I can’t delete:
$ fly volumes delete vol_wod56vjy609vny30
Deleting a volume is not reversible.
? Are you sure you want to delete this volume? Yes
Error failed deleting volume: An unknown error occured.
$ fly volumes delete vol_wod56vjy609vny30
Deleting a volume is not reversible.
? Are you sure you want to delete this volume? Yes
Error failed deleting volume: unhandled upstream service error: error renaming lv: error creating volume: Existing logical volume "app_146409_bootspoon_yugabyte_59877" not found in volume group "data_0"
: exit status 5
$ fly volumes list
ID STATE NAME SIZE REGION ZONE ATTACHED VM CREATED AT
vol_wod56vjy609vny30 created bootspoon_yugabyte 10GB iad 5881 2 days ago
Hi-- thanks for bringing this up! Today, we made some changes to the volume creation/deletion process that might address the issue you’re running into. Can you let us know if you’re able to delete these volumes now?
$ fly volumes list
ID STATE NAME SIZE REGION ZONE ATTACHED VM CREATED AT
vol_18l524y8em3r7zmp created bootspoon_yugabyte 10GB iad 5881 18 hours ago
$ fly volumes delete vol_18l524y8em3r7zmp
Deleting a volume is not reversible.
? Are you sure you want to delete this volume? Yes
Error failed deleting volume: unhandled upstream service error: error renaming lv: error creating volume: Existing logical volume "app_146409_bootspoon_yugabyte_61045" not found in volume group "data_0"
: exit status 5
Thank you for sharing those additional logs. This helped us drill down to a specific host-- once we had that, we could run a more targeted fix. I hope that does the trick!
I successfully deleted the two volumes noted here. However I tested creating + deleting some new volumes, and now have another undeletable volume:
$ fly volumes list
ID STATE NAME SIZE REGION ZONE ATTACHED VM CREATED AT
vol_x915grnzp58rn70q created bootspoon_yugabyte 10GB iad 5881 2 minutes ago
$ fly volumes delete vol_x915grnzp58rn70q
Deleting a volume is not reversible.
? Are you sure you want to delete this volume? Yes
Error failed deleting volume: unhandled upstream service error: error renaming lv: error creating volume: Existing logical volume "app_146409_bootspoon_yugabyte_61827" not found in volume group "data_0"
: exit status 5
`