Migration stalled waiting for volume

Questions / Help
, , ,
1

I attempted to migrate my project (divine-dust-2370) to apps v2, but it is stick part way through the migration because the new machine instances don’t have a volume available. I thought this would be taken care of with the recent enhancement to the migration tool: fly migrate-to-v2: Apps with volumes support 🎉

But my logs for the new machines show that they can’t access the directory where it expected the volume to be mounted and if I run fly list volume I can see that there are no new clones created:

$ fly volumes list
ID                      STATE   NAME            SIZE    REGION  ZONE    ENCRYPTED       ATTACHED VM     CREATED AT   
vol_gez1nvx79qj4mxl7    created drawings        1GB     ord     4f52    true            89af3e86        1 year ago  
vol_ypkl7vzgloerqg60    created drawings        1GB     syd     039e    true            77ed5ccf        1 year ago  
vol_podq4qylj3nvg8w1    created drawings        1GB     lhr     81b8    true            4a0ccae2        5 months ago
2

I tried to get the migration unstuck by running fly machines list and fly machines destroy X for the 3 machines. This stopped the constant crash and re-boot cycle, but the app still shows status: “Suspended”

3

I also tried changing the app back to the v1 platform by running fly apps set-platform-version nomad after making sure that there were no machines running and only the original volumes. This command worked and I now see the prompts to migrate to v2 again, but the app still shows the status of “Suspended”

4

For reasons that made sense at the time, Machines apps with zero machines set the “suspended” flag on an app - which used to describe a Nomad app that was paused (“suspended”). You can run the deprecated command fly resume <appname> to restore the Nomad app.

We’ve recently noticed a possible regression in migrate-to-v2 since moving our volumes API to a different service, although we’re still investigating the root cause. I’ll try to keep you up-to-date when we have more info to share about that.

In the meantime, feel free to keep it on V1. Now that your app is on nomad again, it should be working just as it was before (after resuming it).

1 Like
5

Thanks. I ran fly resume MyApp and that set the scale back to a single instance. I then had to do fly scale count 3 to get back to 1 instance per region in the 3 regions where I have volume allocated for the app.

My app stores region-specific data so I need to keep 1 instance running in each region so that they can cross-talk to find data from other regions.

Thanks for the help

6

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.