Error: creating a new machine in group 'app' requires an unattached 'data' volume. Create it with `fly volume create data`

1

I think I may have messed things up royally with epic-stack-template. It’s a bit of a unique project since it doesn’t have users and I actually would prefer the volumes to not be persistent because I make changes to the database schema and I actually don’t want to do migrations since it’s a template anyway.

Anyway, I needed to make a change to the database schema and so I went to delete the volume and it wouldn’t let me. So then I noticed the app was still on v1 so I followed the migration process and it somehow created two more volumes and ended up failing. I deleted all the volumes and tried again. It then said I needed a new volume so I made a new one and tried again. That time it worked (so I think I’m on v2 now), but now I can’t deploy anymore. I get the following error:

Error: creating a new machine in group 'app' requires an unattached 'data' volume. Create it with `fly volume create data`

But I definitely have a volume by that name:

$ fly vol list
ID                      STATE   NAME    SIZE    REGION  ZONE    ENCRYPTED       ATTACHED VM     CREATED AT     
vol_xme149km176vowpl    created data    1GB     sjc     514a    true            b6e02481        30 minutes ago

Here’s my status:

$ fly status
App
  Name     = epic-stack-template          
  Owner    = personal                     
  Version  = 121                          
  Status   = suspended                    
  Hostname = epic-stack-template.fly.dev  
  Platform = detached                     

Deployment Status
  ID          = 78a86d05-04e1-940c-e1c0-21ff128e6bd4         
  Version     = v121                                         
  Status      = successful                                   
  Description = Deployment completed successfully            
  Instances   = 1 desired, 1 placed, 1 healthy, 0 unhealthy  

Instances
ID              PROCESS VERSION REGION  DESIRED STATUS  HEALTH CHECKS           RESTARTS        CREATED    
b6e02481        app     121     sjc     run     running 2 total, 2 passing      0               10m23s ago

It also says my platform is detached and my status is suspended. So I’m not sure what I’m doing wrong now… Like I said. I think I really messed it up.

2

There’s definitely a few things going on here. Your app may not have fully migrated to V2; your platform should be Machines. So, you can try this solution first:

Edit to add: try deploying at this point and if the volume error persists, try the steps below :slight_smile:

Then for the volume issue. I’d need to see the exact fly deploy command and the complete output, but my guess is that this is because of the Apps V2 HA feature that creates 2 Machines by default for process groups with services.

You could try creating a second volume, also called data, and then running fly deploy again. Alternatively use the --ha option in fly deploy to turn off the “create 2 machines” feature:
fly deploy --ha=false

3

Thanks! Here’s how far I got:

~/code/epicweb-dev/epic-stack (main) 😅
$ fly resume epic-stack-template
Warning: this command is deprecated. Only use it if you have a previously suspended app. Use 'fly scale count 0' if you need to stop an app temporarily.
resume complete - epic-stack-template is now deployed with 1 running instance
~/code/epicweb-dev/epic-stack (main) 😅
$ fly scale count 0 -a epic-stack-template
Update available 0.1.28 -> v0.1.29.
Run "fly version upgrade" to upgrade.
Count changed to 0
~/code/epicweb-dev/epic-stack (main) 😅
$ fly apps set-platform-version machines -a epic-stack-template
Update available 0.1.28 -> v0.1.29.
Run "fly version upgrade" to upgrade.
Error: cannot migrate to machines platform version with existing nomad allocations

~/code/epicweb-dev/epic-stack (main) 😅
$ fly version upgrade
/bin/zsh -c
Running automatic upgrade [curl -L "https://fly.io/install.sh" | sh]
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1475    0  1475    0     0   7966      0 --:--:-- --:--:-- --:--:--  8104
######################################################################## 100.0%
set channel to shell
flyctl was installed successfully to /Users/kentcdodds/.fly/bin/flyctl
Run 'flyctl --help' to get started
Upgraded flyctl v0.1.28 -> v0.0.450
~/code/epicweb-dev/epic-stack (main) 😅
$ fly apps set-platform-version machines -a epic-stack-template
Error: cannot migrate to machines platform version with existing nomad allocations

~/code/epicweb-dev/epic-stack (main) 😅
$
4

I’m going to see if I can get you some help with that. :sweat_smile:

1 Like
5

I just tried another deploy and it seemed to work :man_shrugging: add docs for caching and server timing · epicweb-dev/epic-stack@c130661 · GitHub

However, when I look at the fly status for this app I get that the platform is “detatched” and it doesn’t show any instances.

epic-stack-template at: 22:30:58

App
  Name     = epic-stack-template          
  Owner    = personal                     
  Version  = 123                          
  Status   = deployed                     
  Hostname = epic-stack-template.fly.dev  
  Platform = detached                     

Instances
ID      PROCESS VERSION REGION  DESIRED STATUS  HEALTH CHECKS   RESTARTS        CREATED

My app is definitely accepting traffic though: https://epic-stack-template.fly.dev/ and healthchecks are passing just fine. I’m in a very weird place.

6

Upgraded flyctl v0.1.28 -> v0.0.450

this is strange!

Could you run fly version and see what it says?

If it says v0.0.450 still, try running fly version update again.

It wouldn’t hurt to reinstall flyctl if it doesn’t update to v1.29.

You can just delete it from $Home.fly\bin (on Mac) and then reinstall:

brew install flyctl

or

curl -L https://fly.io/install.sh | sh
7

Once flyctl is sorted out, you can try running fly migrate-to-v2 again and it should pick up where it left off.

8

And if none of those work, try out the new troubleshooting tool - fly migrate-to-v2 troubleshoot: self-service troubleshooting for migration issues

1 Like
9

These versions are kinda weird. But here’s what I’ve got:

~/code/epicweb-dev/epic-stack (main) 🔮
$ fly version
fly v0.1.31 darwin/arm64 Commit: e2b55f1a9fbe2201ca12f621fef02a47ca2dfcae BuildDate: 2023-06-08T00:09:32Z
~/code/epicweb-dev/epic-stack (main) 🔮
$ fly version update
Error: no available update

So I think I’m good. But I am still concerned about the v0.0.450 thing so I’m running brew install flyctl again just to be safe.

10

Ok, here’s where I’m at:

~/code/epicweb-dev/epic-stack (main) 🔮
$ fly version 
fly v0.1.31 darwin/arm64 Commit: e2b55f1a9fbe2201ca12f621fef02a47ca2dfcae BuildDate: 2023-06-08T00:09:32Z
~/code/epicweb-dev/epic-stack (main) 🔮
$ fly version update
Error: no available update

~/code/epicweb-dev/epic-stack (main) 🔮
$ fly migrate-to-v2
? Choose the primary region for this app: San Jose, California (US) (sjc)
This migration process will do the following, in order:
 * Lock your application, preventing changes during the migration
 * Create machines, copying the configuration of each existing VM
 * Remove legacy VMs after health checks pass for the new machines
 * Set the application platform version to "machines"
 * Unlock your application
 * Overwrite the config file at '/Users/kentcdodds/code/epicweb-dev/epic-stack/fly.toml'
? Would you like to continue? Yes
==> Migrating epic-stack-template to the V2 platform
>  Locking app to prevent changes during the migration
>  Enabling machine creation on app
>  Creating an app release to register this migration
>  Starting machines
INFO Using wait timeout: 2m0s lease timeout: 13s delay between lease refreshes: 4s
Updating existing machines in 'epic-stack-template' with rolling strategy
  [1/1] Updating 5683d264a1d128 [app]
failed while migrating: failed to update VM 5683d264a1d128: server returned a non-200 status code: 500
? Would you like to enter interactive troubleshooting mode? If not, the migration will be rolled back. Yes

Oops! We ran into issues migrating your app.
We're constantly working to improve the migration and squash bugs, but for
now please let this troubleshooting wizard guide you down a yellow brick road
of potential solutions...
               ,,,,,
       ,,.,,,,,,,,, .
   .,,,,,,,
  ,,,,,,,,,.,,
     ,,,,,,,,,,,,,,,,,,,
         ,,,,,,,,,,,,,,,,,,,,
            ,,,,,,,,,,,,,,,,,,,,,
           ,,,,,,,,,,,,,,,,,,,,,,,
        ,,,,,,,,,,,,,,,,,,,,,,,,,,,,.
   , ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,

The app's platform version is 'detached'
This means that the app is stuck in a half-migrated state, and wasn't able to
be fully recovered during the migration error rollback process.

Fixing this depends on how far the app got in the migration process.
Please use these tools to troubleshoot and attempt to repair the app.
No legacy Nomad VMs found. Setting platform version to machines/Apps V2.
~/code/epicweb-dev/epic-stack (main) 🔮
$ fly migrate-to-v2 troubleshoot
Error: app has not been migrated to Apps V2
please try running 'fly migrate-to-v2 troubleshoot' later.
if the problem persists, try bringing it up in the community forum (https://community.fly.io),
or if you have one, your plan's support mailbox
11

Your app looks like it’s successfully on Apps v2 to me. What do you see if you run fly status? Is anything broken? Deploys?

I think we just have a bug where fly migrate-to-v2 troubleshoot prints an error even when there is nothing wrong.

12

Looks like we’re somehow alright now :sweat_smile:

~/code/epicweb-dev/epic-stack (main) 🔮
$ fly status
App
  Name     = epic-stack-template                                        
  Owner    = personal                                                   
  Hostname = epic-stack-template.fly.dev                                
  Image    = epic-stack-template:deployment-01H2DPS8VJ4RX3VMSMZ0YYP7YQ  
  Platform = machines                                                   

Machines
PROCESS ID              VERSION REGION  STATE   CHECKS                  LAST UPDATED         
app     5683d264a1d128  122     sjc     started 2 total, 2 passing      2023-06-08T14:33:28Z

I’m not sure when that changed, but I think we’re good now :sweat_smile: Thanks team!

2 Likes
13

Glad it’s working! This PR should correctly make fly migrate-to-v2 troubleshoot tell you when nothing is wrong.

2 Likes
14

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.