Some issues with Volumes on Apps v2

Questions / Help Build debugging
1

Hey here, Apps v2 (Machines) includes a lot of really cool changes that have been useful to me, but I’ve also run into a lot of issues with Volumes, and the following is a list of the issues that exist now.

ISSUE.1 the add Volume listed in the documentation does not apply to v2

In fact, we can create a volume as described in the Volume documentation, but this is under the app scope and not connected to machines, which can be very confusing for new users, so why can I create a volume and not show it on the machine?

When using machine, we may need to use other ways to connect to Volume, but this should be marked in the documentation, as it is completely different from the behavior of v1, where we only declare source in the fly.toml file to connect. (I guess someone may be stuck here.)

ISSUE.2 command machine update cannot update Volume

If you have deployed a machine once and it starts successfully, and then you create a Volume under the App, it is not possible to connect them via fly machine update, because a running machine cannot update the Volume.

$ fly m update <id>  --volume <volume_name>:<path_name>

> failed to update VM <id>: unable to add volume after machine is created

Even if you stop the instance (fly m stop <id>), you cannot update the configuration of the Volume.
If the update command doesn’t support --volume, it shouldn’t be listed in --help, which is very confusing. Perhaps some other temporary factor is suspending this command?

ISSUE.3 command clone does not work properly

To add Volume to Apps v2, we have another way to bypass fly m update, which is to clone an identical machine but add an additional --attach-volume command:

fly m clone <id> --attach-volume <vid>

Although you were able to successfully create a machine with the same configuration, note that this machine does not contain Volume, and the --attach-volume command does not actually work or even throw an error.

ISSUE.4 command fly m run lacks clear guidelines (with Volume)

Now, another way to add Volume to Apps v2 is to preboot a machine with Volume using the fly m run command and then redeploy (or tune) the application.

$ fly m run . -v <vid>:<path> --region <volume_region>

It looks good now, I created a machine in the same region as Volume and successfully added Volume to the machine.
However, machines created by fly m run will be automatically destroyed by an error when deployed using fly deploy. deploy will immediately kill machines that are not started by the app process group.

So we also need to add additional metadata for the machine:

$ fly m update --metadata fly_platform_version=v2 <id>
$ fly m update --metadata fly_process_group=app <id>

Please note that although we appear to have specified both volume and target_path, we have actually only modified volume (volume_id), so the following configuration needs to be added to the fly.toml file again for it to work properly:

[[mounts]]
  destination = "<pathname>"

Well, it’s finally working. But be careful, if you ever accidentally destroy or rebuild a machine, it will all have to start all over again.

I don’t think anyone other than the Fly staff has used Apps v2 (with Volumes) correctly, it’s a terrible development experience where we have to guess at the developer’s intentions from all the error messages and guess at how the parameters work from the fragmented documentation.
Fly is a good platform, and everyone here will agree, but I think Fly’s commitment to documentation and testing is too low, and developers should be using cloud services with a smile on their face, not spending their days guessing how fly’s engineers think.

2 Likes
2

Hey, thanks for posting this! I was having trouble figuring out why a volume wasn’t attaching to my V2 machine, and these steps helped a lot. Now, I can see the mount attached to the machine at /data both via Get a Machine and fly vol list.

However, fly deploy still gives the following error:

[app] does not have a volume configured and fly.toml expects one with destination /data;
remove the [mounts] configuration in fly.toml or use the machines API to add a volume to this machine

Is this something you encountered?

Edit: Funny enough, if I simply remove the [mounts] section, I get an error suggesting that the volume is mounted…

[app] has a volume mounted but app config does not specify a volume;
remove the volume from the machine or add a [mounts] section to fly.toml
3

Ok, it turns out those errors were because somehow I ended up with two machines for the app, one with a volume attached and one without. I deleted the one that had no volume attached and now the fly deploy error looks like this:

WARN Machine $machineId has volume '$volumeName' attached but fly.toml have a different name: ''
Error: failed to update machine configuration for $machineId [app]: machine in group 'app' needs an unattached volume named ''
1 Like
4

After deleting the app and recreating it, I was able to have the machine launch successfully with the volume attached. I needed a section like the following:

[[mounts]]
  source = "volumeName"
  destination = "/data"

The output from fly deploy showed that a volume named volumeName is actually created dynamically.

Creating 1GB volume 'volumeName' for process group 'app'. See `fly vol extend` to increase its size
Process groups have changed. This will:
 * create 1 "app" machine

Note that if you already have a volume created with the same name (you might have one because various docs say to run fly vol create), fly still ignores that one and creates a whole new volume!

I think the version of flyctl actually matters because the behavior is so different across versions; I’m on fly v0.0.522.

3 Likes
5

Thank you @lynshi

Setting mounts in fly.toml file worked as expected.

1. Find machine id 

$ fly machine list

And check machine id and machine name

2. Update metadata

$ fly machine update --metadata fly_platform_version=v2 <machine id>

3. Set mounts

In your fly.toml file

# ...

[[ mounts ]]
source = "YOUR_MACHINE_NAME_FROM_#1"
destination = "/data"

4. Deploy

$ fly deploy
6

I still cannot seem to add a volume to my app. I’ve tried everything in this thread.

I keep getting the error:

Error: machine [id goes here] [app] does not have a volume configured and fly.toml expects one with destination /volume; remove the [mounts] configuration in fly.toml or use the machines API to add a volume to this machine

After adding a [mounts] section to my fly.toml and redeploying the app.

Also, when running fly volumes list, the volume’s “ATTACHED VM” column is empty, no matter what.

Any ideas?

Update: Managed to get a bit further, by fully restarting the app using fly scale count 0 and then deploying again. But now it seems my app is creating a new volume every time I deploy it, which I don’t want. Any ideas?

7

There are some bugs and design imperfections in it, which need to be coordinated and redesigned by Fly officials as a whole to make developers have the expected experience. In your case, I recommend that you follow the following process to redeploy, which will be helpful：

  1. Create volume in app

You must specify the location when creating the volume, otherwise it will not be applied to the machine later.

# create new volume
fly vol create <volume_name> -s 1 --region sjc

# get volume id
fly vol list
  1. Create a machine by run

The run command is now the only command that can specify Volume when creating a machine, and it turns out that fly m clone should work as well, but in my tests, clone has an issue.

fly m run . -v <volume_id>:<your_app_path> --region <volume_region>

# e.g.
# fly m run . -v vol_1aiehbub88ddxz001w:/app/public --region sjc
  1. Validate your machine

Now use the fly m list command to look at your machine, and if all is well, you should see that the machine’s volume item has been marked with the specific volume id.

# get machine id
fly m list

# get machine details
fly m status <machine_id>
  1. Update metadata

Please note that you should not redeploy your project yet, as the machine’s metadata has not been updated and once you run the deployment, you will get an error or simply remove the machine.

# update fly_platform_version
# without this flag your machine cannot be deployed with the `deploy` command
fly m update --metadata fly_platform_version=v2 <machine_id>

# update fly_process_group
# without this flag your machine will be automatically deleted when you deploy
fly m update --metadata fly_process_group=app <machine_id>
  1. Update fly.toml

You can now update the Fly configuration file, but please note that unlike the documentation, you cannot declare both source and destination in the v2 version.

I don’t know when this change was made, but in v2 you can’t declare the source attribute or it won’t deploy. However, since we have already specified that the Volume is mounted on the machine, we only need to specify dest:

[[mounts]]
# The path here is up to you, 
# but usually your project folder is in the "/app" folder directory.
# And it would be easier if you just mapped other absolute paths on your system
  destination = "/app/public/uploads/my_photos"

Yes, although we specified both source:dest in the run command for Volume, only source is actually working in mahcine. After my testing, now that our stated mounts.destination is fully valid, the logic of this product is a bit strange.

After updating to v0.0.526 of flyctl, we are able to declare mounts.source in the configuration file again. Now we modify the following:

# fly.toml

app = "strapi-prod"
kill_signal = "SIGINT"
kill_timeout = 5

# Must be in the same region as volume
primary_region = "sjc"

[[mounts]]
  source = "<volume_name>"
  destination = "/app/public/uploads/my_photos"
  1. Redeploy

Well, now we can make a summary: you have a fly.toml under your project, a .dockerfile at the root of the project, and the Volume is already mounted on the machine. it’s time to redeploy your project:

fly deploy --remote-only --verbose

In the future, if you don’t make any changes, you can always just deploy the project using fly deploy.

1 Like
8

Thank you for all the info above.

There’s also a bug that happens most of the times when deleting a volume and it is attached to a machine (either running or stopped):

$ fly volumes destroy --yes vol_0o6d42g70y3v87gy
Error: failed destroying volume: message

Does this mean that we no longer can “scale down” by deleting a volume? (like with apps V1). The worst problem is that this sometimes happens after the volume reported as AttachedMachine: null.

Also, there’s something not right with the documentation here:

fly machines launch

Should probably be fly launch.

9

Update: Last week, we updated the Volumes docs, specifically Add Volume Storage · Fly Docs, to explain how you can now more easily add volumes, including:

  • adding a volume when you launch an app
  • adding a volume to an existing app
  • adding a volume when you clone a Machine

Hope this helps!