Retain your snapshots from 1 to 60 days, your choice

Fresh Produce
1

Starting today, you can customize how long we retain snapshots of your volumes. You can keep them for 1 day all the way up to 60 days. The default continues to be 5 days.

This helps you tune your volume snapshot availability based on your own retention needs.

Setting the snapshot retention

Set the snapshot retention for your volumes when creating or updating volumes with the Machines API or flyctl. For example, this API call will create a volume that retains snapshots for 10 days:

curl -i -X POST \
      -H "Authorization: Bearer ${FLY_API_TOKEN}" -H "Content-Type: application/json" \
      "https://api.machines.dev/v1/apps/my-app-name/volumes" \
      -d '{
        "name": "my_app_vol",
        "region": "ord",
        "size_gb": 10,
        "snapshot_retention": 10
      }'

This API call will update an existing volume to have a snapshot retention of 14 days:

curl -i -X PUT \
      -H "Authorization: Bearer ${FLY_API_TOKEN}" -H "Content-Type: application/json" \
      "https://api.machines.dev/v1/apps/my-app-name/volumes/vol_340088w293z35lp4" \
      -d '{
        "snapshot_retention": 14
      }'

flyctl can do this with the --snapshot-retention flag when creating or updating a volume. For example:

fly volume create --snapshot-retention 10 my_app_vol
fly volume update --snapshot-retention 14 vol_340088w293z35lp4

You can also set your desired snapshot retention on mounts in your fly.toml. Any volumes that fly deploy creates will have the configured snapshot retention. For example, this will ensure all volumes created by fly deploy will have snapshot retention set to 14 days:

[mounts]
  source = "myapp_data"
  destination = "/data"
  snapshot_retention = 14

This require flyctl v0.2.48 or later.

How we built this

Volumes are LVM Logical Volumes under the hood. A volume snapshot is created using the LVM create snapshot feature. We compress the snapshot and upload it to s3 compatible storage.

Before today, we had a single lifecycle rule on our s3 compatible storage that expired all objects after 5 days, which is why every volume snapshot was retained for 5 days. That was simple and worked great. The tradeoff is all objects in the bucket get the same retention.

Now we add a tag to each object we upload that specifies how long to keep the object. For example, we’ll set a retention=10days tag on snapshot objects when the volume has snapshot retention set to 10 days.

We wrote new lifecycle rules, one for every tag value 1–60days, to implement the configurable retention periods. They look like this:

{
    "Rules": [{
        "ID": "expire-after-01-days",
        "Status": "Enabled",
        "Filter": {
            "Tag": {
                "Key": "retention",
                "Value": "01days"
            }
        },
        "Expiration": {
            "Days": 1
        }
    },
... other rules omitted for brevity ...
    {
        "ID": "expire-after-60-days",
        "Status": "Enabled",
        "Filter": {
            "Tag": {
                "Key": "retention",
                "Value": "60days"
            }
        },
        "Expiration": {
            "Days": 60
        }
    }]
}

And that’s it! Our s3 compatible storage will automatically expire objects based on the configured snapshot retention on the volume.

Caveats

We do support updating the snapshot retention for volumes, but we don’t retroactively change the retention on any existing snapshots. New snapshots created after the volume is updated will use the new retention value.

We’re working on exposing the retention of individual snapshots in the Machines API and flyctl. Stay tuned for updates on that!

We automatically create a new snapshot for each volume every day. You can create snapshots anytime using the Create an on-demand volume snapshot api, or fly volume snapshots create <volume_id>.

We are figuring out billing for volume snapshots. We’ll give you advanced notice and plenty of time to make changes to snapshot retention before we turn on billing.

2 Likes
2

Does that mean there’s potential for storage discounts for turning off snapshots for applications that don’t need them?

2 Likes
3

We are still deciding how we’re going to do billing for snapshots. We might include some number of snapshots in the existing volume pricing. Extra emphasis on might, because we have not made a decision yet.

We’d love to hear how folks are using volume snapshots and what challenges/wins you’ve experienced. That will help inform our pricing decision.

1 Like
4

The Machines API now includes a retention_days field in the Get a list of snapshots for a volume api, so you can tell how long each snapshot will be retained.

retention_days will be the same for all snapshots in most cases. They can be different when you change snapshot_retention on the volume. We also extend the snapshot retention when a volume is on a host that dies to give you extra time to recover that volume.

You can see retention days for each snapshot using flyctl v0.2.52 and higher:

% fly vol snapshots ls vol_v30902mlxmed6nwv
Snapshots
ID                              STATUS  SIZE            CREATED AT      RETENTION DAYS
vs_mkxklBLG0zMQUy0P4PoK1        created 26713263        1 hour ago      6
vs_8YZY3wXb01GLsoeRYKgV         created 26713263        1 day ago       6
vs_Vo5opGbqgnzYHgp5GPKb         created 26713263        2 days ago      7
vs_91513VlZQJ4oCjm9mjRZm        created 26713263        3 days ago      6
vs_mkxklBLG0zMQU9o3XKBYG        created 26713263        4 days ago      6
vs_3BPB3R6zYMLOUDBGajPJG        created 26713263        5 days ago      6
vs_GPzP65DAgkeVUqGeB7oMn        created 26713263        6 days ago      6
vs_91513VlZQJ4oC0gxaKgxJ        created 26713263        1 week ago      8
3 Likes
5

Both your Machines API documentations show a wrong http method for the volume update endpoint. It should be PUT but they say POST.

6

Thanks! Fixed in the main docs:

1 Like