[Fresh Produce] Volumes endpoints in Machines API

Fresh Produce
1

Hi there! We’ve been working behind the scenes to migrate the API endpoints for volumes to our Machines API.
This is better for you because our GraphQL API is only hosted in iad region; and when you create a volume in syd region while already nearby (say, deploying a new side project to production from your holiday in Bondi Beach), you shouldn’t need to make a request all the way to iad if it’s just going to come back to a server near you.
It’s faster and more reliable, and it’s already how creating a Machine works today!

If you’re not already familiar with the Machines API, take a look at the guide on our docs.

We’ve also just released a flyctl prerelease version (v0.1.66-pre-1) with these changes included. Install it with: curl -L https://fly.io/install.sh | sh -s pre.

Endpoints

  • List Volumes: GET /v1/apps/{app_name}/volumes → returns []Volume
  • Create Volume: POST /v1/apps/{app_name}/volumes → returns Volume
  • Get Volume: GET /v1/apps/{app_name}/volumes/{volume_id} → returns Volume
  • Get Volume Snapshots: GET /v1/apps/{app_name}/volumes/{volume_id}/snapshots → returns []VolumeSnapshot
  • Extend Volume: PUT /v1/apps/{app_name}/volumes/{volume_id}/extend → returns ExtendVolumeResponse
  • Delete Volume: DELETE /v1/apps/{app_name}/volumes/{volume_id} → returns Volume

Types

These types should be equivalent to the ones from our GraphQL API.

type Volume struct {
	Id              string    `json:"id"`
	Name            string    `json:"name"`
	State           string    `json:"state"`
	SizeGB          int       `json:"size_gb"`
	Region          string    `json:"region"`
	Zone            string    `json:"zone"`
	Encrypted       bool      `json:"encrypted"`
	AttachedMachine *string   `json:"attached_machine_id"`
	AttachedAlloc   *string   `json:"attached_alloc_id"`
	CreatedAt       time.Time `json:"created_at"`
	Blocks          int       `json:"blocks"`
	BlockSize       int       `json:"block_size"`
	BlocksFree      int       `json:"blocks_free"`
	BlocksAvail     int       `json:"blocks_avail"`
}

type VolumeSnapshot struct {
	Id        string `json:"id"`
	Size      int    `json:"size"`
	Digest    string `json:"digest"`
	CreatedAt string `json:"created_at"`
}

type CreateVolumeRequest struct {
	Name              string `json:"name"`
	Region            string `json:"region"`
	SizeGB            *int   `json:"size_gb"`
	Encrypted         *bool  `json:"encrypted"`
	RequireUniqueZone *bool  `json:"require_unique_zone"`
	MachinesOnly      *bool  `json:"machines_only"`

	// restore from snapshot
	SnapshotID *string `json:"snapshot_id"`
	// fork from remote volume
	SourceVolumeId *string `json:"source_volume_id"`

	ComputeRequirements *api.MachineGuest `json:"compute"`
}

type ExtendVolumeRequest struct {
	SizeGB int `json:"size_gb"`
}

type ExtendVolumeResponse struct {
	Volume       Volume `json:"volume"`
	NeedsRestart bool   `json:"needs_restart"`
}

Please let us know if you find any bugs with these new endpoints.

A technical note you might find interesting: our API has internal numerical IDs for volumes, and up until today our GraphQL API was using an auto-incrementing ID generated by its PostgreSQL database. To create internal volume IDs without going all the way to iad and back, each flyd has a Snowflake ID generator that can mint you a new volume ID without talking to anyone else - much less a database on the other side of the world!

11 Likes
2

Awesome :purple_heart:

Will it also be available on the /.fly/api unix socket?

It looks like the docs link is not working?

3

Docs link is fixed, it was a bad copy-paste.

Right now, the /.fly/api socket only supports getting/setting metadata. We’re looking at expanding this but don’t have any plans yet.

2 Likes
4

@containerops I’m curious about your use case for using /.fly/api directly for volume endpoints. Right now that should be available through api.machines.dev so I wonder if you spotted a benefit of having it right there

1 Like
5

I don’t have a use case for the volume endpoints right now, mostly because Fly already does a good job exposing volume metrics on prometheus. So I’m not really suggesting having those endpoints available on /.fly/api.

But I really like the idea of interacting with an authenticated unix socket (/.fly/api). I can also use it to auto-detect that the app is running on Fly, and it’s easier than calling GCP/Azure metadata endpoints.

1 Like
6

Thanks for sharing. I became very interested on this use case of yours:

Is it something like your app becomes aware it’s running on fly because you read that file?

7

Hi, @lubien
This change breaks the fly terraform provider And one of our starter templates.
Please see

Any feedback from the team is appreciated.

Best,
Atif

2 Likes
8

Bumping this up. For those of us that are using it, we’re entirely down. It’s as if flyctl couldn’t spin up volumes anymore.

Also, please consider bringing the terraform provider in as a first class citizen, and not deleting APIs until the provider has been updated to account for the change.

Terraform may not demo as well as flyctl on a blog post, but for apps w/ any complexity at all, it’s a dramatically superior way to manage infrastructure than anything you can conceivably add to flyctl.

That’s not a knock on flyctl. It’s just that they work fundamentally differently.

Terraform lets me say: “Make everything look like this”.

Fly forces me to know what’s there, and then do:

  • Ok, now add this app, but add the flag to not deploy it.
  • Now add this secret.
  • Now add this db.
  • Ok now deploy for real.
  • Great, now pull that info out, and log in to cloudflare.
  • Oh wait no it crashed because I needed to add that secret from Auth0. Add that
  • Now deploy again
  • Now update cloudflare
  • Now create that cert

And then, I have no idea if my staging environment REALLY looks like sandbox or production at all.

3 Likes
9

Somewhat related: When we switched our Terraform configs to create volumes through the machine endpoints via Mastercard/restapi we also ran into issues with org level tokens, where an app would only become accessible via the machines API after a short delay, breaking the entire plan.

We currently deploy using a personal access token to work around the issue. I assume folks may still run into this problem since I haven’t heard any updates yet since our last exchange with support.

10

@mootari - Would you mind dropping your Mastercard code for managing volumes? I’ve been trying to fix the src in the provider in the PR for this: Migrate to flyctl/api as the source of truth by DAlperin · Pull Request #247 · fly-apps/terraform-provider-fly · GitHub but it seems the overall goal of the PR was much larger than fixing this issue.

1 Like
11

Sure:

terraform {
  # ...
  required_providers {
    # ...
    restapi      = { source = "Mastercard/restapi", version = "1.18.2" }
    time         = { source = "hashicorp/time", version = "0.9.1" }
  }
}

provider "restapi" {
  alias                = "fly"
  write_returns_object = true
  uri                  = "https://api.machines.dev/v1"
  headers              = {
    Authorization = "Bearer ${var.FLY_API_TOKEN}"
  }
}

resource "restapi_object" "fly_volume_db" {
  provider = restapi.fly
  path         = "/apps/${fly_app.apps["db"].name}/volumes"
  data         = jsonencode({
    "name": replace(fly_app.apps["db"].name, "/[^a-z0-9]/", "_"),
    "region": local.region,
    "size_gb": 1,
  })
  # Resizing is done via the /extend endpoint. We're not going to deal with that
  # and instead force the volume to be recreated on any size change.
  force_new    = ["app", "name", "size_gb"]
}

resource "time_sleep" "db" {
  # Wait for the volume to no longer be registered as attached to the machine.
  destroy_duration = "60s"

  lifecycle {
    replace_triggered_by = [ restapi_object.fly_volume_db ]
  }
}

resource "fly_machine" "db" {
  # ...
  mounts = [{
    path   = # ...
    volume = restapi_object.fly_volume_db.id
  }]

  lifecycle {
    replace_triggered_by = [ time_sleep.db ]
  }
}
1 Like
12

Thank you!

13

Also, FWIW, this seems to be working for me:

1 Like
14

Is there any work going on to fix the terraform provider anytime soon?
Thanks.

Atif

1 Like
15

Sorry to keep pushing this, but I really think this needs to be a priority. Or, if it won’t be, then letting us know. This for me is a reason to abandon the platform, in spite of how much I absolutely love everything else about what you’re doing.

1 Like
16

Hi @matifali and @belay, the Terraform provider in the fly-apps repository has been a best-effort project, and after internal discussion I’ve just updated the README to reflect that it’s currently not maintained.

At our current size we have to make deliberate decisions on where to focus our staffing, and for the moment, developing a fully-featured Terraform provider doesn’t get the green light.

We understand that deployment with Terraform is attractive to many (lots of us are fans!) and we may be in a position to support it in the future.

4 Likes
17

Thanks @catflydotio for the clarification, and sad for the decision.

1 Like
18

Hi @catflydotio

Would fly be open to community managers for the repo?

I’m sure there’s enough of the community that would be open to creating PRs but at the moment it sounds like fly doesn’t have the resources to even look at the PRs.

1 Like
19

Thank you for the clarification, though… :cry:

1 Like
20

Hi @charsleysa, I think the best answer I can give for this is that the community is welcome to fork the fly-apps Terraform provider. I can appreciate that this is not the most satisfying answer, given that we also don’t promise never to pick this up again.

Ultimately it’s code that handles users’ API keys. There’s an expectation that we’d ensure no malicious code slipped into the repo we host, so we’d still have to have someone here responsible for vetting PRs even with community management on the fly-apps repo.