Machines OpenAPI spec omissions

General
, , ,
22

@praneetloke the changes went live:

  1. current_version docs updated (link).
  2. PATCH and PUT were doing the same operation because of legacy reasons. PUT is deprecated now. Docs were also updated to reflect what it does to keys that aren’t mentioned.
  3. There is no intention (yet) to convert PUT to “replace all metadata” op
23

@dangra @nullhypothesis thank you for making the updates. I’ll review them asap!

24

@dangra @nullhypothesis I’ve looked at the changes. Here are my thoughts on the current state of the Update Machine endpoint:

  1. Rename the current_version to version in the request body so it lines up with the request body of the Create Machine endpoint. One could take the response from the Create Machine endpoint, tweak the config and update the machine without having to set a new current_version property.

  2. As for the second thing…I previously brought this up and lilian explained that it is POST because the endpoint is creating a new version. I’d argue that the Update Machine endpoint acts as a “replace the configuration of the machine with this config if the version matches”. I think a PUT would be more appropriate since the user would essentially be replacing the entire configuration with a new one. If not, that’s fine and I can try to work around it. I figured I would give it another shot at proposing the change given other changes.

I would love to know what you think.

25

I’d rather not rename attributes or change methods; not because I find your suggestions unreasonable (they’re actually good!) but because every change must be backwards compatible, which means that we’d have to deprecate the old stuff, introduce new stuff, and make sure that nothing breaks. It’s a bit of a hassle and I’d prefer to avoid it as much as possible, even if it means foregoing opportunities to make the API more consistent.

26

It’s a bit of a hassle and I’d prefer to avoid it as much as possible, even if it means foregoing opportunities to make the API more consistent.

Fair enough. I had a feeling that was going to be the case. I’ve been there. I’ll work around those points I brought up about the Update Machine endpoint.

By the way, what is the purpose of the Remove certificate endpoint? I see that both Acme and custom certificates have a corresponding Remove Acme and Remove custom endpoint.

27

that’s for deleting all certificates on the app, whether ACME or custom. (both can be on the app at the same time)

28

Ah, got it. Thank you, @lillian!

29

I’ve made the repo public now that I have gotten a successful generation of Pulumi SDKs from the OpenAPI spec. I will test the SDK with my Fly account and once I confirm things to be working well, I’ll publish the SDKs to the respective registries.

30

I have been testing the Pulumi provider. It mostly works. The following are a few of the observations I made.

Network Policies

Network policies API is listed in the docs here but it’s missing in the spec. I don’t see any other way to manage them.

Moreover, POST /v1/apps/<app_name>/network_policies is used for both create and update. It would be great if there was a separate endpoint for updating a policy similar to the DELETE endpoint: DELETE /v1/apps/<app_name>/network_policies/<policy_id>.

There is also no endpoint to fetch a single network policy by its id.

Storage Objects

I am curious if there is any plan to add APIs to manage the following resources:

  • Fly.io Storage (Tigris) Buckets to the Machines API?
  • Fly Kubernetes Service

I realize it’s Machines API, so maybe a separate “platform” API group?

Rate Limits

The rate limit is easily triggered by the metadata endpoints. Trying to mutate (add/update/delete) multiple metadata keys for a machine will result in 429s. I can work around this by slowing-down the metadata requests for a machine but I wanted to mention it in case there is any room for easing the limits for certain endpoints. The alternate, of course, is that a user manages metadata as part of the machine config rather than using the dedicated /metadata/{key} endpoint. I am going to do more testing and part of it is to port something off Render.

31

tigris is a separate company and product; you can use their own APIs to manage storage buckets, the flyctl integration is mostly just a convenience.

this is deprecated and should no longer be used.

for metadata, raising the limits is probably not workable as each update is expensive on our end, maybe you can use the bulk update endpoint instead?

I’m not personally familiar with network policies.

32

the flyctl integration is mostly just a convenience.

Ah, that’s good to know. I wasn’t sure if there was a special thing with fly’s control plane.

Fly Kubernetes Service

this is deprecated and should no longer be used.

Got it! That’s simple enough :slight_smile:

for metadata, raising the limits is probably not workable as each update is expensive on our end, maybe you can use the bulk update endpoint instead?

Hmm yeah I seem to have forgotten about this endpoint altogether. Thank you for pointing this out to me. I’ll think of a strategy to use this.

I’m not personally familiar with network policies.

Do you know anyone who might be able to look into it?

Thank you as always!

33

I have established a release pipeline to publish the respective SDKs for the Pulumi provider to npm, pypi and NuGet. I’ve spent the last few days diving into the platform as a whole. The basic example in the repo works if you want to give it a try as well.

Here are my notes about the API from trying to use the Pulumi provider and the Fly platform in general.

Machine config

  1. There are some parameters in the machine config that have no description. Morever, the relationship between processes and multi-containers (containers property) is quite confusing to me. Also it looks like multi-containers is only available to certain organizations, so I assume it’s a preview feature?
  2. What is the statics property for? I mean, I am guessing it’s for serving static files but a description would be great. Specifically, I am not sure if this would automatically provision a Tigris storage bucket?

Deleting a machine

  1. I cannot delete a machine if it’s running unless I use the force query param or wait for it to transition to the stopped state automatically if autostop is enabled. I get a 412 Precondition Failed status code if I try to delete a running machine without the force query param. The behavior isn’t a problem but rather there is no mention of this requirement in any of the docs. It would be great if this is documented in the API specs or the core platform docs for Machines.

  2. The response body contains a single boolean property called ok but the API spec says that the response body is supposed to be empty. The platform docs shows what I am seeing. The mismatch in the response body isn’t a problem but I wanted to point out that one of them is incorrect. Either, it’s supposed to return a 204 NoContent without any response body as DELETE endpoints typically do, or, the API spec should be updated to match the correct response as shown in the platform docs.

On a somewhat related note, I was looking into volumes and was curious about SQLite replication with LiteFS, which led me to read about fly consul attach/detach. fly consul ... is another example of an important action that is core to the platform. While this is technically “out-of-scope” for the Machines API, I would love it if a dedicated REST API existed for these sorts of things.

There is also the issue with Network Policies API I brought up previously. I do think it’s important to include this in the Machines API spec as it deals with restricting network egress.

As always, thank you for your time!

34

I have hit a strange problem with app secrets.

The endpoint to create/update an app secret only stages (as if --stage was passed) the secret waiting for the user to click Deploy Secrets which will cause a new machine to be created with the new secrets and the old one to be torn-down. This seems somewhat convenient if I was using flyctl to do everything. But it doesn’t make sense to do this the API.

The strange thing is, according to the docs, staged secrets would be deployed automatically the next time a machine is updated but that didn’t happen for me. I replaced the machine with a different image and yet the status of secrets in the app are showing as staged.

the staged secret will be available only on Machines that are started or updated after the fly secrets set command was run.

And there’s another issue. Machines with volumes. Deploying secrets requires a new machine and if the existing machine has a volume attached to it, the deployment will refuse to proceed since it won’t automatically detach the volume. It would be nice if this was handled automatically by the control plane, i.e. suspend the current machine, detach volume, create a new machine, attach volume, delete old machine…sort of a rolling update. I understand this amounts to downtime if the user only has one machine running their app but I think that should be acceptable.

35

It did start out that way, but it was opened up to everyone in April 2025 (or earlier).

(There are still some rough edges in the multi-container Machines overall, though.)

Hm… I do see the new secret right after updating the Machine, actually, without any intervening steps via flyctl or in the dashboard…

$ curl <token-etc> "https://$API/v1/apps/uc15po/secrets/tuna" \
   --data '{"value": "FISH"}'
{"name":"tuna","value":"FISH","digest":"6f0bc02042fd2646",
 "version":123798940,"Version":123798940}
$ cat config-neo.json
{
  "config": {
    "image": "debian:trixie",
    "init": {
      "cmd": ["sleep", "inf"]
    },
    "min_secrets_version": 123798940
  }
}
$ curl <token-etc> "https://$API/v1/apps/uc15po/machines/<machine-id>" \
   --data @config-neo.json
$ fly ssh console
# printenv tuna
FISH

(Here min_secrets_version matches the value returned by the first POST, and the rest of config-neo.json is the same as when that Machine was originally created.)

How were you determining the status of the secrets? (Some methods of doing so may only be reliable for Machines managed by fly deploy, etc.)

36

the idea behind Machines is to not have a hosted control plane. after you create a Fly Machine, you are talking directly to the physical server running that Machine; the “control plane” is really only a proxy that handles auth and forwards requests to the correct server.

in time we realised this is not what everyone wants; the dashboard now has a “deploy secrets” button that… spawns an ephemeral Fly Machine with flyctl and runs fly secrets deploy, lol.

it is odd that you are not seeing secrets be updated in your Machine after a machine update. can you ssh in and check the secret value? the display has historically been wrong sometimes. there definitely shouldn’t be a need to create an entirely new machine to update the secrets.

if you’re actually seeing secrets not being updated after a machine update while using min_secrets_version, please provide an example machine id so we can take a look.

37

@mayailurus @lillian setting the min_secrets_version on the machine was the key. I was missing that critical piece of information. Neither the Create/Update Machine endpoints have a description for min_secrets_version nor the Create/Update App Secret mentions this.

in time we realised this is not what everyone wants; the dashboard now has a “deploy secrets” button that… spawns an ephemeral Fly Machine with flyctl and runs fly secrets deploy, lol.

Ah I see. That’s what I was seeing and it would fail for the machine in question because I have a volume attached to it.

it is odd that you are not seeing secrets be updated in your Machine after a machine update. can you ssh in and check the secret value? the display has historically been wrong sometimes. there definitely shouldn’t be a need to create an entirely new machine to update the secrets.

Indeed. I think the dashboard is incorrect. Even after I set the min_secrets_version, the dashboard is still showing the app secrets as just staged, requiring a deployment with fly secrets deploy. But I can confirm that the container running in the machine does in fact have the secrets.

There is another thing I noticed with the dashboard. Despite pushing an image to registry.fly.io/<app-name> I don’t see it in the dashboard under Apps > {my-app} > Registry. The page is blank even though the machine is in fact running an image from the registry, which I know was pushed successfully.

By the way, I am using private networking to connect one App (exposed to the internet) to a private app (bound to fly-local-6pn as per the docs). The healthchecks for the publicly exposed app works fine. However, I can’t seem to get the healthchecks for the private app working no matter what I do. I tried the following things to no avail:

  • top-level checks property
  • the config.containers[*].healthchecks property with http, as well as an exec.command by using the FLY_PRIVATE_IP as well as the fly-local-6pn host explicitly.

I just can’t seem to get healthchecks for a private machine to work. I think I am missing some piece of information.

the idea behind Machines is to not have a hosted control plane. after you create a Fly Machine, you are talking directly to the physical server running that Machine; the “control plane” is really only a proxy that handles auth and forwards requests to the correct server.

Got it. How does a machine handle scaling when a persistent volume is attached to it? Or does it not do that? I couldn’t find information about that in the docs that I read about scaling machines.

38

I don’t know very much about the dashboard but I’ll poke the right folks.

healthchecks aren’t supported over 6pn, only on proxy services (using flycast instead of a public ip if you want a private app).

it does not. you need to create a new volume and attach it to a new machine. there is some info about that in this docs page: Fly Volumes overview - Fly.io

39

Great! Thank you.

Ah gotcha. That’s what I suspected on both of those things based on what I was seeing. Thank you for confirming. It makes sense given that these are actual VMs and not containers, even though I am providing “container images” to run on these machines. So they are akin to an AWS EC2 instance with an EBS volume attached, correct? Except, these are fast, lightweight Firecracker VMs.

40

These two older forum threads might be useful in the meantime…

https://community.fly.io/t/cant-see-images-in-my-registry/25436/2

https://community.fly.io/t/should-every-app-have-entries-in-the-registry/26237

My overall impression is that the dashboard is designed primarily for those who are using fly deploy; personally, I don’t mix it with Machines API operations, except for really basic things like start and stop of individual Machines. (Those are just mapped to direct calls to the Machines API, I’m pretty sure, without any extra metadata needing to be juggled.)

(It’ll be interesting to hear where the official dividing line is, though…)

41

Thank you for sharing those links!

This matches with what I am seeing as well.

Ah, gotcha.