Regional capacity in Machines API and flyctl

Fresh Produce
,
1

We’ve just released a new feature in the Machines API and flyctl to display real-time capacity information for each region. We hope this will be useful to help troubleshoot and track capacity-related issues you might encounter while creating Machines in an overloaded region, and also help with spot-checking capacity planning for larger deployments of a large number of Machines.

To check regional capacity, you can use the new Get Regions Machines API (/v1/platform/regions), which includes a capacity field in its JSON response:

$ curl -s https://api.machines.dev/v1/platform/regions | jq -c .Regions[]
{"code":"ams","name":"Amsterdam, Netherlands","latitude":52.374344,"longitude":4.895439,"gateway_available":true,"requires_paid_plan":false,"geo_region":"europe","capacity":969}
{"code":"arn","name":"Stockholm, Sweden","latitude":59.6512,"longitude":17.9178,"gateway_available":false,"requires_paid_plan":false,"geo_region":"europe","capacity":596}
{"code":"atl","name":"Atlanta, Georgia (US)","latitude":33.6407,"longitude":-84.4277,"gateway_available":false,"requires_paid_plan":false,"geo_region":"north_america","capacity":177}
{"code":"bog","name":"Bogotá, Colombia","latitude":4.70159,"longitude":-74.1469,"gateway_available":false,"requires_paid_plan":false,"geo_region":"south_america","capacity":211}
{"code":"bom","name":"Mumbai, India","latitude":19.097403,"longitude":72.874245,"gateway_available":false,"requires_paid_plan":true,"geo_region":"asia_pacific","capacity":32}
[...]

Here, capacity is a number which represents the total additional number of a specific size of Machine (performance-1x by default) that can be launched in the given region. The API accepts a few query parameters for you to customize the response for different machine sizes and kinds (including GPUs):

$ curl -s 'https://api.machines.dev/v1/platform/regions?size=shared-cpu-1x' | jq -c '.Regions[]|[.code,.capacity]' | head -5
["ams",9563]
["arn",7153]
["atl",2731]
["bog",3377]
["bom",586]

$ curl -s 'https://api.machines.dev/v1/platform/regions?size=l40s' | jq -c '.Regions[]|[.code,.capacity]'
["ord",15]

Finally, as of today’s flyctl release (v0.3.121), the existing fly platform regions command will now call this Machines API (instead of the previous GraphQL query), and will include display capacity info in an extra column in its output (now grouped by geographical region). The column uses a red-green gradient for a quick at-a-glance view of which regions have more or less capacity than others:

image
image1514×1672 394 KB

Hope you find this feature useful! Let us know if you have any feedback.

2

This is interesting! I found the screenshot table intriguing on its own, actually. Some of these you guys had mentioned previously, like how small Denver is, but others I wouldn’t have guessed. E.g., the huge spare capacity in Los Angeles (particularly in comparison to San Francisco).

A few small enhancements that come to mind, that might be nice in the future, :thought_balloon:…

  • Column (or at least JSON field) for the cost multiplier, which is currently only available by parsing the pricing page’s HTML.
  • All-pairs round-trip times (RTT) might get a bit bulky for most clients, but maybe opt-in via an ?rtt=all-pairs.
  • An indicator, even a simple one, of the recent extent of host failures and failed synthetic alerts—conditions that fall short of a full notification in the existing status system, but might inform people’s decisions when they’re on the fence about latency, cost, etc.

(These are classic feeds for optimization/placement algorithms, in other industries.)

4

Hi,

The API endpoint is currently reporting capacities of 1 for all regions. Is this endpoint now deprecated? Capacity is not listed in the OpenAPI spec either… so guessing it has been removed?

Byron

5 
[PM01] machines API returned an error: "machine can't start at the moment"

whatever was changed, my machines can’t start anymore.

6

@czpl I got the same issue in Stockholm Sweden, I redeployed to a different region which worked for me at least!

7

That probably wasn’t the Capacity API: that one has unfortunately been returning eyebrow-raising results for a long time now, :snowflake:, since at least December.

If your Machines are in Sweden then it was likely the brief (2 hour) incident that @jonast mentioned:

Capacity issues in ARN region

(As a small tip for future reference… it’s best to report such problems in a new top-level thread, under the Questions / Help category. That way, it gets inducted directly into Fly Support’s ticketing system.)

A patch to the official flyctl repository in February described the reported capacities as “currently inaccurate”, so I would indeed consider it an unreliable placeholder / dummy value, until such time as there’s an official announcement to the contrary, at least.

(It’s too bad, since that was a really useful feature while it lasted…)