Fly machines errors (insufficient memory available to fulfill request, could not decode machine ID)

1

Tried to launch a few machines via REST API, got these errors:

2022-08-16T16:37:28.778 app[6f9a1897] fra [info] FLY ERROR insufficient memory available to fulfill request
2022-08-16T16:37:28.779 app[6f9a1897] fra [info] FLY ERROR could not decode machine ID, 'extract_f24e1ffb-6cf3-45ff-a7e5-cd07da083d43'

Any idea why this could happen? The same requests worked fine a few hours ago.

Request:

http://_api.internal:4280/v1/apps/$IMAGE_NAME/machines

{
          name: name,
          region: 'fra',
          config: {
            image: `registry.fly.io/$IMAGE_NAME:latest`,
            services: [],
            guest: {
              cpus: 2,
              memory_mb: 4096,
              cpu_kind: 'shared',
            },
            restart: {
              policy: 'no',
            },
          },
        };
2

Hey, yea, we saw the issues and are working it now. Also, it looks like when you’re sending the DELETE request on failure it’s not sending a machine ID (because we didn’t create one due to the capacity issue) and you’re sending some internal ID extract_f24e1ffb-6cf3-45ff-a7e5-cd07da083d43 which the API is correctly returning HTTP 400?

3

It also looks like y’all are creating a lot of machines at once, which is fine, but can lead to these temporary capacity issues. Maybe deploy to more than 1 region?

1 Like
4

Good spot, thanks for pointing that out. We’ve had cases where the request failed, but a machine was still created. In that case we where just sending this request to make sure we’re not leaving unwanted remains. But obviously we can’t delete by the name, which is all we have.

@kurt already pointed out that creating many machines wouldn’t be great (resulting in API errors). We’ll change our system to try and maintain a pool of machines, but we’ll need to figure that out.

Thanks for the advice with multiple regions, we’ll check this

1 Like
5

Can you ping me if the machine API issues are solved @JP_Phillips ?

6

The fix is rolling out now for the HTTP 400 and I’m seeing machine creates go through for your app.

2 Likes
7

Hmm, still seeing this @JP_Phillips

8

Okay, looks better now