Error updating Machine via API

Hi all!

I’m working on updating our integration with Fly.io to support Apps V2 machines.

Here’s what we’re doing. We update secrets via a setSecrets() GraphQL mutation. After that request, we fetch all machines for the application and update each machine with a config that has the updated environment variable secrets (as per the docs - “Existing Machines must be updated to pick up secrets set after the Machine was created.”).

The machine fetch works fine. However, the machine update is failing with a 422 with this error:

Authentication required to access image "docker.io/library/flyio-rails:deployment-01H12N2YXHWMNZM7HN1KBGP928"

The application I’m testing with is a generated Rails application created using the getting started guide. In short:

rails new flyio-rails
# update the Rails routes
fly launch
fly deploy

The machine config we’re updating with is literally exactly what the API provided when fetching the machine. The only change is that we update the config.env with appropriate values. Here’s an example of the payload we’re sending in our test:

{
  "config": {
    "env": {
      "TEST": "asdfasdfasdfasdfasdfasdf"
    },
    "init": {
      "cmd": null,
      "tty": null,
      "exec": null,
      "entrypoint": null
    },
    "size": "shared-cpu-1x",
    "image": "flyio-rails:deployment-01H12N2YXHWMNZM7HN1KBGP928",
    "image_ref": {
      "registry": "registry.fly.io",
      "repository": "flyio-rails",
      "tag": "deployment-01H12N2YXHWMNZM7HN1KBGP928",
      "digest": "sha256:999f7e73047f5c90b468f58336d93d0aaf055ec5e1e68c4c663cf83fc336e3e8",
      "labels": {
        "fly_launch_runtime": "rails"
      }
    },
    "restart": {},
    "metadata": {
      "fly_release_id": "OPOl42Nkk3GOmHkKJgmNXZg0",
      "fly_process_group": "app",
      "fly_release_version": "4",
      "fly_platform_version": "v2"
    },
    "mounts": []
  }
}

My guess is that maybe it has something to do with the image being set to use a specific tag that appears to be tied to a now non-existent deploy rather than latest? Not really sure though. Any help would be appreciated!

Regards,
-Joel

Hi @watsonian! While image_ref is a top-level field returned by the API in Machine objects, AFAIK it isn’t part of a Machine’s config. The image field of config is what’s used to set the image. Since yours does not specify a registry, I think it’s defaulting to docker.io.

If you instead send

"image": "registry.fly.io/flyio-rails:deployment-01H12N2YXHWMNZM7HN1KBGP928"

does it work as expected?

I found the problem. It looks like your GraphQL endpoint returns a different schema payload than your Rest API endpoint. Here’s a request I executed in the GraphQL playground:

query($appId: String!, $batchSize: Int!) {
  machines(appId: $appId, first: $batchSize) {
    nodes {
      id
      name
      config
    }
  }
}

That returned the following response:

{
  "data": {
    "machines": {
      "nodes": [
        {
          "id": "328711ea9e3d85",
          "name": "withered-resonance-5415",
          "config": {
            "env": {
              "PRIMARY_REGION": "dfw",
              "FLY_PROCESS_GROUP": "app"
            },
            "init": {
              "cmd": null,
              "tty": null,
              "exec": null,
              "entrypoint": null
            },
            "size": "shared-cpu-1x",
            "image": "flyio-rails:deployment-01H12N2YXHWMNZM7HN1KBGP928",
            "image_ref": {
              "registry": "registry.fly.io",
              "repository": "flyio-rails",
              "tag": "deployment-01H12N2YXHWMNZM7HN1KBGP928",
              "digest": "sha256:999f7e73047f5c90b468f58336d93d0aaf055ec5e1e68c4c663cf83fc336e3e8",
              "labels": {
                "fly_launch_runtime": "rails"
              }
            },
            "restart": {},
            "metadata": {
              "fly_release_id": "OPOl42Nkk3GOmHkKJgmNXZg0",
              "fly_process_group": "app",
              "fly_release_version": "4",
              "fly_platform_version": "v2"
            },
            "mounts": []
          }
        },
        ...
      ]
    }
  }
}

Notice how image_ref is inside the config block. Now the same call hitting the Rest API:

curl -s -H "Authorization: Bearer $FLYIO_API_TOKEN" "https://api.machines.dev/v1/apps/flyio-rails/machines" | jq
[
  {
    "id": "328711ea9e3d85",
    "name": "withered-resonance-5415",
    "state": "stopped",
    "region": "dfw",
    "instance_id": "01H12N3H6MNTBRXVHRYAVD0EXB",
    "private_ip": "fdaa:0:442f:a7b:2203:9d21:b6c0:2",
    "config": {
      "env": {
        "FLY_PROCESS_GROUP": "app",
        "PRIMARY_REGION": "dfw"
      },
      "init": {},
      "metadata": {
        "fly_platform_version": "v2",
        "fly_process_group": "app",
        "fly_release_id": "OPOl42Nkk3GOmHkKJgmNXZg0",
        "fly_release_version": "4"
      },
      "services": [
        {
          "protocol": "tcp",
          "internal_port": 3000,
          "autostop": true,
          "autostart": true,
          "min_machines_running": 0,
          "ports": [
            {
              "port": 80,
              "handlers": [
                "http"
              ],
              "force_https": true
            },
            {
              "port": 443,
              "handlers": [
                "http",
                "tls"
              ]
            }
          ],
          "force_instance_key": null
        }
      ],
      "statics": [
        {
          "guest_path": "/rails/public",
          "url_prefix": "/"
        }
      ],
      "image": "registry.fly.io/flyio-rails:deployment-01H12N2YXHWMNZM7HN1KBGP928",
      "restart": {},
      "guest": {
        "cpu_kind": "shared",
        "cpus": 1,
        "memory_mb": 256
      }
    },
    "image_ref": {
      "registry": "registry.fly.io",
      "repository": "flyio-rails",
      "tag": "deployment-01H12N2YXHWMNZM7HN1KBGP928",
      "digest": "sha256:999f7e73047f5c90b468f58336d93d0aaf055ec5e1e68c4c663cf83fc336e3e8",
      "labels": {
        "fly_launch_runtime": "rails"
      }
    },
    "created_at": "2023-05-17T20:42:27Z",
    "updated_at": "2023-05-23T14:42:43Z",
    "events": [
      {
        "id": "01H14GZXNWRW0YB2A79MWWZS4S",
        "type": "exit",
        "status": "stopped",
        "request": {
          "exit_event": {
            "requested_stop": true,
            "restarting": false,
            "guest_exit_code": 0,
            "guest_signal": -1,
            "guest_error": "",
            "exit_code": 0,
            "signal": -1,
            "error": "",
            "oom_killed": false,
            "exited_at": "2023-05-23T14:42:42.843Z"
          },
          "restart_count": 0
        },
        "source": "flyd",
        "timestamp": 1684852963004
      },
      {
        "id": "01H14GZVN49FXJA50F29BPQRR3",
        "type": "stop",
        "status": "stopping",
        "source": "user",
        "timestamp": 1684852960932
      },
      {
        "id": "01H14GM6037XAPWYP6457A23WY",
        "type": "start",
        "status": "started",
        "source": "flyd",
        "timestamp": 1684852578307
      },
      {
        "id": "01H14GM5SWFBMD2WT5BNWZ8GTX",
        "type": "start",
        "status": "starting",
        "source": "user",
        "timestamp": 1684852578108
      },
      {
        "id": "01H12NK89HN16FG8NMFXSV3EVF",
        "type": "exit",
        "status": "stopped",
        "request": {
          "exit_event": {
            "requested_stop": true,
            "restarting": false,
            "guest_exit_code": 0,
            "guest_signal": -1,
            "guest_error": "",
            "exit_code": 0,
            "signal": -1,
            "error": "",
            "oom_killed": false,
            "exited_at": "2023-05-22T21:24:41.719Z"
          },
          "restart_count": 0
        },
        "source": "flyd",
        "timestamp": 1684790681905
      },
      ...
    ]
  }
]

As you can see, there’s a lot more detail here compared to the GraphQL API. It seems like the GraphQL API’s schema has lagged behind the Rest API. Should we be swapping all our calls over to using the Rest API? When we initially created the integration on our end, I don’t think the Rest API existed, so we were just using GraphQL endpoints. However, between this and the fact that the GraphQL API doesn’t seem to have a mutation for updating machines…makes me feel like that API might be deprecated?

Regards,
-Joel

We’re still tied to the GraphQL API for a lot of things (such as authentication, creating apps, etc) but machines operations should basically all be done through the Rest API these days.

Got it. It might be nice to remove the machine portions of the GraphQL API if they’re not meant to be used as a hint that the Rest API should be solely used. Although, I imagine the fact that it exists at all there is probably due to y’all needing it somewhere internally.

Are you all planning to move everything over to Rest APIs eventually?

-Joel

As an aside, how does pagination work with your REST API? Is there some documentation for that somewhere? I didn’t see it mentioned here anywhere. Did I miss it?

One more follow-up. When performing a machine config update where we’re just changing the variables in env – this seems to work fine when you’re changing values (e.g., changing the value of TEST from 1234 to abcd). The machine reloads as expected and the new value is made available to the application. However, if we just remove variables and update the machine config, it doesn’t seem to trigger this behavior. The old variable value remains available. This continues to be the case even after running:

fly secrets set FOO=123 -a $APPNAME

which triggers a new release. I’ve also tried trigging a manual deploy without success using:

fly deploy

Is there some kind of caching going on here?

Edit: Figured this out. Had mistakenly been passing secrets into the machine config update when they were being deleted, so they were deleted in the app secrets, but present in the machine config. Incidentally, it would be pretty nice if it were possible to configure whether updating secrets would just automatically update existing machines. Having to do this manually always is kind of annoying (particularly considering the CLI does it automatically for you already).

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.