Error: machines could not be retrieved

I just upgraded flyctl to v0.1.84 and I am now unable to list machines for an app. This error happens for all my apps. Any idea?

❯ fly apps list         
NAME                                    OWNER                   STATUS          PLATFORM        LATEST DEPLOY        
abcd                               personal                deployed        machines        2023-08-22T20:43:13Z
❯ fly machine list -a abcd
Error: machines could not be retrieved

With debug flag

❯ fly machine list -a abcd --debug
Error: machines could not be retrieved
Stacktrace:
goroutine 1 [running]:
runtime/debug.Stack()
        /opt/hostedtoolcache/go/1.20.6/x64/src/runtime/debug/stack.go:24 +0x65
github.com/superfly/flyctl/internal/cli.printError(0xc000528000, 0xc000cd85b0?, 0x1b5b480?, {0x236df20, 0xc000cd85b0})
        /home/runner/work/flyctl/flyctl/internal/cli/cli.go:126 +0x365
github.com/superfly/flyctl/internal/cli.Run({0x2389f18?, 0xc00069b880?}, 0xc000528000, {0xc0000501f0?, 0x5, 0x5})
        /home/runner/work/flyctl/flyctl/internal/cli/cli.go:84 +0x6a5
main.run()
        /home/runner/work/flyctl/flyctl/main.go:47 +0x18b
main.main()
        /home/runner/work/flyctl/flyctl/main.go:26 +0x1e

Version

flyctl version      
flyctl v0.1.84 linux/amd64 Commit: 5735a8b979ebc938c74832c3edd52c509abaa7eb BuildDate: 2023-08-23T16:56:07Z

I will try to reproduce it in case I find anything else until somebody from the Fly team gets to this.

1 Like

It worked okay for me. I wonder if you can see machines for that app in the dashboard?

This is so weird. It did not work yesterday, it did not work this morning. It is working now however.

I have no idea what happened. Thanks for investigating and sorry that I used your time.

:pray:

1 Like

No wait I just found it!

❯ fly machines list -a abcd
Error: machines could not be retrieved

❯ fly machine list -a abcd
1 machines have been retrieved from app abcd.
View them in the UI here
....

Can you spot the difference, @redjonzaci ?

According to fly --help there is a fly machine … command, but writing fly machines list --help does also seem to work.

I see they are aliases and fly machine list -a worked okay for me.

image

I can try with fly machines list -a.

Well I just ran the above commands after each other and one worked and the other didn’t.

Wait, this is getting even more strange. It seems to just be very unstable? I run these commands after each other

❯ flyctl machines list --app klimsek --quiet       
Error: machines could not be retrieved
klimsek.se [ update-pocketbase-js-sdk][✘!?] 
❯ flyctl machines list --app klimsek --quiet
Error: machines could not be retrieved
klimsek.se [ update-pocketbase-js-sdk][✘!?] 
❯ flyctl machines list --app klimsek --quiet
Error: machines could not be retrieved
klimsek.se [ update-pocketbase-js-sdk][✘!?] 
❯ flyctl machines list --app klimsek --quiet
Error: machines could not be retrieved
klimsek.se [ update-pocketbase-js-sdk][✘!?] 
❯ flyctl machine list --app klimsek --quiet 
Error: machines could not be retrieved
klimsek.se [ update-pocketbase-js-sdk][✘!?] 
❯ flyctl machine list --app klimsek --quiet
Error: machines could not be retrieved
klimsek.se [ update-pocketbase-js-sdk][✘!?] 
❯ flyctl machines list --app klimsek --quiet
32874472ce0268

klimsek.se [ update-pocketbase-js-sdk][✘!?] 
❯ flyctl machines list --app klimsek --quiet
32874472ce0268

klimsek.se [ update-pocketbase-js-sdk][✘!?] 
❯ flyctl machines list --app klimsek --quiet
Error: machines could not be retrieved

❯ flyctl machine list --app klimsek --quiet
32874472ce0268

klimsek.se [ update-pocketbase-js-sdk][✘!?] 
❯ flyctl machine list --app klimsek --quiet
32874472ce0268

klimsek.se [ update-pocketbase-js-sdk][✘!?] 
❯ flyctl machine list --app klimsek --quiet
32874472ce0268

klimsek.se [ update-pocketbase-js-sdk][✘!?] 
❯ flyctl machine list --app klimsek --quiet
Error: machines could not be retrieved

At this point, somebody from the Fly team would be more helpful.
Hope they get to help you soon!

Hi @eipe. Since this happens to multiple apps my hunch is that you might be having some trouble connecting to our API or the gateway you’re closest to might be failing.

Can you try running LOG_LEVEL=debug flyctl machine list --app klimsek --quiet to see if there’s a non-200 response?

Here’s an example of mine:

[I] ➜ LOG_LEVEL=debug flyctl machine list --app  APP_NAME

DEBUG --> GET https://api.machines.dev/v1/apps/APP_NAME/machines

DEBUG started querying for new release
DEBUG <-- 200 https://api.machines.dev/v1/apps/APP_NAME/machines (2.08s)

DEBUG []

No machines are available on this app APP_NAME
DEBUG Task manager done

Thanks @lubien for taking the time.

Now I run exactly the same command over and over, and get no error message. So for the time being I seem to be OK but it feels shaky.

PS. I am using the wifi of The Swedish Internet Foundation (top level domain provider) so I hope any connectivity problem is not in this building :laughing:

❯ LOG_LEVEL=debug flyctl machines list --app klimsek-update-pocketbase-js-sdk-112 --quiet
DEBUG Loaded flyctl config from/home/einar/.fly/config.yml
DEBUG determined hostname: "x13"
DEBUG determined working directory: "/home/einar/Projects/****REDACTED***"
DEBUG determined user home directory: "/home/einar"
DEBUG determined config directory: "/home/einar/.fly"
DEBUG ensured config directory exists.
DEBUG ensured config directory perms.
DEBUG cache loaded.
DEBUG config initialized.
DEBUG skipped querying for new release
DEBUG client initialized.
DEBUG no app config found at /home/einar/Projects/****REDACTED***fly.toml; skipped.
DEBUG --> GET https://api.machines.dev/v1/apps/klimsek-update-pocketbase-js-sdk-112/machines

DEBUG Starting task manager
DEBUG Config has metrics token

DEBUG <-- 200 https://api.machines.dev/v1/apps/klimsek-update-pocketbase-js-sdk-112/machines (64.86ms)

DEBUG [
  {
    "id": "5683d52ea17228",
    "name": "little-sound-3091",
    "state": "started",
    "region": "fra",
    "instance_id": "01H90JK5QVQMY3TN8D8HK6QZQ6",
    "private_ip": "fdaa:1:358e:a7b:1c2:f8c8:5035:2",
    "config": {
      "env": {
        "FLY_PROCESS_GROUP": "app",
        "PRIMARY_REGION": "fra"
      },
      "init": {},
      "metadata": {
        "fly_flyctl_version": "0.1.84",
        "fly_platform_version": "v2",
        "fly_process_group": "app",
        "fly_release_id": "P5jgApLBB04j3Tyn6kBbe9M1X",
        "fly_release_version": "2"
      },
      "services": [
        {
          "protocol": "tcp",
          "internal_port": 8080,
          "ports": [
            {
              "port": 80,
              "handlers": [
                "http"
              ],
              "force_https": true
            },
            {
              "port": 443,
              "handlers": [
                "tls",
                "http"
              ]
            }
          ],
          "checks": [
            {
              "type": "tcp",
              "interval": "15s",
              "timeout": "2s",
              "grace_period": "1s"
            }
          ],
          "concurrency": {
            "type": "connections",
            "hard_limit": 20,
            "soft_limit": 15
          },
          "force_instance_key": null
        }
      ],
      "image": "registry.fly.io/klimsek-update-pocketbase-js-sdk-112:deployment-01H90JJJN5FHWQA1PFGBD6TVD6",
      "restart": {},
      "guest": {
        "cpu_kind": "shared",
        "cpus": 1,
        "memory_mb": 256
      },
      "stop_config": {
        "signal": "SIGINT"
      }
    },
    "image_ref": {
      "registry": "registry.fly.io",
      "repository": "klimsek-update-pocketbase-js-sdk-112",
      "tag": "deployment-01H90JJJN5FHWQA1PFGBD6TVD6",
      "digest": "sha256:08156019f9dfd65a6c95b330f481fde5fd75a40af676483cd8abfdee3a481c08",
      "labels": null
    },
    "created_at": "2023-08-28T14:32:42Z",
    "updated_at": "2023-08-29T11:59:29Z",
    "events": [
      {
        "id": "01H90JKFTN17BXTAVRC2SFKJDB",
        "type": "start",
        "status": "started",
        "source": "flyd",
        "timestamp": 1693310369621
      },
      {
        "id": "01H90JK5QZ2HH6NDRM4VP072T5",
        "type": "launch",
        "status": "created",
        "source": "user",
        "timestamp": 1693310359295
      }
    ],
    "checks": [
      {
        "name": "servicecheck-00-tcp-8080",
        "status": "passing",
        "output": "Success",
        "updated_at": "2023-08-28T14:32:58.931Z"
      }
    ]
  }
]


5683d52ea17228

DEBUG Task manager done

Now it stopped working!

klimsek.se [ update-pocketbase-js-sdk]
❯ LOG_LEVEL=debug flyctl machines list --app klimsek-update-pocketbase-js-sdk-112 --quiet
DEBUG Loaded flyctl config from/home/einar/.fly/config.yml
DEBUG determined hostname: "x13"
DEBUG determined working directory: "/home/einar/Projects/***REDACTED***"
DEBUG determined user home directory: "/home/einar"
DEBUG determined config directory: "/home/einar/.fly"
DEBUG ensured config directory exists.
DEBUG ensured config directory perms.
DEBUG cache loaded.
DEBUG config initialized.
DEBUG skipped querying for new release
DEBUG client initialized.
DEBUG no app config found at /home/einar/Projects/***REDACTED***/fly.toml; skipped.
DEBUG --> GET https://api.machines.dev/v1/apps/klimsek-update-pocketbase-js-sdk-112/machines

DEBUG Starting task manager
DEBUG Config has metrics token

DEBUG Task manager done
Error: machines could not be retrieved

Oh no, our CLI doesn’t show the error :sweat:

Would you be able to run this instead?

curl --location --request GET 'https://api.machines.dev/v1/apps/klimsek-update-pocketbase-js-sdk-112/machines' \
--header 'Authorization: Bearer [YOUR FLY TOKEN HERE]' \
--header 'Content-Type: application/json' 

You can get your token using fly auth token

I will try this if / next time I encounter this error.

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.