Deploy stuck on "Acquiring lease"

Questions / Help
,
1

I have two Node.js apps that I am deploying, the only changes being updates to the fly.toml for each of them. Both of them are stuck at “Acquiring lease”. I tried to clear the any leases using fly machines leases clear <machine ID> but that didn’t help.

2

Did you happen to update any health checks in the toml? If not and there’s no deployment issues on your end, it’s probably a Fly infra issue. You’ll have to wait until they resolve it or it resolves itself.

3

Nope, all I updated was some env values. I’ll try it again in the morning to see if it’s resolved. Thanks.

4

Neither is working for me. I also tried to destroy the machine, but no luck :frowning:

5

I have the same problem. Nothing helps.

6

For mi it helped to destroy the entire app and then recreate it. I dit this for one of two apps. The other one has the same problem but I’ll wait to see if it gets resolved by fly.

7

Fly team is currently investigating this issue https://status.flyio.net/

8

Hi! Andrés from infra team here.

We are still investigating this issue, testing a possible fix, but we are having troubles to replicate.
If you are still having issues to acquire a lease, would you mind to run LOG_LEVEL=debug flyctl <the_command_you_are_running>?
It would be very helpful for us to narrow down the problem.

9

Hi @aschiavo ,
just doint something like ‘fly secrets set FO=BAR’ is hanging (and currently the machine is unhealty)

Here is the last part of the output with the last command with debug LOG_LEVEL:

DEBUG {0x4000909740}
DEBUG <-- 200 https://api.fly.io/graphql (451.3ms)

DEBUG {
  "data": {
    "createRelease": {
      "release": {
        "id": "MZzaeRX0025zqIZyGwN9yYBo6",
        "version": 28
      }
    }
  }
}

DEBUG --> POST https://api.fly.io/graphql

DEBUG {
  "query": "\nmutation UpdateRelease ($input: UpdateReleaseInput!) {\n\tupdateRelease(input: $input) {\n\t\trelease {\n\t\t\tid\n\t\t}\n\t}\n}\n",
  "variables": {
    "input": {
      "clientMutationId": "",
      "metadata": null,
      "releaseId": "MZzaeRX0025zqIZyGwN9yYBo6",
      "status": "running"
    }
  },
  "operationName": "UpdateRelease"
}

DEBUG {0x4000ef83f0}
DEBUG <-- 200 https://api.fly.io/graphql (251.05ms)

DEBUG {
  "data": {
    "updateRelease": {
      "release": {
        "id": "MZzaeRX0025zqIZyGwN9yYBo6"
      }
    }
  }
}

DEBUG --> POST https://api.machines.dev/v1/apps/notisfier-co/machines/328715d7f55d58/lease?ttl=13

DEBUG <-- 504 https://api.machines.dev/v1/apps/notisfier-co/machines/328715d7f55d58/lease?ttl=13 (1m1.01s)

DEBUG upstream request timeout
DEBUG --> POST https://api.machines.dev/v1/apps/notisfier-co/machines/328715d7f55d58/lease?ttl=13
10

(The script then tries to do the same POST again and again, timing out each time)

11

I ran LOG_LEVEL=debug flyctl deploy and here’s what I’m seeing when it gets stuck at Acquiring lease:

DEBUG --> POST https://api.machines.dev/v1/apps/<app>/machines/<machine>/lease?ttl=13000000000
-------
DEBUG <-- 504 https://api.machines.dev/v1/apps/<app>/machines/<machine>f/lease?ttl=13000000000 (1m1.02s)
-------
DEBUG upstream request timeout<machine_id_postfix>
DEBUG --> POST https://api.machines.dev/v1/apps/<app>/machines/<machine>/lease?ttl=13000000000
-------
DEBUG <-- 504 https://api.machines.dev/v1/apps/<app>/machines/<machine>/lease?ttl=13000000000 (1m1.02s)
-------
DEBUG upstream request timeout<machine_id_postfix>
DEBUG --> POST https://api.machines.dev/v1/apps/<app>/machines/<machine>/lease?ttl=13000000000
-------
DEBUG <-- 504 https://api.machines.dev/v1/apps/<app>/machines/<machine>/lease?ttl=13000000000 (1m1.02s)
-------
DEBUG upstream request timeout<machine_id_postfix>
DEBUG --> POST https://api.machines.dev/v1/apps/<app>/machines/<machine>/lease?ttl=13000000000
-------
DEBUG <-- 504 https://api.machines.dev/v1/apps/<app>/machines/<machine>/lease?ttl=13000000000 (1m1.02s)
-------
DEBUG upstream request timeout<machine_id_postfix>
DEBUG --> POST https://api.machines.dev/v1/apps/<app>/machines/<machine>/lease?ttl=13000000000
-------
DEBUG <-- 504 https://api.machines.dev/v1/apps/<app>/machines/<machine>/lease?ttl=13000000000 (1m1.03s)
-------
DEBUG upstream request timeout<machine_id_postfix>
DEBUG --> POST https://api.machines.dev/v1/apps/<app>/machines/<machine>/lease?ttl=13000000000
-------

-------
 ⠏ Acquiring lease for <machine>
12

Thanks @callmewind ! We will use this info to troubleshoot :bowing_man:

13

@aschiavo Yes, this is what I’m seeing too

14

Thank you too @uyhoang !!

15

Thank you all for the information provided, it helped us to implement a fix and deploy it to our fleet.

Please give it a try and let us know!

For those apps that have been retrying the flyctl commands, like CI/CD jobs, it may take a little longer to get the leases in a good spot, as there are a few releases still running.

1 Like
16

My apps have successfully deployed. Thanks for your help!

1 Like
17

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.