Unable to deploy or scale machines suddenly

Django
,
1

Hi, was just trying to deploy an update for my app.

Ran the usual flyctl deploy command and ran into this new error.

Running <app> release_command: ./manage.py collectstatic --noinput

-------
 ✖ Failed: error running release_command machine: error creating a release_command machine: fail…
-------
Error: release command failed - aborting deployment. error running release_command machine: error creating a release_command machine: failed to launch VM: permission_denied: bolt token: failed to verify service token: no verified tokens; token <token>: verify: context deadline exceeded
request token: failed to verify service token: no verified tokens; token <token>: verify: context deadline exceeded (Request ID: 01JHJ2869G3H7006R089HYSTXK-lhr) (Trace ID: e9953bc829dcacc1f0320b7f5100a4a3)

Puzzled, I have tried the following:

fly doctor // all passed
rm -r ~/.fly
fly agent stop
fly auth logout
brew uninstall flyctl
brew update; brew install flyctl
fly auth login
fly deploy

And the issue persists, I tried scaling down the number of instances with

fly scale count 0

and that worked, but am now unable to scale back up

fly scale count 2
...
Executing scale plan
Error: failed to launch VM: permission_denied: bolt token: failed to verify service token: no verified tokens; token <token>: verify: context deadline exceeded
request token: failed to verify service token: no verified tokens; token <token>: verify: context deadline exceeded
failed to launch VM: permission_denied: bolt token: failed to verify service token: no verified tokens; token <token>: verify: context deadline exceeded
request token: failed to verify service token: no verified tokens; token <token>: verify: context deadline exceeded (Request ID: 01JHJ2HCY8NV62DGNV4EFZS4KB-lhr)

Really confused and have no idea what to do, thought it might be to do with the auto_stop_machines='suspend' option in my fly.toml but that wasn’t the issue.

Here’s my fly.toml

app = <app>
primary_region = 'lhr'
console_command = '/code/manage.py shell'

[build]

[deploy]
  release_command = './manage.py collectstatic --noinput'

[env]
  PORT = '8000'

[http_service]
  internal_port = 8000
  force_https = true
  auto_stop_machines = 'stop' // was 'suspend'
  auto_start_machines = true
  min_machines_running = 0
  processes = ['app']

[[vm]]
  memory = '2gb'
  cpu_kind = 'shared'
  cpus = 2

[[statics]]
  guest_path = '/code/static'
  url_prefix = '/static/'
2 Likes
2

I am encountering the same error with my Django application. Deploys were working yesterday - Today getting the following on every deploy

:heavy_multiplication_x: Failed: error running release_command machine: error creating a release_command machine: fail…

Error: release command failed - aborting deployment. error running release_command machine: error creating a release_command machine: failed to launch VM: permission_denied: bolt token: failed to verify service token: no verified tokens; token X: verify: context deadline exceeded
request token: failed to verify service token: no verified tokens; token X: verify: context deadline

1 Like
3

Seems to be an issue on fly’s end. I can’t think of anything I’ve changed/possibly done to cause this to happen. Double checked my billing, everything.

Is there a way we can directly get this issue to them?

4

Same issue. Exhausting.

5

Same here, on both my dev machine and GitHub actions, when running fly deploy after the image build step:

Error: release command failed - aborting deployment. error running release_command machine: error creating a release_command machine: failed to launch VM: permission_denied: request token: failed to verify serv
ice token: no verified tokens; token 8cd7adaf-64df-51b3-a242-6805355a713b: verify: context deadline exceeded (Request ID: 01JHJ81FZG26066MW3ZVARB422-lhr) (Trace ID: fe125a46cd9a7c144d5e201524035d6e)

and seeing a similar error when attempting fly scale count with a scale up:

Error: failed to launch VM: permission_denied: request token: failed to verify service token: no verified tokens; token 215c48af-fc5f-54e7-a5ae-b2777ac49b62: verify: context deadline exceeded (Request ID: 01JHJ8XMQGQHWBKAF55H43GX04-lhr)

I’ve tried fly auth logout and fly auth login, fly doctor, and updating my GitHub actions fly token.

6

Same issue here, definitely looks like an issue on fly.io’s side.

7

:wave: Sorry for the issues. This was the result of networking issues on a few of our hosts in the lhr region. Those problems should be resolved now, but let us know if you continue to see those errors.

3 Likes
9

I’m currently getting hit by this issue.

1 Like
10

This is still not resolved unfortunately. Tried deploying twice in the last hour - both time the same error failed to launch VM: permission_denied: bolt token: failed to verify service token: no verified tokens....

11

I am still encountering this issue, and am getting exactly the same error when deploying my ‘find-a-book’ app

12

Also given that this is affecting multiple users, it might be good to add something to https://status.flyio.net/ to highlight that this is happening/has been happening.

2 Likes
13

Sorry folks. The networking issue cropped up again. This time we were able to diagnose and fix the root cause though. You’re absolutely right that this should have made it to the status page this most recent time as well as the last time.