Unable to deploy or scale machines suddenly

jafjaf · January 14, 2025, 9:35am

Hi, was just trying to deploy an update for my app.

Ran the usual flyctl deploy command and ran into this new error.

Running <app> release_command: ./manage.py collectstatic --noinput

-------
 ✖ Failed: error running release_command machine: error creating a release_command machine: fail…
-------
Error: release command failed - aborting deployment. error running release_command machine: error creating a release_command machine: failed to launch VM: permission_denied: bolt token: failed to verify service token: no verified tokens; token <token>: verify: context deadline exceeded
request token: failed to verify service token: no verified tokens; token <token>: verify: context deadline exceeded (Request ID: 01JHJ2869G3H7006R089HYSTXK-lhr) (Trace ID: e9953bc829dcacc1f0320b7f5100a4a3)

Puzzled, I have tried the following:

fly doctor // all passed
rm -r ~/.fly
fly agent stop
fly auth logout
brew uninstall flyctl
brew update; brew install flyctl
fly auth login
fly deploy

And the issue persists, I tried scaling down the number of instances with

fly scale count 0

and that worked, but am now unable to scale back up

fly scale count 2
...
Executing scale plan
Error: failed to launch VM: permission_denied: bolt token: failed to verify service token: no verified tokens; token <token>: verify: context deadline exceeded
request token: failed to verify service token: no verified tokens; token <token>: verify: context deadline exceeded
failed to launch VM: permission_denied: bolt token: failed to verify service token: no verified tokens; token <token>: verify: context deadline exceeded
request token: failed to verify service token: no verified tokens; token <token>: verify: context deadline exceeded (Request ID: 01JHJ2HCY8NV62DGNV4EFZS4KB-lhr)

Really confused and have no idea what to do, thought it might be to do with the auto_stop_machines='suspend' option in my fly.toml but that wasn’t the issue.

Here’s my fly.toml

app = <app>
primary_region = 'lhr'
console_command = '/code/manage.py shell'

[build]

[deploy]
  release_command = './manage.py collectstatic --noinput'

[env]
  PORT = '8000'

[http_service]
  internal_port = 8000
  force_https = true
  auto_stop_machines = 'stop' // was 'suspend'
  auto_start_machines = true
  min_machines_running = 0
  processes = ['app']

[[vm]]
  memory = '2gb'
  cpu_kind = 'shared'
  cpus = 2

[[statics]]
  guest_path = '/code/static'
  url_prefix = '/static/'

mirandaKintek · January 14, 2025, 10:33am

I am encountering the same error with my Django application. Deploys were working yesterday - Today getting the following on every deploy

Failed: error running release_command machine: error creating a release_command machine: fail…

Error: release command failed - aborting deployment. error running release_command machine: error creating a release_command machine: failed to launch VM: permission_denied: bolt token: failed to verify service token: no verified tokens; token X: verify: context deadline exceeded
request token: failed to verify service token: no verified tokens; token X: verify: context deadline

jafjaf · January 14, 2025, 10:41am

Seems to be an issue on fly’s end. I can’t think of anything I’ve changed/possibly done to cause this to happen. Double checked my billing, everything.

Is there a way we can directly get this issue to them?

shortventures · January 14, 2025, 11:16am

Same issue. Exhausting.

belkorg · January 14, 2025, 11:29am

Same here, on both my dev machine and GitHub actions, when running fly deploy after the image build step:

Error: release command failed - aborting deployment. error running release_command machine: error creating a release_command machine: failed to launch VM: permission_denied: request token: failed to verify serv
ice token: no verified tokens; token 8cd7adaf-64df-51b3-a242-6805355a713b: verify: context deadline exceeded (Request ID: 01JHJ81FZG26066MW3ZVARB422-lhr) (Trace ID: fe125a46cd9a7c144d5e201524035d6e)

and seeing a similar error when attempting fly scale count with a scale up:

Error: failed to launch VM: permission_denied: request token: failed to verify service token: no verified tokens; token 215c48af-fc5f-54e7-a5ae-b2777ac49b62: verify: context deadline exceeded (Request ID: 01JHJ8XMQGQHWBKAF55H43GX04-lhr)

I’ve tried fly auth logout and fly auth login, fly doctor, and updating my GitHub actions fly token.

agrealish · January 14, 2025, 12:12pm

Same issue here, definitely looks like an issue on fly.io’s side.

btoews · January 14, 2025, 12:46pm

Sorry for the issues. This was the result of networking issues on a few of our hosts in the lhr region. Those problems should be resolved now, but let us know if you continue to see those errors.

kiancross · January 15, 2025, 6:13am

I’m currently getting hit by this issue.

antondyad · January 15, 2025, 7:05am

This is still not resolved unfortunately. Tried deploying twice in the last hour - both time the same error failed to launch VM: permission_denied: bolt token: failed to verify service token: no verified tokens....

joelchippindale · January 15, 2025, 8:01am

I am still encountering this issue, and am getting exactly the same error when deploying my ‘find-a-book’ app

joelchippindale · January 15, 2025, 8:04am

Also given that this is affecting multiple users, it might be good to add something to https://status.flyio.net/ to highlight that this is happening/has been happening.

btoews · January 15, 2025, 3:48pm

Sorry folks. The networking issue cropped up again. This time we were able to diagnose and fix the root cause though. You’re absolutely right that this should have made it to the status page this most recent time as well as the last time.

noy · January 14, 2026, 12:49pm

the same issue occurs just now.

eierbal050 · January 14, 2026, 12:59pm

Still cant deploy, even with `depot=false`

Frustrating that, again, this is not reflected on the status page.

lillian · January 14, 2026, 2:29pm

3 posts were split to a new topic: Deployment issues, failed to start VM: deadline_exceeded

dar · January 14, 2026, 3:49pm

Stuck with this and status page is all green

LAKE_Development · January 14, 2026, 4:33pm

Same issue, trying to deploy in ORD. I found a thread from 2023 having the same issue, and Fly said they added logging to prevent similar issues in the future.

pmrotule14 · January 14, 2026, 7:55pm

Why is this issue being completely ignored? All our deployments (YYZ) are failing at the moment, they always get stuck at “Waiting for depot builder…”.

Earlier today it was the token issue, but at least it was fixed pretty quickly and it was possible to track on the status page. Now this one started 3 hours ago and `fly deploy` still gets stuck, no update on the status page…

halfer · January 14, 2026, 8:05pm

Have you tried the usual fixes? I think fly deploy --depot=false is the one that most often gets a mention here.

If you don’t want to modify your CI for the time being, you can issue this command locally, of course.

TheClunkerJunker · January 14, 2026, 10:27pm

Depot is failing for me with “401 unauthorized” when trying to push to the registry. I used fly deploy --depot=false and it worked.