My production site broke due to a lease held by `*@tokens.fly.io`

[porton]

My productions site broke, see logs:

• > cd backend && npx prisma migrate deploy

• 15:19:42Starting machine

• 15:19:43[PR03] could not find a good candidate within 1 attempts at load balancing. last error: [PM01] machines API returned an error: "machine ID e2863292b50518 lease currently held by 80fb61df-bd22-5d5d-af4c-2eac47b8e520@tokens.fly.io, expires at 2026-02-22T15:19:53Z

I have two machines deployed and previously they worked.

See also meritocracy/deploy/fly.production.toml at main · vporton/meritocracy · GitHub

[jfent]

That looks like you may have a deploy token or an org token or something like that which is holding the lease, e.g. see what happens below when I try to take a lease on the same machine a second time:

$ fly apps create test-app1                       )
? Select Organization: Jacob Fenton (personal)
New app created: test-app1

$ export FLY_API_TOKEN=$(fly tokens create deploy -a test-app1)

$ fly m run -a test-app1 ubuntu sh -c "sleep inf"
Searching for image 'ubuntu' remotely...
image found: img_98dgp88y6q5npxw0
Image: docker-hub-mirror.fly.io/library/ubuntu:latest@sha256:98ff7968124952e719a8a69bb3cccdd217f5fe758108ac4f21ad22e1df44d237
Image size: 30 MB

Success! A Machine has been successfully launched in app test-app1
 Machine ID: 890134b6d54118
 Instance ID: 01KJ5PF2KQAHSVH3V4Y4HF45ZF
 State: created

 Attempting to start machine...

==> Monitoring health checks
No health checks found

Machine started, you can connect via the following private ip
  fdaa:3:7c6d:a7b:2c2:17f7:f7a4:2


$ curl -X POST 'https://api.machines.dev/v1/apps/test-app1/machines/890134b6d54118/lease' -H 'content-type: application/json' -d '{"ttl": 3600}' -H "Authorization: Bearer $FLY_API_TOKEN"
{"status":"success","data":{"nonce":"c3618ea814ef","expires_at":1771869119,"owner":"8f81d697-fc78-58a5-beb3-0518034d1079@tokens.fly.io","description":"","version":"01KJ5PF2KQAHSVH3V4Y4HF45ZF"}}

$ curl -X POST 'https://api.machines.dev/v1/apps/test-app1/machines/890134b6d54118/lease' -H 'content-type: application/json' -d '{"ttl": 3600}' -H "Authorization: Bearer $FLY_API_TOKEN"
{"status":"error","message":"machine ID 890134b6d54118 lease currently held by 8f81d697-fc78-58a5-beb3-0518034d1079@tokens.fly.io, expires at 2026-02-23T17:51:59Z"}

The @tokens.fly.io email is just a random email generated as the identity for the token, it’s not a Fly.io system email or anything like that.

Maybe a previous deploy of yours was still ongoing, meaning a lease was still held by that deploy?