fly secrets deploy sometimes fails or doesn’t update after scaling machines

1

Hi Fly.io team,

I’m reporting an issue with fly secrets deploy that has been annoying us for a while.
Updating secrets behaves inconsistently:

  • Sometimes it fails with “no machines available to deploy,” even though machines exist and are started.
  • Other times it reports success, but the environment variables on the machines are not updated, requiring manual machine update or even destroying and launching new machines,

This is happening on multiple apps and has been causing repeated headaches across projects and use cases.

Scenario 1: fly secrets deploy refusing to update machines

# fly machines list
ID              NAME                STATE   CHECKS  REGION  ROLE    IMAGE                 IP ADDRESS
0807249a257d48  snowy-wind-3100     started 1/1     cdg     app     my-app:deployment-01KJWQKFKNZ  fdaa:35:74a0:a7b:5b2:c3b9:6f28:2

# fly secrets deploy

Error: no machines available to deploy
'fly secrets deploy' will only work if the app has been deployed and there are machines available
Try 'fly deploy' first

# fly scale count 0 --yes && fly scale count 1 --yes --region cdg
Destroyed 0807249a257d48
Created 80e479a6530e08

# fly secrets deploy
Error: no machines available to deploy

# eventually after a few scaling it works

Scenario 2: Secrets deploy sometimes succeeds but environment isn’t updated

  1. Secrets staged:
# fly secrets list
NAME                    DIGEST                  STATUS
DATABASE_URL            xxx                      Staged
AUTH_GOOGLE_ID          xxx                      Staged
AUTH_GOOGLE_SECRET      xxx                      Staged
...
  1. Deploy secrets:
1password-secrets fly import my-app-name
Deploying secrets to Fly app 'my-app-name'...
Updating existing machines in 'my-app-name' with immediate strategy
✔ Machine 0802d16b29e5d8 [app] update succeeded
✓ DNS configuration verified
Secrets deployed to Fly app 'my-app-name'
  1. SSH into machine and check environment:
$ fly ssh console
export | grep DATABASE_URL
DATABASE_URL='OLD_VALUE_REDACTED'

In some deployments, despite fly secrets deploy reporting success, the environment variables do not reflect the updated secrets until the machine is manually updated or the app is redeployed.

Observations

  • fly secrets deploy sometimes fails we haven’t been able to pinpoint when exactly.

  • Sometimes the deploy shows as successful but the secrets are not applied to the runtime environment. This is the most frustrating part. Usually manually performing fly secrets update , unset and set resolved the issue. Unfortunately this is usually not our workflow as it requires passing secret by secret in the terminal. We normally stage import in bulk.

  • Scaling down/up or doing a full deploy after the secrets have been marked as updated usually resolves the issue. Important: secrets sometimes are marked as deployed (they are no longer staged), and still they are not actually set.

  • We have been observing this with all the apps and can prove this with single machine apps. No fancy setups with processes/different images/etc.

Can you please fix this? Can you even reproduce on your side? Feel free to hop on a call with us.

[If you don’t already know our main org is #35721]

NOTE: For privacy reasons the commands and outputs have been redacted by AI

1 Like
2

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.