fly secrets list shows all secrets as Staged because releases endpoint is stale

Questions / Help
1

Hi Fly team/community,

I am seeing a persistent false Staged status from fly secrets list for one Machines app. This looks very similar to the recent report where secrets kept showing as Staged despite deploys, but I found a more specific release-index mismatch that seems to explain it.

Sanitized details:

  • Affected app: one development Machines app only.
  • Current machine release/version: v536.
  • fly releases -a <app> --json shows latest release v536 and the machine metadata also reports release v536.
  • fly secrets list -a <app> --json reports all 58 app secrets as Staged.
  • The app is healthy and runtime behavior confirms newly-set secrets are applied.
  • Production apps in the same account are unaffected; their secret statuses show all Deployed.

What I already tried on the affected development app:

  • fly secrets deploy
  • fly secrets sync
  • setting a non-sensitive config secret without --stage
  • same-image fly deploy
  • re-importing/rehydrating the existing dev secrets without printing values
  • upgrading flyctl from 0.4.51 to 0.4.57

All of those created healthy complete releases / machine updates, but fly secrets list still reports every secret as Staged.

The interesting part: from reading flyctl v0.4.57 source, the secrets list status appears to compute deployment status by comparing each secret updated_at against the current machine release created_at, where release timestamps come from the UIEX releases endpoint.

For this affected app, the UIEX endpoint used by flyctl appears stale:

  • fly releases -a <app> --json: latest release is v536
  • https://api.fly.io/api/v1/apps/<app>/releases?limit=200: latest/max release returned is v487, so it does not include the active machine release v536

If I recompute the status using the Machines secrets timestamps plus the release timestamp from fly releases --json, I get:

{
  "secrets_count": 58,
  "computed_counts": {"Deployed": 58},
  "machine_versions": ["536"],
  "graphql_latest_release_version": 536,
  "uiex_latest_release_version": 487,
  "uiex_has_machine_versions": false
}

So this looks like a false Staged status caused by a stale/incomplete releases endpoint for this app, not an actual secrets deployment failure.

Questions:

  1. Is the UIEX releases endpoint expected to be authoritative for secret status calculation?
  2. Can Fly refresh or repair the release index for this app so fly secrets list stops reporting false Staged?
  3. Is there any runtime risk if fly secrets list shows Staged but the active machine release is newer than all secret updated_at timestamps?
  4. Is there a safer workaround than recreating the development app or machine?

I intentionally omitted secret names, values, digests, account identifiers, and app names from this public post. Happy to provide exact app/machine IDs privately to Fly staff if needed.

Thanks!

2

Hi, can you share something that’d let me identify your app? (you have several and I don’t know which one this refers to). A machine ID should be OK to share - they’re not really sensitive. Otherwise, the last 6 characters of the app name should be OK. If you really want to be cloak-and-dagger, sha256sum the app’s name and share that; I can hash your app’s names on my side and find the one.

3

Thanks! To avoid posting direct app or machine identifiers publicly, here is the SHA-256 of the app name, computed with no trailing newline as printf '%s' '<app-name>' | sha256sum:

67bd923f97c92ed80d217792e4c601946b2c7316461b986726cb32075c62fbd9

I can provide the exact app name or machine ID privately if there is a email

4

Hi, have you tried the workaround suggested here? https://community.fly.io/t/flyctl-secrets-list-keeps-showing-all-secrets-as-staged-despite-deploys/27871/5?u=roadmr