fly secrets list shows all secrets as Staged because releases endpoint is stale

Questions / Help
1

Hi Fly team/community,

I am seeing a persistent false Staged status from fly secrets list for one Machines app. This looks very similar to the recent report where secrets kept showing as Staged despite deploys, but I found a more specific release-index mismatch that seems to explain it.

Sanitized details:

  • Affected app: one development Machines app only.
  • Current machine release/version: v536.
  • fly releases -a <app> --json shows latest release v536 and the machine metadata also reports release v536.
  • fly secrets list -a <app> --json reports all 58 app secrets as Staged.
  • The app is healthy and runtime behavior confirms newly-set secrets are applied.
  • Production apps in the same account are unaffected; their secret statuses show all Deployed.

What I already tried on the affected development app:

  • fly secrets deploy
  • fly secrets sync
  • setting a non-sensitive config secret without --stage
  • same-image fly deploy
  • re-importing/rehydrating the existing dev secrets without printing values
  • upgrading flyctl from 0.4.51 to 0.4.57

All of those created healthy complete releases / machine updates, but fly secrets list still reports every secret as Staged.

The interesting part: from reading flyctl v0.4.57 source, the secrets list status appears to compute deployment status by comparing each secret updated_at against the current machine release created_at, where release timestamps come from the UIEX releases endpoint.

For this affected app, the UIEX endpoint used by flyctl appears stale:

  • fly releases -a <app> --json: latest release is v536
  • https://api.fly.io/api/v1/apps/<app>/releases?limit=200: latest/max release returned is v487, so it does not include the active machine release v536

If I recompute the status using the Machines secrets timestamps plus the release timestamp from fly releases --json, I get:

{
  "secrets_count": 58,
  "computed_counts": {"Deployed": 58},
  "machine_versions": ["536"],
  "graphql_latest_release_version": 536,
  "uiex_latest_release_version": 487,
  "uiex_has_machine_versions": false
}

So this looks like a false Staged status caused by a stale/incomplete releases endpoint for this app, not an actual secrets deployment failure.

Questions:

  1. Is the UIEX releases endpoint expected to be authoritative for secret status calculation?
  2. Can Fly refresh or repair the release index for this app so fly secrets list stops reporting false Staged?
  3. Is there any runtime risk if fly secrets list shows Staged but the active machine release is newer than all secret updated_at timestamps?
  4. Is there a safer workaround than recreating the development app or machine?

I intentionally omitted secret names, values, digests, account identifiers, and app names from this public post. Happy to provide exact app/machine IDs privately to Fly staff if needed.

Thanks!

2

Hi, can you share something that’d let me identify your app? (you have several and I don’t know which one this refers to). A machine ID should be OK to share - they’re not really sensitive. Otherwise, the last 6 characters of the app name should be OK. If you really want to be cloak-and-dagger, sha256sum the app’s name and share that; I can hash your app’s names on my side and find the one.

3

Thanks! To avoid posting direct app or machine identifiers publicly, here is the SHA-256 of the app name, computed with no trailing newline as printf '%s' '<app-name>' | sha256sum:

67bd923f97c92ed80d217792e4c601946b2c7316461b986726cb32075c62fbd9

I can provide the exact app name or machine ID privately if there is a email

4

Hi, have you tried the workaround suggested here? https://community.fly.io/t/flyctl-secrets-list-keeps-showing-all-secrets-as-staged-despite-deploys/27871/5?u=roadmr

5

Yes, we tried that workaround already, and this app behaved differently from the linked case.

What we tried, sanitized:

  • fly secrets sync
  • a non-staged fly secrets set on one existing, non-sensitive config key, using its current value and without printing any secret values
  • same-image deploy / machine update afterward
  • health/runtime checks after each attempt

Those attempts created healthy completed releases / machine updates, and the app stayed healthy, but fly secrets list -a <app> --json still reported all 58 secrets as Staged.

The part that still looks different here is the release index mismatch: fly releases -a <app> --json and the active Machine show release v536, while the UIEX releases endpoint used by flyctl status calculation only returns up to v487 for this app. So my current suspicion is that the non-staged set workaround cannot clear the displayed status because flyctl still cannot find the active Machine release timestamp from that endpoint.

Is there anything Fly can refresh/repair server-side for the app’s releases index? I can provide exact app name / Machine ID privately if there is a secure channel you prefer.

6

Hm… I’m not with Fly.io, but mainly your question is whether this bodes ill for the secrets system as a whole, right?

My understanding is that flyctl is just using heuristics when it says Staged, :raggedy_rabbit:, and that neither of the version numbers that you were looking at are truly authoritative for that purpose.

What is definitive is the independent CRDT(?) counter referenced by min_secrets_version. The platform infrastructure itself may not be interested in recording the exact values† of that one for posterity.

(There’s a local approximation of sorts in ~/.fly/config.xml under app_secrets_minvers. Mine have values in the high 8 digits. I believe this is mainly intended as a constraint on future operations, though, and not as a record of past ones.)

†By this I just mean which specific generation held sway when a given Machine was created, etc.

7

Thanks for the app_secrets_minvers / min_secrets_version pointer. I did a read-only recheck today.

Current sanitized result:

  • fly secrets list -a <app> --json now reports Deployed: 58.
  • The local flyctl config has an app_secrets_minvers entry for this app.
  • fly machine list --json, fly machine status --display-config, and the Machines API did not expose a min_secrets_version / min_secrets_version-like field that I could compare directly; they only showed the current release metadata.
  • The active Machine / fly releases are now on v544, and the releases endpoint used by the prior flyctl-style recomputation now includes the active Machine release.

I did not mutate secrets, Machines, or deploy anything during this recheck. So from my side the displayed secret status is now repaired, but I cannot tell whether this was fixed by a server-side refresh, subsequent releases, or a flyctl/platform behavior change.

Thanks again for the pointer. If Fly staff wants exact identifiers privately, I can still provide them via a secure/private channel.

9

Hi @alexforfun ,

We’ve deployed a change which should fix this problem. Can you check whether this works well for you now?

Thanks!