We’ve pushed out a new version of flyctl which should be available soon which revamps how we manage fly secrets. If things go well you should not see any visible changes. If you experience any problems, please let us know.
Behind the scenes secrets are now managed via the machines API (see Fly Machines API) instead of the graphql endpoints used previously. These endpoints return a “minimum secret version” with each update, which is passed along when deploying machines to ensure that any updated secrets are pulled in to the new machine. The semantics here are that the secrets will be fetched that are at least as new as the minimum secrets version that is passed in (and possibly newer). If you manage secrets with the machines API manually, you should also track the secrets version and pass it along in the min_secrets_version field when launching or updating machines with the machines API.
The secrets versions are now tracked in the flyctl config file. There’s a caveat here – if you manage secrets from flyctl on one machine and then perform a deploy from flyctl on another machine its possible that the secrets will not be up to date. This inconsistency can last for a short while until secrets are propagated and secrets in caches are aged out.
I’ve got the same issue Error: update secrets: failed to update app secrets: failed to fetch discharge tokens: failed to fetch discharge tokens: tp error (401): unauthorized (Request ID:. This is on my CI pipeline so I can’t just do fly auth login. Do I need to attenuate the token I’m using or generate a new one or something?
EDIT - fixed, I regenerated a token thru Fly console UI and updated my CI. It works.
Hi, I got bit by this. I maintain dsd-flyio, which is a django-simple-deploy plugin that automates the deployment of Django projects to Fly.io. I used this plugin in a live demo during a talk at DjangoCon US in Chicago this week. The demo went well, but the next morning as we started sprints e2e tests started failing, and no one could deploy to Fly using the plugin.
It was failing because of this block in the fly plugin:
def _set_secret(self, needle, secret):
"""Set a secret on Fly, if it's not already set."""
# First check if secret has already been set.
cmd = f"fly secrets list -a {self.deployed_project_name} --json"
output_obj = plugin_utils.run_quick_command(cmd)
secrets_json = json.loads(output_obj.stdout.decode())
secrets_keys = [secret["Name"] for secret in secrets_json]
if needle in secrets_keys:
return
cmd = f"fly secrets set -a {self.deployed_project_name} {secret}"
output_obj = plugin_utils.run_quick_command(cmd)
To be specific, it was this line of code:
secrets_keys = [secret["Name"] for secret in secrets_json]
I’ve used the Name key for years without issue. This update seems to have changed that key to name.
It’s an easy fix on our end, but I currently have it wrapped in a try-except block for both possibilities. Are those keys all lowercase from now on? Would there have been any way to know about this change beforehand?
If you create a deploy token with fly tokens create, you get a Macaroon token. We’ve talked about these once or twice on the blog. And when you log into flyctl, it also gets a Macaroon token under the hood. That is, since May 2025.
If you logged in before then (and weren’t in an organization with SSO requirements), flyctl would get an OAuth token instead.
Our Machines API can’t verify OAuth tokens itself, but it broadly knows how to handle them – it can usually check them against the GraphQL API and do the operation you asked for.
An exception is the new secrets APIs – under the hood, the Machines API is talking to PetSem (our service for customer secrets), which requires a customer Macaroon, authorized for the secrets you’re trying to access (this requirement is an important part of how we protect customer secrets within our infrastructure). So if you send the Machines API a legacy OAuth token, it can’t authenticate to PetSem.
So you’ll see this error if you logged in before May 2025, in which case you can just log in again with fly auth login. You’ll also see this if you used fly auth token to grab flyctl’s OAuth token and used that for CI – in that case, you should create a new CI token with fly tokens create.
