Production down - Registry 401 auth errors blocking all deployments

1

My production API is completely down and I cannot deploy ANY app to Fly.io due to registry authentication failures.

Error

Every deployment fails with:
401 Unauthorized from HEAD request to http://_api.internal:5000/v2/pelican-api/blobs/sha256:[various-hashes]?ns=registry.fly.io

What I’ve tried

  • flyctl auth logout/login (multiple times)
  • flyctl wireguard reset
  • flyctl agent restart
  • Destroyed app and recreated - same error
  • Created entirely NEW app with different name - same error
  • This persists across multiple apps: pelican-api, pelican-trading-api

Impact

  • Production API (pelican-api.fly.dev) is completely DOWN
  • Cannot deploy any fixes or new apps
  • This appears to be account-level registry authentication failure
  • Account: nick@pelicantrading.ai
  • Organization: personal

Critical timing

I’m leaving for a week of travel in hours. My team needs this API operational.

This has been happening for the past 4+ hours. I’ve already submitted a support ticket but need immediate help.

The registry authentication is broken at the account/organization level. Every single deployment attempt fails at the push stage with 401 Unauthorized.

Please escalate - production is completely down.

2

Just a thought: I think there is a Fly feature where you can build an image, and then push the image as part of the deployment. I don’t know if that goes through the registry, but it’s worth a try.

Let me have a look at flyctl…

But a clarification first:

Are you deploying via GitHub or the console? If the latter, what exact command are you using?

Update

Maybe flyctl deploy --local-only --image=pelican-api:v123 or similar? I’ve not tried this, but maybe it’s worth a go.

Also, make sure you do flyctl version upgrade, so you’re using the latest command.

3

Thanks for your help! Got two errors:

  1. WARN failed to create build in graphql: input:3:2: createBuild Could not find App
  2. Error: no docker daemon available

The “Could not find App” suggests fly.toml might be pointing to the wrong app. My production app was destroyed during troubleshooting.

Current situation:

  • Original app “pelican-api” was destroyed
  • Cannot create/deploy new apps due to authorization errors
  • Every attempt fails with “unauthorized” or 401 errors

Update: Still getting 401 errors on all deployments. Fly.io support suggested --depot=false but that gives “unauthorized” on VM creation. The authorization issue affects both registry and VM access.

This is definitely an account-level problem requiring Fly.io engineering intervention.

4

Have you tried --local-only? I believe that is different to --depot=false.

5

Would you supply your TOML file here? Did it used to work?

6

is your image large? If it takes more than 5 minutes to build, it will fail with app not found b/c your auth token expired.

7

I too am getting 401 Unauthorized when I do fly auth docker… The recent registry changes have been :poop:

8

What regions are you (@khuezy and @nick-groves-94) in?

I just did this for the LDN region:

fly auth docker
docker push registry.fly.io/my-app-name:small # i.e. custom tag
docker push registry.fly.io/my-app-name2 # i.e. latest

Both were successful (both already pushed).

9

My GPU machine is in ord. I bake my model in the image so it exceeds the 5 minutes timer.

10

But if your flow is failing at fly auth docker, how are you getting to push an image?

(The “401 Unauthorized” error could result from not supplying a token, or from supplying an invalid one, of course.)

11

I created a token that never expires but it was somehow revoked. I recreated a new one but it errored with unauthorized for about 10 minutes.

Something is going on with the platform. No word from staff though

12

same issue here for GRU region machine. “failed to push registry.” with 401

13

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.