Quick fix for today's flyctl regressions

Heads up… unfortunately 2 show stopping bugs went out in a flyctl release about 2 hours ago. The first impacts builds on GitHub Actions. The second could cause every command to throw an “Oops” error in interactive terminals, like your dev environment.

We yanked the offending version, 0.1.59. And we also yanked 0.1.60 because it only fixed 1 of the issues. The current stable release is 0.1.58.

Until we get this resolved, you can revert to 0.1.58 by running the installer. GH Actions will pickup the correct version on the next run.

Sorry for the trouble!


@Michael I’ve been stuck on a standard Github Actions deploy since yesterday 3:30PM PST. No amount of re-running works. Looks like it’s on 0.1.58

I’ve tried changing machine type, memory, nothing works. Latest deploy has minimal code changes and works locally.

Just want to say, thank goodness for the Fly Terminal! One of my apps became unresponsive where the app status shows “Host is down for maintenance”…and with the fly-cli down I couldn’t scale my app to a different region since the UI doesn’t provide this capability :confused:


Yeah, scaling disappeared from the UI, I noticed that too. Was helpful for me too a couple of time when I was on a mobile phone only and needed to adjust something quickly.

1 Like

@michael How can one roll back to a previous version of flyctl exactly?
I assumed
brew install flyctl@0.1.58

But it didn’t work

curl -L  https://fly.io/install.sh  | sh -s - 0.1.58
1 Like

Until we get this resolved, you can revert to 0.1.58 by running the installer . GH Actions will pickup the correct version on the next run.

You mention that GH Actions will pickup the correct version, but mine is still failing over and over again. Here is what my workflow looks like this: Github Actions deploy to fly logs · GitHub

The installation link gives instructions on how to install on one’s local machine. How do we control the flyctl version on the Github actions host machine?

I see this as well

That looks like it’s failing to push the image to the registry. Is it still happening? Does it happen with any other apps?

This has been the issue for me for 36h now. Running it via GitHub produces the same result as @scatterinfo and running it locally produces the screenshot below (also sent via email), where the yellow line constantly loops over and over after it pushes 5GB+, it restarts the same line with same id with another 5GB+ push.

@scatterinfo the logs are showing an error when pushing the image:

Error: failed to fetch an image or build from source: error rendering push status stream: unauthorized: not allowed

That “not allowed” makes me think its an auth issue. Are you certain the token being passed to the first deploy step has access to the app?

@Michael I’ve created new deploy tokens 3 times now across 36h, same issue over and over.

We’ve seen issues with GH Actions where the token contained a newline after submitting through the GH ui. I wonder if that’s happening here.

As for the push failure, a 5gb layer is unsupported and going to cause trouble. Did anything change since the last successful deploy to blow the image size up?

Just checked Github UI with a new deploy token, no newlines when I saved it, and deploy still fails the same way.

I checked the Dockerfile, nothing has changed (you can see contents in the yellow screenshot), not a lot of custom additions beyond the apt-get line on top of python slim. It’s been in the GBs since the first time I launched the app.

I tried destroying running machine and adding a new one, same issue

@Michael deploy token issue went away. then it looks like the 5GB+ container that was working before kept timing out, so did a dive and reduced container size by half. that got the deploy working again! thanks for your help (and we should catch up!)

I’m installing flyctl this way, but it keeps auto updating to the latest version. Is there a way to prevent this?

The latest release (0.1.163) fixes the issues mentioned here, so you should be okay if you let auto update do it’s thing. That said, auto upgrade is too aggressive right now and we’ll be fixing that in the coming days. Sorry for the trouble!

Hi @michael
This has been happening to my team today for the last 2.5 hours where I announced I was about to do a deploy. I’ve been using 0.1.163 for the last hour or so.

In my case, it fails to push a 1.193GB layer. It is weird as it managed to push a 1.9~ GB layer a while back, but no matter how much time passes, it gets stuck trying to push that layer, then stalling for about a minute, then going for a retry.

Remote builder fly-builder-rough-fire-2273 ready

==> Building image with Docker

--> docker host: 20.10.12 linux x86_64

[+] Building 26.1s (15/15) FINISHED
--> Building image done

==> Pushing image to fly

The push refers to repository [registry.fly.io/hydra-production]

1bf2acf1de47: Pushed

a2b73e93eeca: Pushing [==================================================>] 1.193GB

03dca29eb962: Layer already exists

68161e412545: Layer already exists

ddf7daf76a39: Layer already exists

ec81afb7c558: Layer already exists

86545e2162aa: Layer already exists

fb2616ec6129: Layer already exists

a94db5b6b93f: Layer already exists

52764e96f364: Layer already exists

52f6ba971156: Layer already exists

b182c5ada93e: Layer already exists

8ad0088b4e6d: Layer already exists

4b3ba104e9a8: Layer already exists


1bf2acf1de47: Pushed 
a2b73e93eeca: Retrying in 1 second 
a2b73e93eeca: Pushing [========>                                          ]  201.5MB/1.193GB

Thank you for looking into this. It would be great to prevent autoupdate because in some of my builds v0.1.63 still has another blocking issue (build secrets not found).

@butaca I just found out you can disable auto update with fly settings autoupdate disable. Run that then 0.1.58 will stick. Note that you might need to stop the agent first with fly agent stop.

1 Like