Heads up… unfortunately 2 show stopping bugs went out in a flyctl release about 2 hours ago. The first impacts builds on GitHub Actions. The second could cause every command to throw an “Oops” error in interactive terminals, like your dev environment.
We yanked the offending version, 0.1.59. And we also yanked 0.1.60 because it only fixed 1 of the issues. The current stable release is 0.1.58.
Until we get this resolved, you can revert to 0.1.58 by running the installer. GH Actions will pickup the correct version on the next run.
Just want to say, thank goodness for the Fly Terminal! One of my apps became unresponsive where the app status shows “Host is down for maintenance”…and with the fly-cli down I couldn’t scale my app to a different region since the UI doesn’t provide this capability
Yeah, scaling disappeared from the UI, I noticed that too. Was helpful for me too a couple of time when I was on a mobile phone only and needed to adjust something quickly.
Until we get this resolved, you can revert to 0.1.58 by running the installer . GH Actions will pickup the correct version on the next run.
You mention that GH Actions will pickup the correct version, but mine is still failing over and over again. Here is what my workflow looks like this: Github Actions deploy to fly logs · GitHub
The installation link gives instructions on how to install on one’s local machine. How do we control the flyctl version on the Github actions host machine?
This has been the issue for me for 36h now. Running it via GitHub produces the same result as @scatterinfo and running it locally produces the screenshot below (also sent via email), where the yellow line constantly loops over and over after it pushes 5GB+, it restarts the same line with same id with another 5GB+ push.
We’ve seen issues with GH Actions where the token contained a newline after submitting through the GH ui. I wonder if that’s happening here.
As for the push failure, a 5gb layer is unsupported and going to cause trouble. Did anything change since the last successful deploy to blow the image size up?
Just checked Github UI with a new deploy token, no newlines when I saved it, and deploy still fails the same way.
I checked the Dockerfile, nothing has changed (you can see contents in the yellow screenshot), not a lot of custom additions beyond the apt-get line on top of python slim. It’s been in the GBs since the first time I launched the app.
I tried destroying running machine and adding a new one, same issue
@Michael deploy token issue went away. then it looks like the 5GB+ container that was working before kept timing out, so did a dive and reduced container size by half. that got the deploy working again! thanks for your help (and we should catch up!)
The latest release (0.1.163) fixes the issues mentioned here, so you should be okay if you let auto update do it’s thing. That said, auto upgrade is too aggressive right now and we’ll be fixing that in the coming days. Sorry for the trouble!
Hi @michael
This has been happening to my team today for the last 2.5 hours where I announced I was about to do a deploy. I’ve been using 0.1.163 for the last hour or so.
In my case, it fails to push a 1.193GB layer. It is weird as it managed to push a 1.9~ GB layer a while back, but no matter how much time passes, it gets stuck trying to push that layer, then stalling for about a minute, then going for a retry.
Thank you for looking into this. It would be great to prevent autoupdate because in some of my builds v0.1.63 still has another blocking issue (build secrets not found).
@butaca I just found out you can disable auto update with fly settings autoupdate disable. Run that then 0.1.58 will stick. Note that you might need to stop the agent first with fly agent stop.
