Flyctl has always been more than just a wrapper for API calls. That said, until recently, you could easily get by with very infrequent updates. With the switch from Nomad-based orchestration to Apps V2 – essentially an entire rework of the platform from the ground up – most of the smarts of our PaaS deployments have been moved from the server into the CLI.
This is great! It means we can iterate very fast, it means that our infrastructure can be so much simpler (and therefore more reliable), and it also means that you could theoretically add custom features to the platform – such as deploy strategies – through a fork of the CLI. (by the way: flyctl is open source!)
The problem with this is that if you don’t have an up-to-date flyctl, you could be missing crucial bug fixes and updates to platform features. I think everyone’s clicked the “ignore OS update” button a couple times, but old versions of flyctl are a liability when production services are at stake. We want to make sure that people don’t accidentally break their apps (that’s kind of the nightmare scenario here), and we’ve seen old versions of flyctl subtly break things time and time again.
All of that considered, here’s the gameplan to try to make this situation better:
Flyctl autoupdate. Some have wanted this for ages, some hate the very idea of it! It will be enabled by default, but it will be optional.
Done!
You’ll start to get error messages from the API when your flyctl is too old.
The current target is roughly: two weeks after a significant breaking change. We’ll probably adjust this goal over time, but the real goal is to balance reliability with not being a pain.
Please note: none of this will affect people using Fly.io APIs directly. We’ll only check version info for user agents beginning with fly-cli.
For those of you who want full transparency, here’s the long-term roadmap, but it’s still very much subject to change:
As of flyctl 0.1.51: Autoupdating
This will be configurable with fly settings autoupdate, and will be automatically disabled if it seems like flyctl is running in a CI environment.
Monday, Jul 10: the API will start erroring on flyctl versions below 0.1.20, requiring people to update to a more recent build of flyctl. There are a few major bugs present in the 0.1.1x series, for example, which would be quite frustrating to learn about by breaking production services.
Long-term: flyctl will start checking against the API to see if it’s significantly outdated, meaning the client itself will determine its status. This will allow us more fine-grained controls, such as yanking versions that have serious issues.
Roughly two weeks after: the API will start requiring the version of flyctl that checks its own version. That gets everyone on a version of flyctl that’s smart enough to tell when it is too far out of date.
The goal throughout this process is to prevent frustrating experiences and surprising issues. If you have any concerns or feedback about this effort, please let us know in this thread! We want to make sure that this goes smoothly and doesn’t cause too many headaches.
Quick update: autoupdating just got merged into flyctl.
Flyctl 0.1.51 will have automatic updating. Here’s what this looks like:
macOS and Linux
updates will happen in the background. When you run any command, if flyctl detects that it’s out of date, it’ll run the updater in the background. (this is a separate process, so it won’t hang your terminal or anything!)
Windows
I couldn’t settle on a similar background update system for Windows that didn’t compromise the UX. Updating on Windows ends up causing a UAC prompt, and randomly agreeing to (or asking people to agree to) unexpected UAC prompts is a horrible idea, so I just scrapped it for Windows.
I don’t think it’s impossible to solve this, but that’s a problem for another time.
For all supported platforms
If your flyctl is severely out of date (currently that’s defined as 5 versions out-of-date), it’ll update before executing your command, then relaunch on the newer flyctl. That way, if you try to do something on an old computer sitting around and we’ve completely changed the internal API or something, it should still Just Work.
If you absolutely hate autoupdating, there’s an escape hatch: fly settings autoupdate disable
If something goes wrong, or if you have any other feedback, please post about it in this thread or on the Github Issues page for flyctl. I hope this makes it a bit easier to keep up with bugfixes and platform features
It’s a neat feature in terms of functionally although personally I stopped updating solely because there were so many updates. I felt I spent more time upgrading flyctl than actually trying to do the task I was trying to do unlike say; Hugo where I happily upgrade because they’re released in semver batches
I can understand the need to continually ship patches given the CLI is evolving alongside the API and all that though.
Apparently I hit this new functionality but it puts my instance of flyctl into an infinite upgrade loop, presumably because some of my PATHing isn’t quite correct.
It’d be nice if the upgrade message had a note of the proper paths to set up. I imagine some users may have installed flyctl a long time ago and have forgotten where it might live or what is expected to be set up.
oh no! I’m so sorry it broke like this. I’m going to ship some fixes to prevent this from happening in the future, but in the meantime, could you provide the output of which fly and which flyctl? (feel free to redact anything personal that might show up in these) I’d like to figure out why it detected homebrew at first, then switched to updating using our installer script (which is what caused the loop in this case).
You should be able to disable autoupdate with FLY_NO_UPDATE_CHECK=1 fly settings autoupdate disable while we work on a fix here. Sorry for the trouble!
Invoking ~/.fly/bin/flyctl directly works as expected but without it in my PATH, the Homebrew version presumably just sees the version has failed to change and keeps redownloading.
It’s worth noting that the first call is to brew upgrade flyctl but I’m guessing brew update is missing here as (to my knowledge) brew upgrade doesn’t fetch new manifest files. It can only upgrade to versions that it currently knows about so it believes that the latest version is whatever is installed.
Manually running brew update and then invoking fly means the upgrade process completes as expected however but not all users may be aware that these things are related of course.
Personally I’d be a bit wary of running a brew update for the user given this can spiral into minutes long upgrade processes with older Homebrew versions “helpfully” running a full upgrade. That said, I think they changed that behaviour in newer versions?
I dunno, none of these problems are easy so it’s all tradeoffs really
you’re telling me! relaunching running executables is always a bit of a nightmare haha
I’m pretty sure (but I could be wrong, for sure!) that the homebrew updater only detects updates based on the local homebrew cache.(I was wrong, lol) There’s a whole rube goldberg machine of subtle things that all had to go just right to cause this to happen.
I think what happened in this case is:
we call into homebrew to upgrade
then relaunch the executable we get from realpath(argv[0])
so I think we ended up with the resolved binary path, which for homebrew would be /opt/homebrew/Cellar/flyctl/<version>/bin/flyctl
we’re still running from a different path than ~/.fly, though, so those updates never apply either.
My “quick fix” here is to just detect if we’re running from $(brew --prefix)/bin, and if so, our relaunch path will just be $(brew --prefix)/bin/flyctl. Hopefully that should be enough to fix this specific issue.
It’s difficult to test the Homebrew update code (because it relies on there actually being updates through homebrew!), so sometimes things like this break through. Hopefully the updater can stabilize soon and we can stop touching it
This thread has already helped me out! I ran into the same symptom without homebrew involved; my team uses nixpkgs to keep our dependencies in sync (currently pinned to 2de8efefb6ce7f5e4e75bdf57376a96555986841), and we encountered the same self-update loop in 0.1.53.
Setting the environment variable to skip self-updating worked like a charm to unblock us in this case, and if it will eventually error out when our CLI version is incompatible, I think that’s perfectly acceptable for us.
What if you make breaking API/CLI changes? How do we know about this?
Ideally I’m fine with autoupdate only if there’s no major/breaking changes to the CLI commands. Because getting breaking changes suddenly can break apps that use it in their CI pipeline
Sorry for the delay, I was out of office for a bit!
As far as I know, we don’t like to commit to maintaining command compatibility between versions. We try not to break existing commands, of course, but the flexibility is really important for shipping things as quickly as we do
If this is something you need, we encourage you to disable autoupdating via fly settings autoupdate disable (just make sure to update every now and then!)
Note, however, that if you’re specifically concerned about how this affects CI, it shouldn’t affect that at all. We disable autoupdating if we detect any of a few common CI environment variables, but if you want to be extra sure, you can set FLY_NO_UPDATE_CHECK=1 in your CI config.
@allison Not sure if I maybe should create a new topic / feature request about this, but:
I would really like the flyctl release log to be a bit more sorted, and also adhere to something resembling semver (major, minor/feature, patch/fix).
In the best of worlds the CLI update would just aggregate what has changed from version A to version B in these categories but that may be wishful thinking. But a release log where not all things are just mixed should be doable imho.
I needed to get a hotfix out and flyctl chose this moment to update. Also, I couldn’t even run the disable command – fly settings autoupdate disable w/o it waiting for fly to update.
I don’t know what to say besides a surprise “you can’t run fly commands for 5 minutes” is sometimes very unwelcome.
But also, I do not want flyctl to update itself, because I want to live in my glass house where binaries stay put and don’t fetch new versions of themselves from the internet.
I’m pretty sure the update-in-place will fail if flyctl is in the nix store, which is read-only. It looks like the autoupdate is currently removed via patch when flyctl is built for the nix store.
flyctl contains a lot of the logic for our platform features, and we make changes to it daily, as described in the OP. We generally assume people are running the latest or near-latest version. What’s the most nix-friendly way to make that happen? We do send error messages from our API to require a minimum version but that hasn’t been updated in a while.
Users of a rolling release channel like nixpkgs-unstable should be fine as update PRs are created automatically via nixpkgs-update (whose logs anyone can look at if there’s a problem).
Users of the “stable” releases could be up to six months out of date, but it’s quite easy to add an unstable overlay and install a newer version of flyctl from there.
Fly could provide a flake.nix, but I don’t think there’s much value to be gained from that.
I don’t mind the auto-update, but I wish the tool wouldn’t auto-update itself literally three times a day – because that slows down my workflow. I think the tool should store an internal timer and not update itself more than once a week.
If I’m really in the flow of thinking something through and fluently chattering at the command line then it really bugs me when some tool decides that it will just disrupt my chain of thought with an attempt to auto-update. If it happens once, then I’m annoyed but that’s it. If it happens again an hour later and then again two hours after that then I start to feel positively hassled.
It is almost anxiety-inducing, in fact. I get the sense that the tool changes every 20 minutes.
On top of that, the update is constantly failing for me because I guess my Mac OS also did an update and now I’m running 15.0 (Sequoia) just because Apple somehow pushed that update on me as well.
So now flyctl constantly triggers Homebrew warnings about being a bad person for running a beta version of Mac OS (even though I was just updating like Apple encouraged me to). But it also never succeeds in updating. This means that it ALWAYS thinks it has to update because it never succeeds.
The “fly is updating” harassment was already annoying, but now it is positively destroying my day – so I had to set the flag to turn it off. Hopefully it will one day update successfully and then just shut up and do it’s work quietly for a full week.
