Rollback

I’d love to be able to roll-back to a previous version. Imagine a bug was pushed to production, and having to wait 15 min for the fix to be deployed seems too long. Can we have rollback to at least previous version, even just retained for 24 hours would be great.

9 Likes

Yeah I want this too. This is on our medium term list, we’re doing some architecture work first and then we can (hopefully) ship orchestration changes much quicker.

2 Likes

Is this still in progress? Or, is there a way to do this by specifying a docker image in the fly.toml file?

I’d also love to be able to specify a previous release number and roll back to it, something like fly deploy --rollback=212

We, unfortunately, haven’t shipped this yet. We’re furiously replacing Nomad so we’ve gotten behind on quick fixes for now.

You can run fly deploy -i <url> to deploy a specific Docker image. The only way to get that is to look at the logs during a deploy, though. We don’t have a simple way to show those from previous versions.

2 Likes

Interesting! Would love to see a blog post about this.

1 Like

Sorry for an ‘any update’ post but… any update on this potential feature?

Would be awesome for closing the gap with our current platform (h*roku), even if only available in API (vs dashboard/flyctl).

4 Likes

+1 on ‘any update’. No rollback is kind of a non-starter for any app serving customers. Gotta be able to rollback quickly if a mistake happens. Thanks for any updates you can provide!

5 Likes

Need this fly deploy rollback feature now:pray:
While trying to setup an SSH server on our Fly VM, we managed to “break” it and the deployment failed so now nothing works … :cry:

1 Like

Well, a specific rollback feature is one for Fly however in the meantime I think you can kind of make your own: run fly releases --image to get a list of the releases. And in that table you can see the image/hash of each deployment. So if you want to rollback to a particular version, look for its image (right-hand column) and deploy that image e.g fly deploy -i registry.fly.io/your-app-name@sha256-long-hash-here-from-that-table.

I’m not sure how long Fly stores images for, however I assume all the ones listed are available to deploy.

17 Likes

Is this still the only way - a command in the fly cli would be nice.

A caveat: This won’t rollback secrets and other Fly-specific changes (like app config), just the code (ie, deploys a previously built docker image).

2 Likes

@ignoramous That’s a good point. I was thinking someone would use a rollback within minutes of a prior deploy and so that would be less of an issue. But you are right to highlight it. That’s why this is kind of improvised. I would guess any future official rollback feature would include both the code and config (the state). This approach doesn’t.

1 Like

I would love to have the rollback feature too. My site just crashes after deployment and I have no idea how to recover it. It took me 20mins to figure out what happened and try to redeploy the app.

1 Like

Ref: Add "fly app rollback 731" command · Issue #579 · superfly/flyctl · GitHub I imagine this feature won’t be easy for Fly to implement.

The way we do it is, deploy to two Fly apps. One runs last known good deployment but with only one or two VMs (I call this env, lgtm-box), typically a release behind. The other is a proper prod environment, which means, it is also our test environment :wink:

If something goes wrong, I scale up the lgtm-box and flip the DNS entries to point to it, while I figure out just what went wrong when testing in prod…

This means, if you’ve a database layer or any external state, it must at all times work with code running in the lgtm-box.

1 Like

I created a small CLI to make this process easier.

Shell script, zero dependency, no config required, uses the fly cli underneath.

Cheatsheet:
fly-rollback list: list releases
fly-rollback HEAD^1: rollback to the last release (one before the current one)
fly-rollback v25: rollback to the v25 release

PRs and issues welcome :slight_smile:

3 Likes

Intersting. Anyone can do rollbacks easily using FlyCD.

There’s a button for “instant rollback” that rolls back to the last production deployment as well. You can even retry failed deployments easily and track commits related to your release.

2 Likes

This looks sick!

But, to be honest, I wouldn’t wanna share my fly api token to a third party service. Especially if it is not open source.

You can use create and use deploy tokens if you’re concerned about that. With that you’re only limited to deploying to a particular org, and the token is limited to what it can do. If you have any other concern, we can talk more about it via a video call.

I’m sure there are services which you trust with secrets/tokens and password (e.g Secrets managers like Doppler). Anyway, secrets are securely stored via a service, so I can only say you should trust it :thinking:.
If the app doesn’t have your API token, there’s no way it knows how to deploy your apps… just like you trust GitHub Actions with your token when you configure deployment from there.

1 Like

FlyCD looks perfect. Does it support Docker deployments/rollbacks?