[How to / Ideas] Rollbacks

Show & Tell
, , , ,
1

So you have done a “bad” deploy.

Don’t panic. Everyone does one at some point, we’ve all been there. We don’t have a full UI-integrated rollback system, but it does not mean you can’t roll back. Actually you can and it’s not that hard. See for yourself:

Most common case, you’ve only made a (bad) change in your app’s code that you want to revert:

We do keep your previous app’s image builds. To see them, run:

fly releases --image

Spot a previous release you want to redeploy, find its image name in the DOCKER IMAGE column, and run:

fly deploy -i <image-name>

(The image name should be like registry.fly.io/app-name@sha256:some-hash.)

:zap: Shortcut: if you have jq installed, the steps above can be combined in this one-liner:

fly deploy -i `fly releases -j | jq ".[1].ImageRef" -r`

→ You’re saved! :ring_buoy:

If also you want to be able to revert changes made to your app’s configuration (services, env…), keep reading for an idea of a more resilient deployment process:

We keep your app’s previous builds, but your app’s configuration (stuff that’s in your fly.toml like services, environment variables…) is harder to retrieve. But here’s an idea: you can easily™ run a command to back up your fly.toml contents as you deploy:

Instead of just running fly deploy you can run (assuming you have jq installed):

mkdir -p fly_deployments && \
export FLY_DEPLOY_DATE=`TZ=GMT date +"%Y-%m-%dT%H:%M:%SZ"` && \
cp fly.toml fly_deployments/"$FLY_DEPLOY_DATE"_fly.toml && \
fly deploy && \
echo "\n\n[build]\n  image = \"`fly releases -j | jq ".[0].ImageRef" -r`\"\n" >> fly_deployments/"$FLY_DEPLOY_DATE"_fly.toml

What this does is:

  • back up your config fly.toml in a local fly_deployments folder by prepending the ISO8601-formatted date and time of the deployment to the backup filename (fly_deployments/DATE_OF_DEPLOYEMENT_fly.toml)
  • deploy
  • get the build & deployed image name, and append it to the backed-up fly.toml in the [build]section

→ This way if you want to roll back later (including services, environment variables…), all you need to do is:

fly deploy -c fly_deployments/DATE_OF_PREVIOUS_DEPLOYEMENT_fly.toml

This will use the previously backed up fly.toml config, that will include previous services, env. vars… and your previously built app image! :sweat_smile:

Now tell us what you think! What’s your experience about rollbacks? What do you dream / have nightmares of?

2

Neat. Keep in mind though, backing up fly.toml isn’t going to be enough in some scenarios:

  • As part of deploys, changes in secrets are propogated too. Harder to roll that back.
  • Unsure how infrastructure related changes driven by fly.toml (like mountpoints, say) might behave.

For our setup right now, for rollbacks we keep around 2 Fly apps: One runs the last known good version (say, GOOD), and one runs the current code (say, LATEST). If LATEST is going bust, we switch over the DNS entries to point to GOOD: Rollback - #15 by ignoramous

3

We don’t really have a rollback plan for infrastructure, if it’s changed then a “rollback” is another change.

We have a staging environment which is a complete like-for-like setup of our production environment which all our changes pass through including infrastructure changes. This allows us to catch infrastructure change issues in an environment which we can destroy without worry.

4

Any ETAs on instant rollbacks via UI like Vercel? We have some long running workers hosted on Fly and a Remix app on Vercel. Would really like to move everything to Fly to reduce the number of platforms we manage, but the lack of a instant rollback is the one thing that’s preventing us from migrating the Remix app.

When you need to rollback, it’s easy to further screw things up if it requires 2-3 terminal commands.