Boot-first rolling deployments

The default deployment strategy was explained in another thread:

What I would like to see instead is an option, or a new default, where once the canary succeeds and replaces the instance it was meant to replace, a new instance is booted up and the old instance is taken down after the new instance is running and health checks are passing.

This was how I had docker swarm configured (using the update-order: start-first flag). This prevents an issue where a client connected to a killed instance gets booted off to a) an instance in a further-away region, and/or b) an instance that is about to be killed as the rolling deployment progresses.

Probably not the hugest deal, but a nice-to-have…

This is relatively high on our list. Should be possible shortly after we get Postgres out!

1 Like

Great! While you’re at it, please consider using the same “start-first” strategy for changing vm sizes, so that scaling to another VM type spins up new VMs before killing the old ones. :blush:

2 Likes

Just adding on here, but it would be nice to orchestrate these. I think actually this could be done on fly’s side in a way that would just save whole ton of glued together config. Basically green/blue but works for staging as well…

fly deploy --stage

Would deploy the app to green.appname.fly.dev, allowing you to then set up as many alternates as you want. If you ran into a bug mid-way, you could just

fly destroy --stage

Once all tests passed, swap them:

fly deploy --swap

Later on those could take arguments for having more than one, or something more elegant.

Nothing new here of course, just hadn’t seen any discussion anywhere of it and wondered how high it ranks generally. We’re still pre launch and I don’t mind firefighting a bit if we had weird deploys, but would certainly be one of those last pieces of the puzzle that would be quite a value add.

2 Likes

I really like that idea, @nate. FWIW, lack of this feature is the reason I ended up migrating over to Fargate.

That’s clever. We’d have to attach a second IP to really do it properly (because we route that way), but that’s totally doable.

1 Like

@luke you moved to fargate for branch preview releases, or for proper blue/green deploys? If I got blue/green deploys into flyctl today would you switch back? :smiling_imp:

(We have the plumbing in, we just haven’t shipped it yet)

The lack of start-first deployments (or blue/green deploys) was definitely irritating as it would disrupt users’ experience here and there during updates of my slow-booting app. But if I think back, it was also because your platform seems to be evolving quickly, and there was some quirky behavior here and there. Fast progress is exciting and I’m totally rooting for you guys, but for this project I really just want a boring stable service to host a simple nodejs Docker container, and Fargate fit the bill. I hate changing providers so I will only do it if Fargate fails me for some reason, but if that happens someday I’ll certainly check back in with fly.io which I’m sure will have everything I’ll need by then. :blush:

We had a whole terraform / k8s setup before that I, as a non-ops dev, wasn’t happy working with it generally. But perhaps commands like this would open to making a custom terraform provider, which would bring back some of the nice part of that? I’m actually not 100% on the limitations of tf, but I like the idea generally.

Ooh yes. Our service discovery got overloaded and caused errors during deploy it shouldn’t have. We rebuilt that recently, it’s really nice now. I’m not aware of anyone getting errors during deploys like that any more (especially with good graceful shutdown settings).

flyctl deploy --strategy bluegreen works now! It’ll launch all the new instances, wait for them to become healthy, then shut down the old ones. Give it a try sometime and see how it goes.

2 Likes

Very cool! Though separate from what I discussed above right? As in, this one works within the same app, but having a separate trigger to actually finish the deploy would let you stage different apps and then release at once once ready.

Anyway, I assume you know that and are just pushing out nice things incrementally, but would still love to have that option.

Awesome to see all the progress.

Right! Staged deploys, or even branch previews, are something we still haven’t figured out. A terraform provider is likely though. :smiley:

1 Like