A new deployment strategy beyond bluegreen

1

I use bluegreen deployment strategy to deploy my app. It works as intended but I really have the need of a slightly different strategy that serve well especially in the context of TCP connections. The idea is the following:

  • green machines are created
  • green machine health checks are awaited
  • blue machines are kept running until they have any active connection (can also be based on soft_limit)
  • blue machines do not accept any new connection, only green machines (from the moment the are ready)
  • blue machines are destroyed only when there’s no more active connections to them

In essence is very similar to bluegreen but less destructive. This way we need less logic in our app to handle deployment because there’s always a version of the app running as soon as it is used.

What do you think?

2

It would also help in case of multiple services in a monorepo, because a consistent version of them all is running at the same time

4

This sounds like a very interesting use case! I have multiple angles on how to answer that

Prior context:

  • cordon means "make this machine not routable for folks anymore, just keep whoever is still connected to it.
  • cordoned machines wont get new connections BUT will keep existing ones alive.

Option A) Bluegreen already does that

This scenario is something bluegreen already seems to solve, you might just need to tweak a couple things.

When green machines are started and have passing health checks we uncordon them (so they are reachable via proxy) and then we cordon the blue machines.

Bluegreen waits for blue machines to stop after it sends SIGINT (can be configured, see fly deploy --help and look for --singal). If you want to change how long to wait for blue machines to stop look for --wait-timeout.

If your app can gracefully terminate after all connections are closed then you’re golden! No new connections will be handled after

Note: check if your CI does not have a maximum timeout that would suddenly stop everything!

Option B) If you still want a custom workflow

Assuming increasing the timeouts is not the fix for your use case.

What fly deploy currently does is essentially calls to machines API to create/update machines.

May I suggest that you orchestrate this custom deployment logic either via machines API via your framework of choice or just by using flyctl commands?

Here’s how I’d do it myself (disclaimer: I used an agent to generate these commands and did not try it myself, I recommend trying in a test app)

# Step 1: Build only, capture image ref
fly deploy --build-only --push 2>&1 | grep "^image:" 
# → image: registry.fly.io/my-app:deployment-01234567

# Step 2: List existing machine IDs
fly machine list -q
# → <id1> <id2> <id3>

# Step 3: Deploy new machines, excluding all existing ones
# --ha=false prevents spawning extra standby machines per group
fly deploy --image registry.fly.io/my-app:deployment-01234567 \
  --exclude-machines <id1>,<id2>,<id3> \
  --ha=false

# Step 4: (after verifying new machines are healthy) Cordon the old ones
fly machine cordon <id1> <id2> <id3>

The decision on how to make them get destroyed when they’re not serving traffic anymore. One trick you can use for each machine that needs to be destroyed:

fly machine update <machine_id> \
  --autostop stop \
  --machine-config '{"auto_destroy": true}'

Let the proxy stop it after its idle and the machine also destroys itself instead of staying there forever :slight_smile:

I want to reiterate that I asked an agent to generate those commands as I did not have time to try it myself! (But Im a human and I read your question and suggested things according to how I’d have done myself)

C) Make an agent orchestrate your deployment

I still think that Option A is the way to go but…

flyctl launch is just a fancy open source client for https://api.machines.dev/

Copy your question and my reply into an agent and ask it to generate a custom shell script to orchestrate your deployments with all your details! Im not joking, it works surprisingly well

Here’s flyctl source: GitHub - superfly/flyctl: Command line tools for fly.io services · GitHub

Let me know if this helps!

5

Hello, thanks for replying.

So, option A is not an option for my use case, I cannot rely simply on wait_timeout. But I get that it is an option for other users and if they step into this post they should take it in consideration.

But you’re definitely right, there cannot be a valid deployment strategy for everybody and writing our own script is the solution. For example your option B is already a good strategy for me, I just have to investigate better on using the machine API.

I let you know if I’m able to create something meaningful and maybe share it with the community.

6

My own addition to this topic is not to struggle with Bash. It simply isn’t sufficient as a fully-fledged scripting language for the task - eventually you’ll need JSON decoding, conditionals, loops etc. I recommend that folks use what they normally use on the application side - Ruby, Java, Python, PHP, etc.

Once you’ve got some simple cURL primitives working, and some error handlers to catch common failure cases, you can start using it like Lego. I’d also not be surprised if the major library providers (NPM, Packagist, etc) already have a Fly API library you can use out of the box.

7

One problem I have with this solution is that machine update api will reboot the machine and this is exactly what should be prevented. Any hint on how to change the stop strategy of a machine without restarting it? Is it possible? Is there any alternative?

8

I do have some ideas but first Id like to check with you when that’s needed.

  • Are you talking about blue machines?
  • Can you confirm at which point it is needed for those to not be rebooted by proxy? (I assume it is after green machines are uncordoned and blue machines are cordoned)
9

Yeah, sorry, I’ve been not very precise. After the green machines are marked as healthy, we cordon the blue machines and we update their configuration. At this point, due to calling machine update, the blue machines reboot. This is what should be avoided.

10

Thanks for the confirmation.

If you mean the update call itself is rebooting the machine I think we have an option to make the machine keep stopped if it was previously stopped which is skip_launch I think but…

If the blue machine is stopped then you can just delete it right? That means the started blue machines are what’s left.

They:

  • Might be serving connections or not
  • Need to be destroyed when they become idle
  • Proxy will stop them when they become idle
  • New connections won’t happen because they’re cordoned.

The easiest way I can think to tackle this would be to have something in the background that destroy those when they stop. Here’s a brainstorm of different ways to handle that scenario

  • If you have some kind of orchestrator somewhere that can run regular jobs you could use it to watch for when these stop and then remove them. Heck, I bet even GitHub actions can do that.
  • If you want to put something under your own app code you could create a secret endpoint that schedules a “destroy myself” function when your machine stops by the proxy (remember that the proxy stops machines with that signal I mentioned above, you can trap that to trigger the function)
  • You could use machine metadata API to update the machine without a reboot and then in your app read that metadata and then destroy the running machine when it stops.
  • Leave old machines cordoned and make the CI clean then on the next deploy or on the next run or using a cron job

I think I’m missing some obvious things, don’t take that list as the only ways you can do this!

I’m happy to continue rubber ducking this with you

11

This is exactly what I’m going to do, destroy a machine that is cordoned at some point, either marking it with metadata or by checking if it’s cordoned or checking if it’s running an old version of the app.

The ideal would be to have a way to change machine configuration without rebooting it, but I see why that may not be possible.