I haven’t been able to find any answers to this online or in this forum. There is one unanswered post from 2021 - Search results for 'hot code' - Fly.io
I’m assuming the answer is “no” due to containerised nature of Fly, but it would be nice to know if I’m missing something. My app has long-lived live sessions (chat/gaming) that don’t play nicely with frequent traditional deployments.
(To avoid an XY problem, I would also consider alternatives that allow for blue-green handovers of genserver state during deploys).
I also think it’s no for now too unfortunately. But I’m also very interested in solutions for your problem which is very interesting too
We do support bluegreen btw!
So here’s a few random ideas ranging from ‘kinda easy’ to ‘we are building a platform on top of Fly’ so feel to dismiss any or all.
Application logic to reconnect
This is a wild guess since I don’t know your frontend specially but could you in theory just reconnect all folks in the same channel to a newly machine?
Processes gracefully handover work to newly deployed machines
If you launched your app recently Phoenix would have generated an env.sh.eex roughly like this:
#!/bin/sh
# configure node for distributed erlang with IPV6 support
export ERL_AFLAGS="-proto_dist inet6_tcp"
export DNS_CLUSTER_QUERY="${FLY_APP_NAME}.internal"
export RELEASE_DISTRIBUTION="name"
export RELEASE_NODE="${FLY_APP_NAME}-${FLY_IMAGE_REF##*-}@${FLY_PRIVATE_IP}"
RELEASE_NODE is the main important bit here: we make it left@right where left is a combo of app name and deployment image ref because we don’t want your updated machines to cluster with older machines.
You could remove that and do some application logic to handover to new genservers? That definitely requires application code.
Releases with volumes
That’s a thing I’d love to try someday myself but the gist is each machine would have a small volume (say 1GB) and we’d not update the machine using the regular fly deploy command, we’d just scp into the machine the new code and do whatever it takes for hot code reloading.
Machines would only be needed to be fly deployed in case there was a change in the Dockerfile (say install new stuff).
This is a wild guess since I don’t know your frontend specially but could you in theory just reconnect all folks in the same channel to a newly machine?
That works for clients connected to liveviews (ephemeral processes) but not for the long-lived process backing their game state. (In this case, yes, we could explore things like persisting the game state and reloading on a new machine, but this gets complex).
Digging into bluegreen a little more:
It’s my understanding that they use a healthcheck at their core. Are there other ways to control the bluegreen? Because I’d need to do something like: “launch new machines, wait for health, transfer process state to new servers, only mark an old machine as blue once all processes are migrated off it”.
Given point 1, are there timeouts or other weirdness that we would hit if we had a server taking a long time to hand-off?
What would happen if a machine was never marked as blue? E.g. it has a problem and the process won’t handoff.
Are there other ways to achieve something a bit like bluegreen without using your specific bluegreen pipeline? E.g. scale from 5 machines to 10, the new machines having the new code, then scale back down to 5 when ready, culling the old machines?
To reply to all of them at once. Our blue green strat will do its best to ensure all new machines are passing health checks to route traffic to them you can even include your own health checks if you feel like you should:
Maybe you could sync new machines with their old counterparts and pass the check when it’s done? I’m not sure how that would go, just throwing an idea out there to see if it helps.
If the new machines never pass health we are not going to route to them
Another approach: our machines API is very exposed to the public. You could deploy with a strategy like:
Build the image with fly deploy --build-only and store the image ref.
Clone machines (either via flyctl or machines API) but just change the image ref.
Your app would cluster all images and machines know how to takeover state
I feel like flyctl can handle steps 1 and 2 neatly. Step 3 is application code specific though.
