Deploying from a rolling DockerHub tag

Hello. We deploy our application from a “rolling” tag on DockerHub. Every time we commit to our main branch on GitHub, we push to a tag called next on DockerHub (a lot of projects on DockerHub use a tag called latest in the same way). Our deploys are manually triggered. When we run a deploy, our deploy command is:

flyctl deploy --remote-only --image

We also regularly run scaling events on a schedule to increase/decrease the number of VMs we are running at different times of day, so at various points throughout the day we will run a command like

flyctl scale count 14 --yes

Recently, I noticed a problem which boiled down to some of our machines running one version of the application and some machines running a different version. Both versions concerned corresponded to code that was pushed after the most recent deploy (i.e: code I thought had not been deployed yet!).

My theory is that when scaling events occur and we need to add machines, they are now pulling the latest image from instead of using whatever image the currently running machines are using. So if there is something new pushed to the next tag, any new machines come up running that image but any existing machines in the cluster are still running an older one.

I am pretty sure this is new behaviour. We’ve deploying from the next tag since we migrated to over a year ago and I’m fairly certain that until recently adding new machines used whatever image the currently running machines are using, even if a newer image was pushed to the next tag on DockerHub. Maybe this is a new change since the migration to fly apps v2?

For the moment I am going to switch to deploying from a specific digest. So instead of deploying, our deploy command will become something like

flyctl deploy --remote-only --image

each time.

Would it be possible to confirm:

  • Is what we are seeing intended behaviour, or unexpected?
  • Is this behaviour new since the migration to fly apps v2, or some other change?


This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.