Machines and Deployment Strategies

1

I have deployed two workers that just have to constantly run in the background to process tasks for my application.

Having deployed them with exactly the same structure in the fly.toml, which is no more then:

app = "mybgservice"
primary_region = "fra"

[build]
  dockerfile = "mybgservice.dockerfile"

Now after doing some deploys on each of them i checked the machines with

fly machine list

One service has just one machine, which is running.
The other service has two machines, from wich one is started and one is stopped.

Where can this diffference come from?

For this use case, I have to be sure that only one (and always one, that never shuts down) machine per service is running at a time.
From where does the fly platform know, hat the machine must never be shutdown (like done with http services when there’s no traffic for a while) so that it keeps running?

Regarding the deployment strategy, since the service provides no http service to perform health checks against, what does the default “rolling” strategy mean when my service has one running and one stopped machine?
Could it be the case, that for short period of time, both machines are running?

Sorry lots of questions, but just trying to understand what’s happening behind the scenes to be sure no unexpected behaviour occurs.

2

Hi @chrismetz

Are these two separate apps with their own fly.toml that you deploy separately?

And is that the whole fly.toml file?

If these are just apps without services, then they could have standby machines that were created during deploy.

Try posting the output of fly status for the apps as well so we can see the status of the Machines.

3

Yes, they’re two separate apps with separate fly.toml files, being exactly the same, only differing in Env Variables / Secrets.

And yes, that’s the whole fly.toml file. They dont start an http server, just running in the background.

The output of the status:

App
  Name     = mybgservice-test
  Owner    = personal
  Hostname = mybgservice-test......fly.dev
  Image    = mybgservice.....:deployment-.....
  Platform = machines

Machines
PROCESS ID              VERSION REGION  STATE   CHECKS  LAST UPDATED
app     6e82dd16c12048  4       fra     started         2023-06-19T09:01:16Z

App
  Name     = mybgservice-production
  Owner    = personal
  Hostname = mybgservice-production......fly.dev
  Image    = mybgservice.....:deployment-.....
  Platform = machines

Machines
PROCESS ID              VERSION REGION  STATE   CHECKS  LAST UPDATED
app     e784436f24d148  3       fra     started         2023-06-19T12:40:02Z
app†    2874470a021948  3       fra     stopped         2023-06-19T12:40:00Z

  † Standby machine (it will take over only in case of host hardware failure)
4

You can see in the fly status output that the app with two Machines has a standby Machine. This standby will never start unless the first Machine fails for some reason, so those two Machines should never be running at the same time.

It’s strange that your first app doesn’t have a standby also. This can happen if you scale down to 1 machine: if you do that then the standby will be deleted.

I’m not sure about the rolling deployment strategy in this case or whether you need to set something on the individual Machines beyond the defaults, but I’ll see what I can find out.

5

Alright, more info would be nice.

Regarding scaling down,no - I did not scale down to 1 machine with the cli tool.

And your answer leads me to another question :slight_smile: :
What happens in either of the two scenarios (1 machine / 1 machine+1 standby), when the app should crash. Is it also automatically restarted if there’s only one machine? If yes, then for me it seems I would only need the one running machine.

6

OK. So I think you should set the Machine restart policy to always for these Machines (based on this info: https://fly.io/docs/apps/migrate-to-v2/#make-sure-the-machines-restart-policy-suits-your-app).

You would set the restart policy with:

fly m update --restart always <machine_id>

(run in the each app’s directory, per Machine, or you can specify the app with -a)

You can then verify that it’s set with this command (which outputs the Machine config in json format):

fly m status <machine-id> -d

You should see:

...
 "restart": {
    "policy": "always"
  },
...

By default restart seems to be empty and the default behaviour is to attempt restart 10 times (unless the host reboots, in which case a Machine with no services will stay stopped). You want it to restart no matter what though, so always is better.

As far as the standby Machines go. They are a kind of "insurance: if there’s a hardware failure. They won’t ever start unless needed. It’s up to you if you want to keep them or not.

If you want them, then you should be able to get one back in your first app by scaling down to zero Machines and then running fly deploy.

If you don’t want the standby, then you can just destroy it with fly m destroy <machine-id>.

And your answer leads me to another question :slight_smile: :
What happens in either of the two scenarios (1 machine / 1 machine+1 standby), when the app should crash. Is it also automatically restarted if there’s only one machine? If yes, then for me it seems I would only need the one running machine.

I hope this is answered by the info about restart policy and the behaviour of standby Machines above. :slight_smile:

7

The rolling deployment strategy won’t affect this setup since the standbys are never started. When you deploy, there will only be the one, non-standby, Machine to start up.

8

Great, yes that answers all my current questions.
Already tried the commands successfully.

Thank you :slight_smile:

1 Like
9

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.