I have an app that I’ve scaled with fly scale count to 8 machines and when I looked at my dashboard it only showed 1 started machines and 7 stopped machines. But the app was showing signs of getting overloaded so I tried redeploying with:
And after that there were 3 started machines and 5 stopped machines. I expected there would be at least 4 started machines. Am I misunderstanding something?
Also, is there a way to update a deploy so as to get new values from fly.toml without having to rebuild the image? Or is that baked into the Docker image somehow?
Were all your Machines in your primary region? min_machines_running only applies in the primary region. If yes, then you should have seen 4 Machines started, not 3.
If you try again and the same thing happens, you could have a look at the logs to see what the “downscale” message says about the number of Machines. And let us know if it’s still not aligning with your config in fly.toml.
You should be able to deploy without building the image by specifying the already-built image to use. You can get the image when you run fly status. If it’s a public image, just copy the image name, but if it’s stored in the Fly.io registry (it has the format my-app-name:deployment-xxxxxxx), then you need to append registry.fly.io/ to the front of the image name. Run fly deploy like this:
fly deploy --image registry.fly.io/my-app-name:deployment-xxxxxxx
or fly deploy --image publicimage:version
# fly.toml app configuration file generated for jobe on 2023-09-09T17:49:05-07:00
#
# See https://fly.io/docs/reference/configuration/ for information about how to use this file.
#
app = "jobe"
primary_region = "sjc"
[build]
[http_service]
internal_port = 80
force_https = true
auto_stop_machines = true
auto_start_machines = true
min_machines_running = 4
processes = ["app"]
But it seems that there’s only one machine running:
jobe (main) $ fly status
App
Name = jobe
Owner = personal
Hostname = jobe.fly.dev
Image = jobe:deployment-01HHQ5PXX2YAQBH3ARWH0VSEY8
Platform = machines
Machines
PROCESS ID VERSION REGION STATE ROLE CHECKS LAST UPDATED
app 148e534eb36089 17 sjc started 2023-12-15T16:42:55Z
app 17811093f27089 17 sjc stopped 2023-12-15T16:43:11Z
app 3287312b10d185 17 sjc stopped 2023-12-15T16:43:10Z
app 6e82d2e3a55928 17 sjc stopped 2023-12-15T16:43:12Z
app 6e82d396a36608 17 sjc stopped 2023-12-15T16:43:12Z
app 9080e452b25518 17 sjc stopped 2023-12-15T16:43:10Z
app e286000c557286 17 sjc stopped 2023-12-15T16:43:11Z
app e286001b067186 17 sjc stopped 2023-12-15T16:43:11Z
Where are the logs where I’d see the downscale message? I don’t see anyting obvious on the dashboard that links to logs.
You can view logs under Monitoring in the dashboard.
A better option could be to open a second terminal and run fly logs -a <app name>, and run fly deploy again in the first terminal window to watch the logs live.
All the Machines should start up when you deploy and then gradually get stopped (1 every few minutes) until you reach 4 Machines. But it looks like those Machines are not starting at all.
You could also try manually starting one of the stopped Machines to see if you get an error: fly machine start <machine ID>
Q: Is your app working normally with the 1 machine? or are you seeing issues?
It mostly works fine with one machine. This app is used for hosting a code runner for my high-school CS class and when kids are taking a test it gets a lot more load but I’m not sure I’ve set things up right so fly is recognizing when one machine is overloaded so I was just trying to make sure I had several running during a test. Later I’ll look into how to set up a better health check so fly can help me scale automatically.
I’ve clicked the start button on the web dashboard to start up some more machines and that seems to have done the trick for now but I am puzzled why immediately after the deploy there aren’t four started.
The stopped machines should start when the app gets more requests (docs) based on concurrency settings in the fly.toml. You could try playing around with setting the soft_limit lower to get more Machines to start during busy times.