fewer started machines than min_machines_running?

I have an app that I’ve scaled with fly scale count to 8 machines and when I looked at my dashboard it only showed 1 started machines and 7 stopped machines. But the app was showing signs of getting overloaded so I tried redeploying with:

  auto_stop_machines = true
  auto_start_machines = true
  min_machines_running = 4

And after that there were 3 started machines and 5 stopped machines. I expected there would be at least 4 started machines. Am I misunderstanding something?

Also, is there a way to update a deploy so as to get new values from fly.toml without having to rebuild the image? Or is that baked into the Docker image somehow?

hi @peterseibel

Were all your Machines in your primary region? min_machines_running only applies in the primary region. If yes, then you should have seen 4 Machines started, not 3.

If you try again and the same thing happens, you could have a look at the logs to see what the “downscale” message says about the number of Machines. And let us know if it’s still not aligning with your config in fly.toml.

You should be able to deploy without building the image by specifying the already-built image to use. You can get the image when you run fly status. If it’s a public image, just copy the image name, but if it’s stored in the Fly.io registry (it has the format my-app-name:deployment-xxxxxxx), then you need to append registry.fly.io/ to the front of the image name. Run fly deploy like this:

fly deploy --image registry.fly.io/my-app-name:deployment-xxxxxxx
fly deploy --image publicimage:version

Hmmm. I just redeployed with this config:

# fly.toml app configuration file generated for jobe on 2023-09-09T17:49:05-07:00
# See https://fly.io/docs/reference/configuration/ for information about how to use this file.

app = "jobe"
primary_region = "sjc"


  internal_port = 80
  force_https = true
  auto_stop_machines = true
  auto_start_machines = true
  min_machines_running = 4
  processes = ["app"]

But it seems that there’s only one machine running:

jobe (main) $ fly status
  Name     = jobe                                        
  Owner    = personal                                    
  Hostname = jobe.fly.dev                                
  Image    = jobe:deployment-01HHQ5PXX2YAQBH3ARWH0VSEY8  
  Platform = machines                                    

app    	148e534eb36089	17     	sjc   	started	    	      	2023-12-15T16:42:55Z	
app    	17811093f27089	17     	sjc   	stopped	    	      	2023-12-15T16:43:11Z	
app    	3287312b10d185	17     	sjc   	stopped	    	      	2023-12-15T16:43:10Z	
app    	6e82d2e3a55928	17     	sjc   	stopped	    	      	2023-12-15T16:43:12Z	
app    	6e82d396a36608	17     	sjc   	stopped	    	      	2023-12-15T16:43:12Z	
app    	9080e452b25518	17     	sjc   	stopped	    	      	2023-12-15T16:43:10Z	
app    	e286000c557286	17     	sjc   	stopped	    	      	2023-12-15T16:43:11Z	
app    	e286001b067186	17     	sjc   	stopped	    	      	2023-12-15T16:43:11Z	

Where are the logs where I’d see the downscale message? I don’t see anyting obvious on the dashboard that links to logs.

You can view logs under Monitoring in the dashboard.

A better option could be to open a second terminal and run fly logs -a <app name>, and run fly deploy again in the first terminal window to watch the logs live.

All the Machines should start up when you deploy and then gradually get stopped (1 every few minutes) until you reach 4 Machines. But it looks like those Machines are not starting at all.

You could also try manually starting one of the stopped Machines to see if you get an error:
fly machine start <machine ID>

Q: Is your app working normally with the 1 machine? or are you seeing issues?

It mostly works fine with one machine. This app is used for hosting a code runner for my high-school CS class and when kids are taking a test it gets a lot more load but I’m not sure I’ve set things up right so fly is recognizing when one machine is overloaded so I was just trying to make sure I had several running during a test. Later I’ll look into how to set up a better health check so fly can help me scale automatically.

I’ve clicked the start button on the web dashboard to start up some more machines and that seems to have done the trick for now but I am puzzled why immediately after the deploy there aren’t four started.

The stopped machines should start when the app gets more requests (docs) based on concurrency settings in the fly.toml. You could try playing around with setting the soft_limit lower to get more Machines to start during busy times.

Is there a way to see what the default values for configuration options like soft_limit are?

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.