Machine created via API is not suspending

, ,
1

Hi :waving_hand:

I’m using the machines api to deploy my application. Everything is working fine except for the application suspension. I’m using the following endpoint: https://api.machines.dev/v1/apps/{{ _['app-name'] }}/machines

This is the body of the request

{
    "region": "{{ _.region }}",
    "mounts": [
    {
        "source": "ubuntu_home",
        "destination": "/root"
    }],
    "config":
    {
        "memory_gb": 2048,
        "image": "registry.fly.io/my-image:5.3.1",
        "env":
        {
            "USER_NAME": "{{ _['app-name'] }}",
            "USER_PASSWORD": "{{ _['app-secret'] }}"
        },
        "guest":
        {
            "memory_mb": 2048,
            "cpus": 2,
            "cpu_kind": "shared"
        },
        "services": [
        {
            "protocol": "tcp",
            "internal_port": 8080,
            "auto_stop_machines": "suspend",
            "auto_start_machines": true,
            "min_machines_running": 0,
            "max_machines_running": 1,
            "force_https": false,
            "ports": [
            {
                "port": 80,
                "handlers": ["http"]
            },
            {
                "port": 443,
                "handlers": ["tls", "http"]
            }]
        }],
        "checks":
        {
            "healthcheck":
            {
                "type": "http",
                "protocol": "http",
                "method": "GET",
                "path": "/",
                "interval": "15s",
                "timeout": "10s",
                "grace_period": "40s",
                "port": 8080
            }
        }
    }
}

I’ve tried with / without the health check but that doesn’t make a difference. Any guidance would be appreciated!

2

Hi… This key doesn’t exist, :taruggiz_spiral:; it’s just autostop, with no underscores and no _machines suffix, when using the API.

(There are various subtle discrepancies between that and fly.toml, unfortunately.)

Hope this helps!

Aside: max_machines_running seems suspect, too.

https://fly.io/docs/launch/autostop-autostart/#maximum-number-of-machines-running

A couple quick tips…

command purpose
fly m destroy remove unwanted Machines
fly deploy --ha=false avoid making two at the outset
4

thanks @mayailurus that has helped and when I make the API request I’m now seeing autostop being returned. The bad news though is that the app is still not suspending. Here is my updated request and response

{
    "region": "{{ _.region }}",
    "mounts": [
    {
        "source": "ubuntu_home",
        "destination": "/root"
    }],
    "config":
    {
        "memory_gb": 2048,
        "image": "registry.fly.io/image:5.3.1",
				"metadata": {
					"fly_process_group": "app"
				},
        "env":
        {
            "USER_NAME": "{{ _['app-name'] }}",
            "USER_PASSWORD": "*****"
        },
        "guest":
        {
            "memory_mb": 2048,
            "cpus": 2,
            "cpu_kind": "shared"
        },
        "services": [
        {
						"autostop": true,
						"autostart": true,
						"min_machines_running": 0,
					
            "protocol": "tcp",
            "internal_port": 8080,
            "ports": [
            {
                "port": 80,
                "handlers": ["http"]
            },
            {
                "port": 443,
                "handlers": ["tls", "http"]
            }]
        }],
        "checks":
        {
            "healthcheck":
            {
                "type": "http",
                "protocol": "http",
                "method": "GET",
                "path": "/",
                "interval": "15s",
                "timeout": "10s",
                "grace_period": "40s",
                "port": 8080
            }
        }
    }
}

Response

{
	"id": "48e3556f722218",
	"name": "weathered-sunset-2867",
	"state": "created",
	"region": "ams",
	"instance_id": "01JT8EAZBRHRYXTGHJGP9W49GP",
	"private_ip": "fdaa:17:1f72:a7b:3bb:4243:a685:2",
	"config": {
		"env": {
			"USER_NAME": "*********",
			"USER_PASSWORD": "***************"
		},
		"init": {},
		"guest": {
			"cpu_kind": "shared",
			"cpus": 2,
			"memory_mb": 2048
		},
		"metadata": {
			"fly_process_group": "app"
		},
		"services": [
			{
				"protocol": "tcp",
				"internal_port": 8080,
				"autostop": true,
				"autostart": true,
				"min_machines_running": 0,
				"ports": [
					{
						"port": 80,
						"handlers": [
							"http"
						]
					},
					{
						"port": 443,
						"handlers": [
							"tls",
							"http"
						]
					}
				],
				"force_instance_key": null
			}
		],
		"checks": {
			"healthcheck": {
				"port": 8080,
				"type": "http",
				"interval": "15s",
				"timeout": "10s",
				"grace_period": "40s",
				"method": "GET",
				"path": "/",
				"protocol": "http"
			}
		},
		"image": "registry.fly.io/unicenta-web:5.3.1",
		"restart": {
			"policy": "on-failure",
			"max_retries": 10
		}
	},
	"incomplete_config": null,
	"image_ref": {
		"registry": "registry.fly.io",
		"repository": "unicenta-web",
		"tag": "5.3.1",
		"digest": "sha256:9a58378236cb6e20a82898ae8630d210f25040106fc4bf89a5bd7bb9f9e685d9",
		"labels": {
			"org.opencontainers.image.ref.name": "ubuntu",
			"org.opencontainers.image.version": "22.04"
		}
	},
	"created_at": "2025-05-02T12:08:01Z",
	"updated_at": "2025-05-02T12:08:01Z",
	"events": [
		{
			"id": "01JT8EAZDSFDV3VERS8S1HVQEK",
			"type": "launch",
			"status": "created",
			"source": "user",
			"timestamp": 1746187681209
		}
	],
	"checks": [
		{
			"name": "healthcheck",
			"status": "warning",
			"output": "the machine hasn't started",
			"updated_at": "2025-05-02T12:08:01.209Z"
		}
	],
	"host_status": "ok"
}
1 Like
5

Try "suspend" instead of true there.

autostop: string or bool (off) - One of off (or false), stop (or true), suspend. […] If suspend, Fly Proxy instead suspends Machines if possible and stops them if not.

(It accepts both booleans and strings, for backward compatibility.)

6

@mayailurus the api accepted suspend but the app is still not suspending. I waited about 10 mins and it remained in the deployed state.

Here is the response

{
	"id": "1853220b4d2968",
	"name": "purple-dust-7293",
	"state": "created",
	"region": "ams",
	"instance_id": "01JT8FQGBDNDP69VQYGR6SVNJW",
	"private_ip": "fdaa:17:1f72:a7b:3ba:d90e:f224:2",
	"config": {
		"env": {
			"USER_NAME": "*******",
			"USER_PASSWORD": "*******"
		},
		"init": {},
		"guest": {
			"cpu_kind": "shared",
			"cpus": 2,
			"memory_mb": 2048
		},
		"metadata": {
			"fly_process_group": "app"
		},
		"services": [
			{
				"protocol": "tcp",
				"internal_port": 8080,
				"autostop": "suspend",
				"autostart": true,
				"min_machines_running": 0,
				"ports": [
					{
						"port": 80,
						"handlers": [
							"http"
						]
					},
					{
						"port": 443,
						"handlers": [
							"tls",
							"http"
						]
					}
				],
				"force_instance_key": null
			}
		],
		"checks": {
			"healthcheck": {
				"port": 8080,
				"type": "http",
				"interval": "15s",
				"timeout": "10s",
				"grace_period": "40s",
				"method": "GET",
				"path": "/",
				"protocol": "http"
			}
		},
		"image": "registry.fly.io/unicenta-web:5.3.1",
		"restart": {
			"policy": "on-failure",
			"max_retries": 10
		}
	},
	"incomplete_config": null,
	"image_ref": {
		"registry": "registry.fly.io",
		"repository": "unicenta-web",
		"tag": "5.3.1",
		"digest": "sha256:9a58378236cb6e20a82898ae8630d210f25040106fc4bf89a5bd7bb9f9e685d9",
		"labels": {
			"org.opencontainers.image.ref.name": "ubuntu",
			"org.opencontainers.image.version": "22.04"
		}
	},
	"created_at": "2025-05-02T12:32:20Z",
	"updated_at": "2025-05-02T12:32:20Z",
	"events": [
		{
			"id": "01JT8FQGCV7F8ZZMG88PAE71K3",
			"type": "launch",
			"status": "created",
			"source": "user",
			"timestamp": 1746189140379
		}
	],
	"checks": [
		{
			"name": "healthcheck",
			"status": "warning",
			"output": "the machine hasn't started",
			"updated_at": "2025-05-02T12:32:20.379Z"
		}
	],
	"host_status": "ok"
}
7

Hm… Try suspending it manually, just to verify that it’s eligible:

https://fly.io/docs/flyctl/machine-suspend/

Also, what do you see in the Machine’s logs (fly logs)?

8

thanks for the help @mayailurus - looks like the app is suspending but restarting after only 3 / 4 mins with a log request? Its not an application request. Here are the logs

2025-05-02 14:13:21.669	
machine started in 524.705313ms
2025-05-02 14:13:21.668	
Machine started in 195ms
2025-05-02 14:13:21.575	
2025-05-02T13:13:21.575898980 [01JT8FQGBDNDP69VQYGR6SVNJW:fc_api] The API server received a Put request on "/logger" with body "{\"log_path\":\"logs.fifo\",\"level\":\"info\"}".


2025-05-02 14:13:21.574	
2025-05-02T13:13:21.574439633 [01JT8FQGBDNDP69VQYGR6SVNJW:main] Running Firecracker v1.7.0
2025-05-02 14:13:21.144	
Starting machine
2025-05-02 14:09:16.353	
Virtual machine has been suspended
9

Hm… These have come up in the forum before, and the jury’s out about whether they’re really what’s waking the Machine up…

(The Fly.io platform unfortunately has a nonzero number of spurious log lines, which complicates debugging.)

I’m pretty inclined to believe this, but it would probably help to go into more detail about what the exact evidence is… For example, is it Grafana? Your own request logging? The Events list in the Machine dashboard? Just the fact that you yourself didn’t make any HTTP requests during that interval?

10

if I deploy the same application i.e. Docker image with the fly cli I don’t get these requests and the app suspends correctly. Here is the fly.toml. Could it be the health check?

  app = 'test-app'
  primary_region = 'ams'
  
  [build]
  
  [[mounts]]
  source = 'ubuntu_home'
  destination = '/root'
  
  [http_service]
  internal_port = 8080
  force_https = true
  auto_stop_machines = 'suspend'
  auto_start_machines = true
  min_machines_running = 0
  max_machines_running = 1
  processes = ['app']
  
  [checks]
  [checks.healthcheck]
  type = 'http'
  interval = '30s'
  timeout = '10s'
  grace_period = '40s'
  method = 'GET'
  path = '/'
  protocol = 'http'
  
  [[vm]]
  memory = '2gb'
  cpu_kind = 'shared'
  cpus = 2
1 Like
11

Interesting… The suspend feature overall is still considered a little experimental, last I heard. (There are definitely enduring problems with clock lag, for example, :snowflake:.)

I would try all of the following (not necessarily cumulatively or in this order)…

  • Use the same Machines API invocation as earlier but with "autostop": "stop"—and see if the sporadic wakeups persist.
  • Double-check that the test application has only one Machine in existence.
  • Add super-verbose request logging—absolutely everything that contacts port 8080—within the application itself.
  • Remove health checks temporarily (like you already mentioned).
  • Compare the fly m status --display-config output of a flyctl-created Machine with that of a (directly) API-made one.

It’s good news that you have at least one case that does work, since now you can start bisecting between them…

1 Like
12

Thank you @mayailurus for you help. I’ll give that a go!

13

hi @mayailurus - so I’ve done a bit more testing. It looks like if you set "autostop": "suspend" fly.io is sending these requests

2025-05-04T08:01:40.569853740 [01JT8YCVACTA6QR8Z1RH9YZG0Z:fc_api] The API server received a Put request on "/logger" with body "{\"log_path\":\"logs.fifo\",\"level\":\"info\"}".

If you use "autostop": true the log request no longer appears but the machine is stopped rather than suspended and the machine takes a lot longer to spin up.

14

Thanks for the additional details… This part is expected, :dove:, by way of avoiding lingering uncertainty, since true is a synonym for stop (and not suspend).

Any luck with super-verbose logging on port 8080? The transport for fc_api is probably VSOCK or a Unix-domain socket (perhaps /.fly/api), and, like @rubys was saying, a causal connection to the unexpected auto-start really shouldn’t be taken for granted.

I made an auto-suspending Machine myself via the API, using a fly tokens create deploy Macaroon as Authorization, and it’s been asleep for 30 minutes straight now. Here is the -d payload:

{
  "config": {
    "image": "registry.fly.io/same-app:<long-string>",
    "services": [
      {
        "internal_port": 8080,
        "autostop": "suspend",
        "autostart": true,
        "ports": [{"port": 80, "handlers": ["http"]}]
      }
    ],
    "restart": {"policy": "no"},
    "guest": {"cpu_kind": "shared", "cpus": 1, "memory_mb": 256}
  }
}

I see the Put request on "/logger" [...] message in fly logs when it does wake up, but that only happens when I explicitly contact the Machine from the outside via HTTP (port 80 on the .flycast address). It doesn’t spontaneously restart, on its own.

16

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.