Auto-stop cordons then uncordons my machine every ~6 min but never stops it

Questions / Help
, ,
1

Hi all,

My API app’s only public machine (process group app, region fra) often won’t auto-stop when idle. Every ~6 minutes the proxy cordons it, and ~15–23s later flyd uncordons it. A stop event never happens, and the machine keeps running for hours with zero traffic. Last week it was up about 120 of 168 hours, with single runs of 16h, 18h and 38h. Sometimes it does stop normally ~8 min after going idle, so it’s intermittent.

What I’ve already ruled out:

  • Traffic: an 8-minute fly logs tail showed only health-check requests, all 200, including during each cordon window.
  • Open connections: a check inside the machine found no connections established to port 8901. - Leases: the Machines API returns lease not found.
  • Machine state: it’s started and host_status is ok.

Questions:

  1. What makes the proxy cancel an auto-stop after cordoning it? 2. Is there a way to see that reason from my side (logs, metrics, API)? 3. Could the volume, the extra process groups, or a proxy-side connection count I can’t see cause this?
    Thanks!
5

Hi, this could be an issue with the host where the machine is. Can you try cloning the machine (should land the replacement on another host) and then destroying the original once you’ve confirmed the new machine is auto-stopping properly?

6

Same here, also fra, on two machines (d895791c21e448, shared-cpu-2x 4 GB and
2872464ced9758, shared-cpu-1x 1 GB), both auto_stop_machines = “suspend”.

Identical pattern: proxy cordon, flyd uncordon ~25 s later, no suspension event,
nothing logged, zero traffic. Both machines get stuck on the same days and
recover on the same days. Hours awake per day from
fly_instance_memory_mem_total (UTC), for machines used well under an hour a day:

        4 GB     1 GB

Sep 25 24.0 24.0
Sep 26 24.0 22.7
Sep 27 24.0 20.9
Sep 28 2.8 2.1
Sep 30 24.0 24.0
Oct 01 24.0 20.7
Oct 03 24.0 24.0
Oct 04 22.8 24.0

A manual fly machine suspend works in 5 s and resumes fine, after which the
machine goes straight back to the cordon/uncordon cycle.

Two things that support the “it’s the host” theory:

  • 2026-10-05 15:51 UTC the 4 GB machine failed to start on a request
    (“machine can’t start at the moment” x3, then “could not find a good candidate
    within 40 attempts at load balancing”) and cold-booted instead of resuming.
  • The same evening another 4 GB machine of mine in fra refused a config update
    with “could not reserve resource for machine: insufficient memory available to
    fulfill request on the current host” (Request ID:
    01M46R4TJAWPKS1FF1279QAFSV-fra).

@Vladescu, did moving to another host fix it for you?

Fly: both machines have volumes, so I’ll fork the volumes to new hosts and
report back. Can the extra uptime on the stuck days be looked at by billing?

7

Forking the volume to a new zone and cloning the machine seems to have worked. Auto-suspend did restore immediately on both machines.

8

Thank you for the assistance. However, it solved by itself right before investigating further.