Stopped Machine with autostart: false (and cordoned) is started by fly-proxy

General
, ,
1

We pause a customer’s service by stopping its Machine and turning off autostart in the Machine’s service config. For a Machine that was deployed (created) a short time before the pause, fly-proxy still starts the stopped Machine when a request arrives. It does so even though:

  • the Machine reads autostart: false
  • in some trials it also reads cordoned: true

The Machine’s own event log names the proxy as the starter: type: start, status: starting, source: proxy, about one second after the request. No Machines API call of ours falls in that window. With no request, the Machine stays stopped.

The behaviour lasts for a while after the pause, then stops:

Request sent after the pause Machine started by the proxy
~9 s nearly every time
60 s 3 of 5 with autostart off; 4 of 5 with autostart off and a cordon
2 minutes 0 of 5
5 minutes 0 of 4
10 minutes 0 of 5
20 minutes 0 of 5

On Machines that were not just deployed, the same settings are honoured:

  • a stopped Machine with autostart: false, or cordoned, got 503 and stayed stopped;
  • a cordon on a Machine that had been running for a while was honoured within seconds, and requests 60 s and 120 s after its stop got 503;
  • on bare Machines (fra and iad, a fresh app each), a cordon survived a stop, a start and a config update, and a cordoned stopped Machine was not started even with autostart: true

The sequence

  1. Create a Machine (a deploy, launched by the create). It passes its checks and serves 2 requests.
  2. Within about 30 s: POST /machines/{id}/stop (SIGTERM, grace), then wait for stopped
  3. POST /machines/{id} with the same config, the service’s autostart: false, autostop: false, min_machines_running: 0 and skip_launch: true. The Machine reads back stopped, autostart: false
  4. In some trials, also POST /machines/{id}/cordon. The Machine reads back cordoned: true
  5. One public request to https://.fly.dev/, 9 s to 60 s later.

Result: 200. The Machine is started. Its newest events are start/starting/proxy then start/started/flyd, and its config still reads autostart: false (and cordoned: true where applied).

Variants that were also started: the cordon applied while the Machine was still running, before the stop (5 of 5); and stop + cordon with no config update at all (5 of 5).

Requests that started a stopped Machine

Each row is one request. The timestamp is the proxy’s start/starting event from the Machine’s event log, in UTC. The apps have since been deleted.

fly-request-id app machine request after pause machine config at the request proxy start event (UTC)
01M3JHP34DY0JPQPW985GQN3H8-fra lmprobe-lm-wake-d009-2x7whk 83d1351a67dd28 9 s state=stopped, autostart=false, cordoned=true (autostart off + cordon) start/starting/proxy 2026-09-27 23:02:30.814
01M3JHXGV63XP5G553MG95Q7DA-fra lmprobe-lm-wake-d009-2x7whk 829e17f7739638 9 s state=stopped, autostart=false, cordoned=true (autostart off + cordon) start/starting/proxy 2026-09-27 23:06:33.759
01M3JJ45YRCF92PZX9M76V3KQ9-fra lmprobe-lm-wake-d009-2x7whk 80e9039c009548 9 s state=stopped, autostart=false, cordoned=true (autostart off + cordon) start/starting/proxy 2026-09-27 23:10:12.536
01M3JJBZ5GJVZMS9P249J5MH77-fra lmprobe-lm-wake-d009-2x7whk 80d969f65511d8 9 s state=stopped, autostart=false, cordoned=true (autostart off + cordon) start/starting/proxy 2026-09-27 23:14:27.171
01M3JJJNMZYC4ME409HAGDQT7C-fra lmprobe-lm-wake-d009-2x7whk 683e12ef23e928 9 s state=stopped, autostart=false, cordoned=true (autostart off + cordon) start/starting/proxy 2026-09-27 23:18:08.210
01M3KDNJC3CS83JG2549MX0PEM-fra aetherfy-lm-brand-d54e-ghfxwc 80d225b6597458 9 s state=stopped, autostart=false, cordoned=true (autostart off + cordon) start/starting/proxy 2026-09-28 07:11:35.140
01M3KDV0CASMANAE610YG76H7Y-fra aetherfy-lm-brand-d54e-ghfxwc 2870231c077628 9 s state=stopped, autostart=false, cordoned=true (autostart off + cordon) start/starting/proxy 2026-09-28 07:14:31.565
01M3KE3MXDGQ5XEPHHZVCX4AS3-fra lmprobe-lm-wake-d009-2x7whk 78111120f56948 9 s state=stopped, autostart=false, cordoned=true (cordon while running, then stop, then autostart off) start/starting/proxy 2026-09-28 07:19:14.657
01M3KE8SG71JNXZM8DYSJYTSA6-fra lmprobe-lm-wake-d009-2x7whk 863662ce726998 9 s state=stopped, autostart=false, cordoned=true (cordon while running, then stop, then autostart off) start/starting/proxy 2026-09-28 07:22:25.624
01M3KEEVDT959WV6J8FRS9Z7JG-fra lmprobe-lm-wake-d009-2x7whk 894dd5f64d2e78 9 s state=stopped, autostart=false, cordoned=true (cordon while running, then stop, then autostart off) start/starting/proxy 2026-09-28 07:25:25.331
01M3KEVVQHBXRCGRRKW2K3FCTT-fra lmprobe-lm-wake-d009-2x7whk 811d692bddd468 9 s state=stopped, autostart=false, cordoned=true (cordon while running, then stop, then autostart off) start/starting/proxy 2026-09-28 07:32:28.064
01M3KF2Q3HMMHVJGNBX0PX19Q0-fra lmprobe-lm-wake-d009-2x7whk 8530d0a4195528 9 s state=stopped, autostart=true, cordoned=true (stop, then cordon (autostart left on)) start/starting/proxy 2026-09-28 07:36:12.866
01M3KF7YE38893PDKJ0S636H9X-fra lmprobe-lm-wake-d009-2x7whk 830375a7e61708 9 s state=stopped, autostart=true, cordoned=true (stop, then cordon (autostart left on)) start/starting/proxy 2026-09-28 07:39:04.053
01M3KFD1N1A8B65WB0R8XJHQZE-fra lmprobe-lm-wake-d009-2x7whk 8e145eb10746e8 9 s state=stopped, autostart=true, cordoned=true (stop, then cordon (autostart left on)) start/starting/proxy 2026-09-28 07:41:51.187
01M3KFK9QF5FTYYVM61ZRE14YM-fra lmprobe-lm-wake-d009-2x7whk 8576656c3770d8 9 s state=stopped, autostart=true, cordoned=true (stop, then cordon (autostart left on)) start/starting/proxy 2026-09-28 07:45:16.235
01M3KFRKAJAPSYH249GJMBA5CJ-fra lmprobe-lm-wake-d009-2x7whk 08045e7b7e9908 9 s state=stopped, autostart=true, cordoned=true (stop, then cordon (autostart left on)) start/starting/proxy 2026-09-28 07:48:09.755
01M3KHCXP0G2CRGQX0X3NV25MN-fra lmprobe-lm-wake-d009-2x7whk 82d1131ade3708 60 s state=stopped, autostart=false, cordoned=true (autostart off + cordon) start/starting/proxy 2026-09-28 08:16:44.424
01M3KHM0B2DPKHS4FPA1BS552D-fra lmprobe-lm-wake-d009-2x7whk d8de391c655de8 60 s state=stopped, autostart=false, cordoned=true (autostart off + cordon) start/starting/proxy 2026-09-28 08:20:56.930
01M3KHX8KPBWY42A0QTMWH93CR-fra lmprobe-lm-wake-d009-2x7whk d8d99907c24778 60 s state=stopped, autostart=false, cordoned=true (autostart off + cordon) start/starting/proxy 2026-09-28 08:25:39.750
01M3KJ40CZZ7EQFQM1DCMQHM9H-fra lmprobe-lm-wake-d009-2x7whk 84ed410cd20498 60 s state=stopped, autostart=false, cordoned=true (autostart off + cordon) start/starting/proxy 2026-09-28 08:29:20.895
01M3KJK2N2B319Q213AK3065V9-fra lmprobe-lm-wake-d009-2x7whk d8d99909f24e48 60 s state=stopped, autostart=false, cordoned=false (autostart off) start/starting/proxy 2026-09-28 08:37:38.445
01M3KK26386BGBY0FZ9WQDEHX1-fra lmprobe-lm-wake-d009-2x7whk 869292ce239078 60 s state=stopped, autostart=false, cordoned=false (autostart off) start/starting/proxy 2026-09-28 08:45:49.591
01M3KK91J9ZXJ2SJYSFVPNM7YZ-fra lmprobe-lm-wake-d009-2x7whk 897961b4541528 60 s state=stopped, autostart=false, cordoned=false (autostart off) start/starting/proxy 2026-09-28 08:49:34.781

Our questions

  1. Is fly-proxy expected to start a stopped Machine whose service has autostart: false, and one that is cordoned, shortly after it was created or launched?
  2. Which state does the proxy consult, and how long does a change to autostart or to the cordon take to reach it for a recently created Machine?
  3. Is there a supported way to make a stopped Machine unstartable by traffic immediately, short of destroying it?

Until then, our platform detects a paused Machine the proxy started, stops it again, and does not bill the customer for that time.

3

Hi… This older thread mentions a third thing that could be tried:

Having said that, the Fly.io platform is eventually consistent in general, meaning that almost all operations do take some time to propagate all the way out. So, if your design is conceived in terms of “immediately” then there’s probably going to be mismatches, of one kind or another…

4

What is auto start set to for your machines on create? If it is true, then as @mayailurus mentioned above, it is expected that there may be delays applying that configuration. In general, it is also hard to enforce that this takes effect immediately since there could be a request attempting autostart at the same moment you update the config. Service configs are also not designed to be updated frequently as part of normal operation.

Since you’re doing this for customers, I wonder if it would be better to make use of fly-replays from a central “router” app, like described here. This gives you better control over when requests are allowed or not without having to update machines regularly. You could offload some replays to fly-proxy via fly-replay-cache, which does bring back this problem technically but you have more control over how much time is allowed between stopping the machine and the last request in that case.

5

Thanks, both. Some context from our side:

  • Autostart on create: yes, service machines are created with autostart: true. Pausing stops the machine, then pushes autostart: false with skip_launch, and cordons it. We don’t change configs often; this only happens when a user pauses or resumes.
  • Cordon: we measured it. On a machine that has been running for a while, a cordon holds. Within about a minute of a deploy it doesn’t. The machine reads back stopped, autostart: false, cordoned: true, and a request 9 s later starts it (source: proxy). At 2 minutes and later we saw 19 of 19 hold. So it looks like the proxy’s copy of the machine state lags for about two minutes after a deploy, which fits what you say about eventual consistency. We re-stop the machine when this happens, so it isn’t blocking us.
  • Leases: we haven’t tried these and will test them in the same window. Two questions: is there a maximum ttl? And does the proxy check leases from fresher state than cordons, or would a lease taken right after a deploy have the same lag?
  • fly-replay: our router is a Cloudflare worker, and the requests that woke the machine came through *.fly.dev as well, so a Fly-side router wouldn’t cover those.
6

The docs say that the lease gives you exclusive control of the Machine, and that’s the usual meaning of the term within a distributed systems context.

I’d still try to verify it with experiments, though, and it would be wise to keep your existing fallback, just in case.

Also, like @PeterCxy was saying, you should be prepared for other things to be going on while lease acquisition is in progress (i.e., requested but not yet granted).

The subsection linked to above shows that parameter as being optional, which implies that there is none (barring errors in the docs).

Still, holding a lease for a really long time is probably outside the originally intended usage; it seems better to use the leased interval to apply a cordon and allow it to propagate, etc.

7

Thanks. Measured: a lease taken by the pause itself (ttl 300, never renewed, the cordon applied inside it) held the machine 10 of 10 through a deploy, versus 7 of 7 woken with the cordon alone. We take the lease before our own config push, so the “requested but not yet granted” window no longer overlaps our other calls, and the re-stop fallback stays. No maximum ttl was hit (86400 accepted).

8

Leases should always be up to date and shouldn’t have the same delay. They’re validated when the machine actually gets stared/stopped.

9

Thanks, that explains it: the lease is checked at the actual start, while autostart and the cordon are read from a replicated copy that lags after a deploy. Our pause now leases before anything else, and it held 10 of 10.