tl;dr a process entering iowait on a shared-cpu machine instance appears to effectively peg it at 100% “busy” (depleting the burst balance and preventing the balance from being replenished).
I have been testing Elide on both shared and performancemachine instances and I think I have encountered a problem with how the burst/balance logic interacts with iowait.
Elide uses ublk which is built on io_uring internally, placing processes in iowait.
Minimal repro here - GitHub - soulware/iowait-repro: Minimal repro: io_uring completion waits booked as iowait throttle Fly.io shared-cpu machines · GitHub
2 stages to the test -
- “poll” mode where things behave as expected. The process on the machine instance is not doing much (approx 0.5s of “work” every 60s). This remains well within the 6.25% cpu shared-cpu instance. And the burst balance slowly increases.
- “enter” mode (iowait). As soon as the process enters iowait we see significantly worse behaviour. The burst balance is quickly depleted and prevented from being replenished. And we see both steal and throttling occurring. This is the same 0.5s of work per 60s period but from the logs below it is now taking 6-10s each 60s period.
[...]
2026-07-18T13:45:47Z app[8ed43dc77e5338] lhr [info]cpu%: user 0.9 sys 0.0 idle 99.1 iowait 0.0 steal 0.0 | poll-idle n/a | burn 554ms
2026-07-18T13:46:47Z app[8ed43dc77e5338] lhr [info]cpu%: user 1.0 sys 0.2 idle 98.9 iowait 0.0 steal 0.0 | poll-idle n/a | burn 554ms
2026-07-18T13:47:48Z app[8ed43dc77e5338] lhr [info]cpu%: user 0.9 sys 0.0 idle 99.1 iowait 0.0 steal 0.0 | poll-idle n/a | burn 554ms
2026-07-18T13:48:48Z app[8ed43dc77e5338] lhr [info]cpu%: user 0.9 sys 0.0 idle 99.0 iowait 0.0 steal 0.0 | poll-idle n/a | burn 559ms
2026-07-18T13:49:49Z app[8ed43dc77e5338] lhr [info]cpu%: user 1.0 sys 0.1 idle 98.8 iowait 0.0 steal 0.0 | poll-idle n/a | burn 556ms
2026-07-18T13:50:49Z app[8ed43dc77e5338] lhr [info]cpu%: user 0.9 sys 0.0 idle 99.0 iowait 0.0 steal 0.0 | poll-idle n/a | burn 558ms
2026-07-18T13:51:50Z app[8ed43dc77e5338] lhr [info]cpu%: user 0.9 sys 0.0 idle 99.1 iowait 0.0 steal 0.0 | poll-idle n/a | burn 556ms
2026-07-18T13:52:50Z app[8ed43dc77e5338] lhr [info]cpu%: user 1.0 sys 0.0 idle 99.0 iowait 0.0 steal 0.0 | poll-idle n/a | burn 559ms
2026-07-18T13:53:51Z app[8ed43dc77e5338] lhr [info]cpu%: user 1.0 sys 0.1 idle 98.9 iowait 0.0 steal 0.0 | poll-idle n/a | burn 556ms
2026-07-18T13:54:52Z app[8ed43dc77e5338] lhr [info]cpu%: user 0.9 sys 0.0 idle 99.1 iowait 0.0 steal 0.0 | poll-idle n/a | burn 554ms
2026-07-18T13:55:52Z app[8ed43dc77e5338] lhr [info]cpu%: user 0.9 sys 0.0 idle 99.1 iowait 0.0 steal 0.0 | poll-idle n/a | burn 554ms
2026-07-18T13:56:39Z app[8ed43dc77e5338] lhr [info]switch: wait mode poll -> enter (same process, request still in flight)
2026-07-18T13:56:53Z app[8ed43dc77e5338] lhr [info]cpu%: user 1.0 sys 0.1 idle 76.9 iowait 21.9 steal 0.0 | poll-idle n/a | burn 556ms
2026-07-18T13:57:53Z app[8ed43dc77e5338] lhr [info]cpu%: user 1.0 sys 0.1 idle 0.0 iowait 98.9 steal 0.0 | poll-idle n/a | burn 560ms
2026-07-18T13:58:59Z app[8ed43dc77e5338] lhr [info]cpu%: user 0.8 sys 0.0 idle 0.0 iowait 90.6 steal 8.6 | poll-idle n/a | burn 6227ms
2026-07-18T14:00:06Z app[8ed43dc77e5338] lhr [info]cpu%: user 0.7 sys 0.0 idle 0.0 iowait 90.6 steal 8.7 | poll-idle n/a | burn 6326ms
2026-07-18T14:01:14Z app[8ed43dc77e5338] lhr [info]cpu%: user 1.2 sys 0.1 idle 0.0 iowait 87.0 steal 11.6 | poll-idle n/a | burn 8551ms
2026-07-18T14:02:21Z app[8ed43dc77e5338] lhr [info]cpu%: user 0.8 sys 0.0 idle 0.0 iowait 90.6 steal 8.6 | poll-idle n/a | burn 6240ms
2026-07-18T14:03:27Z app[8ed43dc77e5338] lhr [info]cpu%: user 0.7 sys 0.0 idle 0.0 iowait 90.6 steal 8.7 | poll-idle n/a | burn 6324ms
2026-07-18T14:04:36Z app[8ed43dc77e5338] lhr [info]cpu%: user 1.4 sys 0.1 idle 0.0 iowait 86.0 steal 12.5 | poll-idle n/a | burn 9190ms
2026-07-18T14:05:42Z app[8ed43dc77e5338] lhr [info]cpu%: user 0.8 sys 0.0 idle 0.0 iowait 90.6 steal 8.6 | poll-idle n/a | burn 6168ms
2026-07-18T14:06:49Z app[8ed43dc77e5338] lhr [info]cpu%: user 0.7 sys 0.0 idle 0.0 iowait 90.4 steal 8.9 | poll-idle n/a | burn 6241ms
2026-07-18T14:07:58Z app[8ed43dc77e5338] lhr [info]cpu%: user 1.4 sys 0.2 idle 0.0 iowait 86.1 steal 12.3 | poll-idle n/a | burn 9116ms
[...]
It looks like the burst logic is treating iowait as “busy”. In the grafana charts this is just noise. But with the burst balance affected by this it is having a significant impact on the overall performance (and latency) of this machine instance, so it is not just a monitoring/reporting issue.
I am assuming this behaviour will be present on any shared-cpu machine instance where a process enters iowait(for any reason). This is not specific to io_uring or ublk - it is just very visible with ublk/io_uring due to how iowait is used by io_uring.
