Deployment failing with cpu_util_exceeds_max during rolling update

Questions / Help
1

Hi Guys.

I have not been able to deploy for sometime, I wait a while then try again but issue is persistent, at least for the past 40mins or so.

Updating existing machines in ‘[APP_NAME]’ with rolling strategy

Failed to update machines: failed to update machine [MACHINE_ID]:
failed to update VM [MACHINE_ID]:
internal: could not reserve resource for machine:
governor policy blocked start:
hard gate failed: cpu_util_exceeds_max
(CPU utilization exceeds max + buffer for existing)

Retrying…

Error: failed to update machine [MACHINE_ID]:
internal: could not reserve resource for machine:
governor policy blocked start:
hard gate failed: cpu_util_exceeds_max
(CPU utilization exceeds max + buffer for existing)

Region: syd

2

Hi… Are you using volumes, by any chance?

(Those can pin you to a single, congested physical host machine.)

3

Yes, exactly — we have a 1GB volume (data) mounted at /data that’s pinned to a specific host in zone 6cfa (syd region). The host appears to be overloaded, and Fly.io’s governor keeps blocking any machine from starting there (cpu_util_exceeds_max).

We’ve already tried:

  • fly deploy (rolling strategy) — blocked

  • fly deploy --strategy immediate — blocked

  • fly machine stop then deploy — still blocked

  • fly machine destroy --force then deploy — fails because the volume can only attach to the same congested host

Our plan is to destroy the volume and create a fresh one, which should land on a different, less congested host. The volume holds a SQLite database (Prisma) for session data — the app’s main data lives in Shopify metafields, so we’d just lose user sessions (they’d need to re-authenticate).

Is destroying and recreating the volume the right move here, or is there a better way to detach from the congested host without losing the volume?

4

Volume forking is usually the best route:

https://fly.io/docs/volumes/volume-manage/#create-a-copy-of-a-volume-fork-a-volume

You can do that across regions, which may also be needed here.

(Australia has been kind of congestion-prone lately…)

6

Thanks for the suggestion — volume forking would have been the ideal approach. Unfortunately we discovered the issue after the machine was already stuck (governor blocking all starts on the congested host), so we couldn’t SSH in to fork the volume before destroying it.

Lesson learned for next time: fork first, then destroy.

We’ve since destroyed the old volume, deployed to Singapore (sin) temporarily, created a fresh volume there, and redeployed. The app is back up. The DATABASE_URL was set as a Fly secret pointing to file:/data/sqlite.db, so the volume was indeed holding the SQLite database — we lost sessions and install tracking data, but the real app data lives in Shopify metafields and Supabase, so impact was minimal.

We’ll move back to Sydney once capacity recovers. Good to know about the congestion issues in Australia — we’ll keep a multi-region fallback in mind going forward.