App: postgres-hose-2025, region sin. Primary machine e825411f01e998, replica e82de15a213148 stayed healthy throughout.
Primary’s pg/role checks flapped 0/3→3/3 in a loop for a while. Logs showed sorry, too many clients already and repmgr losing/regaining contact with the local node every 30-60s. Ran top on the primary during a flapping episode and saw 89% CPU steal time (%st), load average 30+ on a 2-vCPU box, while individual Postgres backends were each only using 3-7% CPU.
Killed stale idle connections and did a fly machine restart — no lasting change. Then resized the VM from shared-cpu-2x to performance-2x, and shortly after resized it back down to shared-cpu-2x again — it’s been stable since. On our Grafana dashboard, four panels all show a clean step-change right around that point: the CPU Utilization panel’s throttling indicator drops to zero, 5min Load Avg drops from a flat ~1100-1600% down to ~0% and stays there, memory usage drops to a lower steady baseline, and App Concurrency goes from violent 0↔90 thrashing to a calmer pattern.
Update: it recurred roughly 40 minutes later, same signature (throttling band back on the CPU panel, load avg spiking to 1000-1500%), on the same machine, still on shared-cpu-2x — so whatever the resize did wasn’t a lasting fix.
Questions:
What typically causes CPU steal time this high (89%) and sustained throttling on shared-cpu-2x?
What does a vm-size change actually do to a machine’s placement — restart in place, or reschedule onto different hardware?
Any recommendations for what to check or do if this happens again?Would appreciate input from Fly staff if anyone’s around — happy to share more logs/timestamps.
This is typical of CPU throttling. You can see the burst balance being slowly consumed over the course of ~30 minutes, in the “CPU Quota Balance and Throttling” graph, and then you’re back into steal.
The smaller shared-class CPUs are underpowered for serious Postgres, in my opinion. A shared-2x only has ~12% of a CPU on a sustained basis, i.e., without the initial burst balance, and that 12% has to accommodate not only the Postgres server processes themselves but also the replication daemon, the swap daemon, the guest Linux kernel, etc.
Possibly you’d be ok with shared-4x, but this might be a good time to consider (/ reconsider) Managed Postgres…
Usually it’s in place, but you can also be migrated, I believe. (It’s seems like Machines with smaller volumes can be auto-migrated now.)
You can determine which physical host machine you’re on by looking at a substring of the 6PN address:
@mayailurus’s reply was spot-on: the cause here is your workload being too much for shared CPUs, as explained on the CPU throttling page. The usual remedies: 1- reduce # of queries you make (may be unfeasible), 2- optimize your queries so they don’t eat up so much CPU, or 3- move to a machine size with more CPU capacity (shared-4x or performance-1x - performance machines are not throttled at all).
Worth noting that Managed Postgres also runs on shared machines if you select the “Shared CPU” plans so it won’t solve the issue in and of itself. The main difference is that your cluster would alert us and we’d tell you that you need to either reduce/optimize your query load, or move to an MPG plan with Performance CPUs.
