High private-network latency (~55 ms RTT) between two machines in gru

Questions / Help
,
1

Hi! Since around 2026-10-02 23:27 UTC we see unusually high latency on the private network (6PN) between two of our apps, both in gru:

  • parking-garage-cloud, machine d8d1376a004418 (shared-cpu-1x, 1 GB). It was recreated by a deploy at 23:27 UTC.
  • parking-garage-db, machine 7847920cede598 (Postgres flex 17.7, shared-cpu-1x, 1 GB).

Measurements (from inside the app machine, 2026-10-03 ~01:00 UTC):

  • Plain TCP handshake to the DB’s private address (fdaa:78:46e1:a7b:5c5:acc4:83b4:2, port 5433): p50 ≈ 54 ms, min 48 ms, spikes up to ~1 s. The DB → app direction gives the same numbers (symmetric).
  • SELECT 1 ×100 through parking-garage-db.flycast:5432 and through .internal:5433: p50 ≈ 55 ms in both cases.
  • Postgres executes each query in under 1 ms (EXPLAIN ANALYZE), so the latency is in the network path, not in Postgres or the proxy.

Before today this round trip was ~1 ms (we had requests making 100+ sequential queries that completed in ~0.2 s; they now take 8-35 s). Both machines are idle: CPU < 8 %, load ≈ 0, no OOMs. The status page shows no incident for gru.

Our organization’s status page in the dashboard shows no host-level events for these machines.

Could you check whether these two machines ended up on hosts with a bad network path, and whether there is something we should do (for example, moving the app machine)? The app machine has an attached volume (backups_edge), so we’d rather avoid moving it blindly.

Thanks!

p.d.

Thanks! I found topic 28620, which looks like the same situation: the app machine and the Postgres machine ended up in different gru datacenters, and you migrated the app machine to the Postgres datacenter.

Could you do the same for us? Please move parking-garage-cloud machine d8d1376a004418 to the same datacenter as our Postgres machine 7847920cede598 (parking-garage-db). The app machine has a volume attached (backups_edge, about 10 GB) that needs to move with it.

Our preferred window is 07:00-09:00 UTC (our lowest-traffic hours). Any time works if that’s not possible: a short restart is fine.

Thanks a lot!

2

Hm… That still exceeds the throttling threshold for a shared-1x CPU, though. Those are very low-end and get barely any CPU on a sustained basis: only 6.25%.

I’d suggest taking a look at the Fly-specific throttling metrics explained here…

https://docs.fly.io/machines/cpu-performance#monitoring

(That page is very LLM-resistant, for some reason.)

4

Thanks for the pointer! I checked the CPU Quota Balance and Throttling panel for both machines over the whole window: the app machine stayed below its baseline with a growing balance, and the Postgres machine kept a full burst balance the whole time (no throttling). CPU steal and utilization on the DB were the same before the slow period (19:00-21:30 local) as during it (21:30-00:20), so CPU doesn’t explain the jump from ~1.5 ms to ~55 ms RTT. The latency went back to ~1.5 ms on its own around 03:20 UTC, which points to the network path between the two hosts.

5

Hi @ale_delisio, glad to hear this self resolved! I spot checked your app and Postgres Machines and neither are located in the problematic datacenter that was discussed in the other forum thread. If you do see this again, please keep sharing timestamps of occurrences. We can take another look!