dfw: intermittent one-way 6PN delays (550–950 ms) app → Postgres, across 4 different hosts

Hi,

Since mid-November we see recurring episodes where packets from our app machine to our Postgres machine in dfw (both on 6PN, connecting via terpsystem-db.internal:5432) are delayed by 550–950 ms, in bursts, for 45–110 minutes at a time. The machines stay healthy, nothing appears on status.flyio.net or the dashboard, and the problem has followed us across host changes.

Apps / machines (org personal, region dfw)

  • App terpsystem: current machine 8655514cee2318 (host 4386, 6PN fdaa:4d:e10:a7b:9e:6940:9964:2); previous machine 32879767c92385 (host 5545, destroyed 2026-11-17).
  • DB terpsystem-db (postgres-flex 17.2, single node): current machine 781e661be3e038 (host c957, 6PN fdaa:4d:e10:a7b:742:46ea:94ba:2); previous machine 784e775f3d1748 (host 3aa3, destroyed 2026-11-16).

Episodes (UTC)

# When App host DB host
1 2026-11-16 00:44 – 01:50 5545 3aa3
2 2026-11-17 16:00 – 17:49 5545 c957
3 2026-11-18 05:05 – 05:52 4386 c957

Each time the signature is the same: Postgres BEGIN statements take 550–950 ms (occasionally ~1.3 s), with zero time spent waiting for a pool connection. The stalls often come in pairs ~2–3 s apart, repeated every few seconds to minutes.

What we measured during episode 1 (probes run from inside the VMs)

  • The delay is one-way, app → DB: a separate connection running SELECT clock_timestamp() showed ~850–950 ms on the way in and ~1 ms on the way back (clock offset between the two VMs: 164 µs).
  • Port 5432 (haproxy) and port 5433 (Postgres directly) freeze in the same millisecond, so it isn’t haproxy or Postgres.
  • From the app VM, only the DB stalled. Other 6PN targets in the same org and external hosts were clean in parallel. From the DB VM, only the path toward the app stalled.
  • A timing loop inside each VM showed no pauses, and CPU, steal, IO and memory were flat on both machines.
  • A clone of the app placed on host c957 also saw stalls toward the DB on 3aa3.

What we measured during episode 2

  • TCP retransmission timeouts (TCPTimeouts in /proc/net/netstat): 1105 on the app VM in ~9 h since boot, versus 106 on the DB VM in 36 h.
  • fly_instance_net_*_drop and *_errs were zero on both VMs, so the loss seems to happen outside the guests.

What we changed

  • We moved the DB to a new host (3aa3 → c957) on 2026-11-16 05:31.
  • We moved the app to a new host (5545 → 4386) on 2026-11-17 18:40.
  • Episode 3 happened with both ends on the new hosts, so we no longer think a single host is at fault.

Questions

  1. Is there known packet loss or congestion on the 6PN / WireGuard path between hosts in dfw around those windows?
  2. Are hosts 4386 and c957 (and earlier 5545 and 3aa3) behind the same network segment or gateway?
  3. Is there anything on our side (MTU, machine placement) you’d recommend changing?

We’re happy to run additional probes during the next episode if that helps. Thanks!