Hi,
Since mid-November we see recurring episodes where packets from our app machine to our Postgres machine in dfw (both on 6PN, connecting via terpsystem-db.internal:5432) are delayed by 550–950 ms, in bursts, for 45–110 minutes at a time. The machines stay healthy, nothing appears on status.flyio.net or the dashboard, and the problem has followed us across host changes.
Apps / machines (org personal, region dfw)
- App
terpsystem: current machine8655514cee2318(host4386, 6PNfdaa:4d:e10:a7b:9e:6940:9964:2); previous machine32879767c92385(host5545, destroyed 2026-11-17). - DB
terpsystem-db(postgres-flex 17.2, single node): current machine781e661be3e038(hostc957, 6PNfdaa:4d:e10:a7b:742:46ea:94ba:2); previous machine784e775f3d1748(host3aa3, destroyed 2026-11-16).
Episodes (UTC)
| # | When | App host | DB host |
|---|---|---|---|
| 1 | 2026-11-16 00:44 – 01:50 | 5545 | 3aa3 |
| 2 | 2026-11-17 16:00 – 17:49 | 5545 | c957 |
| 3 | 2026-11-18 05:05 – 05:52 | 4386 | c957 |
Each time the signature is the same: Postgres BEGIN statements take 550–950 ms (occasionally ~1.3 s), with zero time spent waiting for a pool connection. The stalls often come in pairs ~2–3 s apart, repeated every few seconds to minutes.
What we measured during episode 1 (probes run from inside the VMs)
- The delay is one-way, app → DB: a separate connection running
SELECT clock_timestamp()showed ~850–950 ms on the way in and ~1 ms on the way back (clock offset between the two VMs: 164 µs). - Port 5432 (haproxy) and port 5433 (Postgres directly) freeze in the same millisecond, so it isn’t haproxy or Postgres.
- From the app VM, only the DB stalled. Other 6PN targets in the same org and external hosts were clean in parallel. From the DB VM, only the path toward the app stalled.
- A timing loop inside each VM showed no pauses, and CPU, steal, IO and memory were flat on both machines.
- A clone of the app placed on host
c957also saw stalls toward the DB on3aa3.
What we measured during episode 2
- TCP retransmission timeouts (
TCPTimeoutsin/proc/net/netstat): 1105 on the app VM in ~9 h since boot, versus 106 on the DB VM in 36 h. fly_instance_net_*_dropand*_errswere zero on both VMs, so the loss seems to happen outside the guests.
What we changed
- We moved the DB to a new host (
3aa3→c957) on 2026-11-16 05:31. - We moved the app to a new host (
5545→4386) on 2026-11-17 18:40. - Episode 3 happened with both ends on the new hosts, so we no longer think a single host is at fault.
Questions
- Is there known packet loss or congestion on the 6PN / WireGuard path between hosts in
dfwaround those windows? - Are hosts
4386andc957(and earlier5545and3aa3) behind the same network segment or gateway? - Is there anything on our side (MTU, machine placement) you’d recommend changing?
We’re happy to run additional probes during the next episode if that helps. Thanks!