Several times a day, for 10 to 60 seconds, the API cannot open a new TCP connection to the database and its existing connections stop moving. Both VMs, Postgres itself and the Flex machine’s HAProxy are demonstrably healthy the whole time, so I believe the loss is on the 6PN path between the two hosts. I’d appreciate someone looking at the two hosts for that window.
Episodes (UTC)
2026-09-09: 01:55–01:58, 11:05 (one established connection died mid-query, Connection terminated unexpectedly), 18:25–18:27, 20:19
The first four happened with DATABASE_URL pointing at pedale-sempre-db.flycast:5432. On 2026-09-10 05:37Z I switched it to pedale-sempre-db.internal:5432 to take fly-proxy out of the path. The last four happened after that, with the same shape, so the proxy is not it.
What the client sees
pg-boss (10 pollers, pg-pool, connectionTimeoutMillis 10 s) logs timeout exceeded when trying to connect and Connection terminated due to connection timeout from every worker, repeating every 10 s until the window ends. Between windows everything is normal, including a cron that queried the database at 20:45:00Z, between the 20:42 and 20:47 bursts.
What is ruled out
Neither machine restarted (API up since 06:04Z on 09-10, DB since 08-19). HostStatus: ok on both.
Postgres never stalled: postgres_exporter runs inside the DB VM and scrapes Postgres on 5433 every 15 s; pg_up stayed 1 and pg_exporter_last_scrape_error 0 through every episode, scrape duration 10–30 ms. Postgres log at each instant: checkpoints and repmgrd “normal state” only.
HAProxy on the DB machine (the 5432 listener): stats over 22.5 days show econ 0, eresp 0, wretr 0, cli_abrt 0, srv_abrt 0, primary never marked down (chkdown 1, downtime 0; the pg1 server had one 4 s downtime at boot).
No CPU throttling, steal flat (DB ~1–2 %, API ~0), no memory pressure, zero interface errors/drops on either side (fly_instance_net_*). Fly status page clean for both days.
Not DNS-shaped: established flows hang too, and the 11:05Z event was an established connection dying mid-query.
What points at the path
fly_instance_net_recv_packets on both VMs falls to 10–25 % of baseline during each window (the DB VM’s traffic is almost entirely this API’s polling), then recovers.
API kernel counters (/proc/net/snmp, /proc/net/netstat) 17 h after boot, on a VM whose TCP traffic is almost exclusively this one 6PN peer: ActiveOpens 1408, TCPSynRetrans 141, TCPTimeouts 411, TCPLostRetransmit 337, RetransSegs 597, AttemptFails 0, InErrs 0.
ss -tin on the DB machine for the API’s ten established sockets (age ~2 h at the time): seven of ten carry retransmits (retrans:0/1 up to retrans:0/15), RTT 1.6–11.6 ms with rttvar up to 16 ms, for a same-region hop.
pg_stat_database_numbackends for the app database drops from 11 to 7 (09-10) and to 1 (09-09 18:26 and 01:56) during the windows and rebuilds afterwards — the client pool giving up on hung sockets and reconnecting.
Question
Is there anything known about the gru hosts these two machines sit on (per-host network issue, a WireGuard → mTLS transport migration in progress, a neighbour saturating the host’s 6PN queues)? And is there a way, short of a support plan, to get the host-side view of the path for one of the timestamps above? I can add any counter or capture you want from either VM.
Hi! Our networking team’s aware of your report and will take a look at the route between these two hosts where your API and Postgres are running.
In the meantime, you could try adding a new Machine in your API app and see if it works better: fly machine clone 28747327c61958. This places a clone of your existing Machine on a different host. If intermittent instability is indeed between one specific pair of hosts, you should see an improvement in the new Machine (and you can destroy 28747327c61958, where you’re seeing connection issues).
I’d rather hold off on the clone for now and wait for your networking team’s view of the route between the two hosts. This app runs as a single Machine on purpose, and moving it before you’ve had a look would take away the pair you’re investigating — right now both Machines are still where they were, nothing has restarted, and the episodes are continuing with the same shape (10–60 s, a few times a day, none at night), so the path is there to be observed.
If it helps, I can add timestamps as new episodes happen, or run any capture or counter you want from either VM during one of them. Just say what would be useful.
If your team finds nothing on the host side, I’ll do the clone then and report back with the new Machine id and whether the episodes followed it.
that would be helpful. this appears to be related to packet loss events between the datacenters your two machines are in. we have tooling to automatically reroute traffic when that happens, and we’re investigating whether it’s working properly or needs some changes (or if it is a different issue entirely).
For the record: no episode since 2026-09-10 22:24Z, the last one in my original list — 3+ days on the same pair of Machines (the API Machine was restarted by a deploy on 09-12 but kept its id and host; the DB has not restarted since 08-19).
Before that it was eight episodes in two days, so whatever changed, changed around the time you looked. I’ll post timestamps here if it comes back.
One more, milder: 2026-09-15 15:53:08Z — a single pg-boss worker hit the 10 s connect timeout (“Connection terminated due to connection timeout”); the other nine workers, both VMs and Postgres were fine, packet rates flat, transaction rate unchanged.
Not a 10–60 s blackout like the ones listed above, just one lost handshake.
Same pair of Machines, neither restarted (API 28747327c61958 up since 09-14 09:33Z, DB 2874de5a903928 since 08-19).
Update.
2026-09-17 was the worst day so far: six episodes between 15:57Z and 16:55Z (15:57, 15:59, 16:12, 16:24, 16:37–16:39, 16:52–16:55), all ten pg-boss workers hitting the 10 s connect timeout in the two long ones. Same shape as before: neither VM restarted, Postgres healthy, packet rate on both VMs dropped during each one.
One new fact: at 16:38:47Z the API also failed a TLS request to oauth2.googleapis.com (Google sign-in token exchange), in the same seconds the 6PN path to Postgres was dead. So egress to the public internet from the API host was affected too, not only the route to the DB host.
I have taken your suggestion: on 2026-09-18 00:29Z I cloned the API Machine and destroyed 28747327c61958. The API is now 080e9079f93e98 (host cdc2; the old one was on e042, the DB 2874de5a903928 is on 5f0f). I’ll report whether the episodes stopped or followed the new Machine.
Do you have any update from the investigation on your side?
Since the clone on 2026-09-18 00:32Z (API now 080e9079f93e98 on host cdc2, DB 2874de5a903928 still on 5f0f), two more:
2026-09-18 10:39:42–46Z — three pg-boss workers hit the 10 s connect timeout
2026-09-19 07:17:02Z — three workers, same second
Both shorter than the 09-17 ones (one round of timeouts rather than 10–60 s), but the same shape: packet rate dropped on both VMs in the same minute, Postgres backends fell from 11 to 8–9 and rebuilt over a few minutes, pg_up stayed 1 throughout, Postgres logged nothing but checkpoints, zero interface errors or drops on either VM, and neither Machine restarted (the API has been up continuously since 09-18 01:28Z).
So moving the API off e042 did not remove it, which leaves the DB host 5f0f or the route into it as the common factor. Two questions:
Does your side see packet loss into 5f0f at those two timestamps?
If the host is the problem, is cloning the DB Machine (it has a volume) the recommended way to move it, or can you migrate it for me?
Happy to run any capture from either VM during the next one.
Thanks for your follow ups! I checked the new API Machine you created and although it’s placed on a different host, it’s still in the same datacenter as your original.
I can confirm a packet loss event affecting gru hosts in that same datacenter correlated with the extended issue you saw on Sept 17th. To be transparent, we have growing network stability concerns there and in the near future, we may end up moving customer workloads elsewhere.
Your Postgres Machine is in a different datacenter - I think connectivity should be more steady if your API is moved there, too. Since there isn’t a user-facing way to enforce this placement strategy (well, without trial and error), we can definitely migrate your API Machine for you.
If you’d prefer this migration to take place at a particular time of day, we can keep that in mind when we move the Machine.
Thanks for the diagnosis — that explains why the clone followed the same pattern.
Yes, please go ahead and migrate the API Machine (app pedale-sempre-api, Machine 080e9079f93e98) into the same datacenter as the Postgres Machine (2874de5a903928, app pedale-sempre-db).
Preferred window: any time between 03:00Z and 09:00Z (00:00–06:00 America/Sao_Paulo). That is when the shop is closed and traffic is essentially zero, so a restart is harmless. Any day works.
Two things worth knowing about the Machine: it runs a single instance with auto-stop disabled (it hosts pg-boss crons that need a live process), and it holds no volume, so a stop/start or a clone-and-destroy is fine from our side. It should keep the same app-level config and secrets.
If you could post here once it is done, I will watch the connection timeouts over the following days and report back either way.
Hi again - we’ve moved that Machine. It’s been running in the same datacenter as your Postgres since 09-22 04:51 UTC. Hope to hear that helps with the connectivity issues!