Two Machines in the same org and the same region (nrt), talking over 6PN: an API app and a roll-your-own Postgres app (stock postgres:18-alpine, no [[services]] block, reached at my-db-app.internal:5432). Both shared-cpu-1x, both started — no autostop, no suspend.
Symptom: the first TCP connection after the path has been idle for a while takes 1–3 seconds. Every subsequent connection is ~4 ms until it goes quiet again.
I’ve instrumented both ends and I’m fairly confident the packet is being dropped inside 6PN, but I’d like to know whether that’s expected and whether there’s anything I can do about it beyond keeping traffic flowing.
Measurement
Run from the API Machine after a cold window. It reads the kernel’s TCPSynRetrans counter either side of one bare TCP connect, and runs a 50 ms timestamp ticker alongside so a descheduled VM would show up as a gap:
cat >/tmp/probe.sh <<'EOF'
#!/bin/bash
HOST=my-db-app.internal
PORT=5432
sr() { awk 'NR==1{for(i=1;i<=NF;i++)h[i]=$i} NR==2{for(i=1;i<=NF;i++) if(h[i]=="TCPSynRetrans") print $i}' /proc/net/netstat; }
TICKF=$(mktemp)
( while :; do date -u +%s.%N; sleep 0.05; done ) >"$TICKF" &
TICKS=$!
sleep 0.5
A=$(sr); T0=$(date -u +%s.%N)
exec 3<>/dev/tcp/$HOST/$PORT
T1=$(date -u +%s.%N); B=$(sr)
sleep 0.5; kill $TICKS 2>/dev/null; exec 3>&-
echo "connect_secs=$(awk -v a="$T0" -v b="$T1" 'BEGIN{printf "%.3f", b-a}')"
echo "syn_retrans_delta=$((B-A))"
awk 'NR>1{d=$1-p; if(d>m){m=d}} {p=$1} END{printf "max_tick_gap=%.3f\n", m}' "$TICKF"
rm -f "$TICKF"
EOF
bash /tmp/probe.sh
Result on a cold path:
connect_secs=2.041
syn_retrans_delta=1
max_tick_gap=0.053
So: the kernel sent a SYN, got no answer, and retransmitted once — and the VM was awake the whole time it waited.
Separately, with log_connections = 'all' on the Postgres side, the timestamps line up exactly:
client: 12:52:46.706225345 before connect()
client: 12:52:48.719480845 after connect() = 2.013 s
server: 12:52:48.719 connection received
The server sees the connection at the same sub-millisecond instant the client’s connect() returns, so nothing is queueing on the receiving end.
One detail I can’t explain: a single SYN retransmit should put the connect at ~1 s (Linux retransmits the first SYN after 1 s), but it lands at 2.04 s. That implies the retried SYN also waited ~1 s for its SYN-ACK — i.e. the path seems to take about a second to come up, and the first SYN dies while that’s happening.
Ruled out
- DNS —
getent hosts my-db-app.internalis 5 ms on a cold path. - Postgres —
log_connections='all'with PG18’ssetup_durationsreportstotal=5.4–6.3 ms, fork=0.34–0.49 ms, authentication=2.4–3.2 ms, identical cold and warm.psqlover the local Unix socket on the DB Machine is < 10 ms. - TLS — the connection string is
sslmode=disable, and the probe above is a bare TCP connect with no Postgres protocol at all. - Client-side work / cold start — a fresh process doing identical work 32 s later completes in 26 ms.
- CPU starvation on the API Machine — that’s what
max_tick_gap=0.053is for. This is the explanation guessed at in 21467, and here it’s measurably not the cause.
How long “idle” has to be
Loosely correlated at best, which is the annoying part for reproducing it:
| idle before probe | stall |
|---|---|
| ~22 min | 1.01 s |
| ~84 min | 4 ms (no stall) |
| ~4 h 26 min | 2.04 s |
A bare TCP connect is enough to warm it, and the warm state appears to be per-peer, not per-flow — after one connect, brand-new sockets on different source ports are also fast.
What I’ve done about it, and what it confirmed
Two changes, now deployed:
- the connection pool holds one connection open permanently (
min_connections = 1), instead of emptying after ten idle minutes and opening cold on the next request; - Postgres starts with
tcp_keepalives_idle=60, so that held connection actually emits packets rather than sitting silent on a path that decays underneath it.
Either alone is useless, which is itself a hint about the mechanism: an open but silent connection does not keep the path alive.
Two things fell out of this that may be useful evidence.
The pool’s own recycling turns out to keep the path warm as a side effect. The pool ages connections out on a ~10 minute cadence and immediately opens a replacement, and 10 minutes is comfortably inside the 22-minute shortest decay I measured. So there are now two independent mechanisms holding the route open, and the request path can’t reach a cold path by construction.
Machine boot still pays it every time, which is the part I can’t work around — a freshly booted Machine has no warm connection to inherit. Two consecutive boots after deploying:
machine A starting 01:26:50.926 → connected 01:26:52.992 2.066 s
machine B starting 01:27:09.237 → connected 01:27:11.262 2.025 s
Same 1–3 s, same shape, on a Machine whose only prior act was to exist. It also briefly fails its own health check on the way up, which the grace period absorbs — but it means the cold-path cost is effectively a tax on every deploy and every restart.
Questions
- Is this expected behaviour for 6PN — is there on-demand route/tunnel setup between Machines that haven’t exchanged packets recently, and is dropping the first packet part of that?
- Is there any way to keep the path warm other than generating traffic myself?
- The boot case is the one I’d most like an answer on, since it’s the one no application-level workaround reaches: is there anything that can pre-warm the route to a peer before a freshly started Machine’s first connection, or any way to have the first packet queued rather than dropped while the path comes up?
Happy to re-run the probe with any extra instrumentation, or to share app names / org if that helps someone look at it from the inside.
Related and, as far as I can tell, unresolved: 21467 and 21000. Both are outbound-internet with TLS in the path, which I think muddied the diagnosis — this one is plaintext, intra-region, Machine-to-Machine.