Quick disclaimer: I’m not a DevOps engineer and am relying heavily on assistance from Claude to troubleshoot why our Brazilian players are having such drastic latency issues that even European players are not experiencing.
We run ody-server, a Node WebSocket game server on a single always-on machine with primary_region = "dfw". Measurements below were taken against both our shared IPv4
(66.241.125.197) and a dedicated IPv4 we allocated during this investigation
(137.66.34.162). Both show the same behavior.
Players in Brazil report high latency and unstable connections. We’ve measured the path end to end, and the problem looks like it sits inside your network on the `gru` → `dfw` leg. Everything up to your edge measures clean, so we wanted to bring you the data rather than guess.
Test player: São Paulo, on Telefônica Brasil / Vivo (AS27699). IPv4 only — no IPv6 on their network, so none of this involves v6 routing.
What works: they reach your São Paulo edge perfectly
Accepted at gru, confirmed by the response header:
fly-request-id: 01M12B1HX34128A4HCH9YPR6PS-gru
Traceroute to ody-server.fly.dev — 12 hops, never leaves São Paulo, terminating on your
shared IPv4:
4 4 ms 3 ms 5 ms 201-1-224-160.dsl.telesp.net.br \[201.1.224.160\]
6 11 ms 11 ms 11 ms 187-100-61-8.dsl.telesp.net.br \[187.100.61.8\]
7 \* 14 ms 14 ms 187-100-57-122.dsl.telesp.net.br \[187.100.57.122\]
9 13 ms 13 ms 15 ms 84.16.7.116
10 10 ms 10 ms 14 ms 5.53.0.72
11 11 ms 14 ms 9 ms vl221.sao-asc4-dist-1.cdn77.com \[79.127.195.139\]
12 11 ms 14 ms 9 ms ip-66-241-125-197.shared.customer.flyio.net \[66.241.125.197\]
ICMP to 66.241.125.197, 74 consecutive samples:
min/avg/max = 10 / 12.35 / 20 ms
packet loss = 0%
mean deviation (jitter) = 1.26 ms
12 ms to your edge, zero loss, ~1 ms jitter. Their ISP’s path to you is in excellent shape.
What doesn’t: the backhaul from that edge to our dfw machine
40 consecutive requests to /api/health\ – a trivial handler (c.json({ ok: true }), no
database access, no I/O). Run as two sets of 20, each pinned with curl --resolve to a
different one of our ingress IPv4s, so DNS resolution and shared-IP SNI routing are both
eliminated as variables:
curl -4 --resolve ody-server.fly.dev:443:<IP> \\
https://ody-server.fly.dev/api/health \\
-w "%{time_connect} %{time_appconnect} %{time_starttransfer}\\n"
ttfb - tls isolates everything after TLS terminates at your edge: edge → our machine →
back, plus ~15 ms of client<->edge and ~2 ms of application time.
The two ingress addresses behave identically, which is why we believe the fault is downstream
of your edge rather than in how we are reached:
shared 66.241.125.197 dedicated 137.66.34.162
TCP handshake p50 14.6 ms 14.8 ms
TLS handshake p50 30.7 ms 29.6 ms
backhaul p50 175.5 ms 176.0 ms
slow-path rate 2/20 (10%) 5/20 (25%)
(The difference in slow-path rate between the two is not something 20 samples each can
distinguish; pooled it is 7/40. The medians are the meaningful comparison and they match.)
Pooled across both, the backhaul is not noisy – it is bimodal, in two tight clusters with a
110 ms void between them:
FAST 33/40 (82.5%) 151.9 - 185.3 ms
SLOW 7/40 (17.5%) 295.1 - 349.9 ms
150-174 ms 15 ###############
175-199 ms 18 ##################
200-224 ms 0 <- no samples
225-249 ms 0 <- no samples
250-274 ms 0 <- no samples
275-299 ms 2 ##
300-324 ms 1 #
325-349 ms 4 ####
Two things stand out:
- Even the fast cluster (~152-185 ms) is high for Sao Paulo ↔ Dallas, where we would expect
roughly 130 ms. - The distribution is bimodal rather than long-tailed. Congestion produces a continuous tail;
two discrete clusters separated by a 110 ms void, reproduced independently on two different
ingress addresses, look like two different paths with the slower one selected some of the
time.
Control: our machine is not the cause
The same endpoint, 100 consecutive requests from North Carolina (accepted at dfw):
ttfb - tls: min 37.6 | p50 39.9 | p95 44.5 | p99 50.3 | max 57.8 ms
jitter 1.7 ms; all 100 samples within +24 ms of the minimum
The application answers in ~2 ms with a 13 ms p99 tail, so the variance above isn’t coming from
our side.
Also ruled out
- IPv6 misrouting (the known Brazilian case) — this player has no IPv6 at all.
- Ingress/peering — they reach
gruin 12 ms with 0% loss. - Last mile — 1.26 ms jitter across 74 samples.
- Our application — see the control above.
What we’re asking
Could someone look at the gru → dfw path for this app?
- Why is the baseline ~150-170 ms rather than the ~130 ms the distance implies?
- Why does roughly 17% of traffic take a ~295-350 ms path, with nothing in between?
- Is a secondary route being selected, and can it be dropped or retuned?
Context on why the instability matters more than the mean: we hold long-lived WebSockets (10 s
server-side ping sweep) and the game reconciles player movement against measured RTT. A stable
170 ms is far better for us than an average of 240 ms that oscillates — jitter is what actually
breaks gameplay, not latency.
Minor secondary observation: this player’s TLS handshake costs ~30 ms against a ~15 ms RTT,
about two round trips, where our North Carolina connections complete TLS in ~1.2. This is
unchanged between our shared and dedicated IPv4, so it isn’t SNI routing. Possibly a
HelloRetryRequest from a curve mismatch – minor next to the backhaul, but mentioning it in
case it points at an edge config difference.
Happy to run further measurements from the Brazilian connection — we can provide a larger sample of the backhaul timings, or packet captures, on request.