gru edge → dfw machine backhaul is bimodal (~165 ms / ~330 ms) for Brazilian players

Quick disclaimer: I’m not a DevOps engineer and am relying heavily on assistance from Claude to troubleshoot why our Brazilian players are having such drastic latency issues that even European players are not experiencing.

We run ody-server, a Node WebSocket game server on a single always-on machine with primary_region = "dfw". Measurements below were taken against both our shared IPv4
(66.241.125.197) and a dedicated IPv4 we allocated during this investigation
(137.66.34.162). Both show the same behavior.

Players in Brazil report high latency and unstable connections. We’ve measured the path end to end, and the problem looks like it sits inside your network on the `gru` → `dfw` leg. Everything up to your edge measures clean, so we wanted to bring you the data rather than guess.

Test player: São Paulo, on Telefônica Brasil / Vivo (AS27699). IPv4 only — no IPv6 on their network, so none of this involves v6 routing.


What works: they reach your São Paulo edge perfectly

Accepted at gru, confirmed by the response header:

fly-request-id: 01M12B1HX34128A4HCH9YPR6PS-gru

Traceroute to ody-server.fly.dev — 12 hops, never leaves São Paulo, terminating on your

shared IPv4:

     4    4 ms    3 ms    5 ms  201-1-224-160.dsl.telesp.net.br \[201.1.224.160\]

     6   11 ms   11 ms   11 ms  187-100-61-8.dsl.telesp.net.br \[187.100.61.8\]

     7    \*      14 ms   14 ms  187-100-57-122.dsl.telesp.net.br \[187.100.57.122\]

     9   13 ms   13 ms   15 ms  84.16.7.116

    10   10 ms   10 ms   14 ms  5.53.0.72

    11   11 ms   14 ms    9 ms  vl221.sao-asc4-dist-1.cdn77.com \[79.127.195.139\]

    12   11 ms   14 ms    9 ms  ip-66-241-125-197.shared.customer.flyio.net \[66.241.125.197\]

ICMP to 66.241.125.197, 74 consecutive samples:

    min/avg/max = 10 / 12.35 / 20 ms

    packet loss = 0%

    mean deviation (jitter) = 1.26 ms

12 ms to your edge, zero loss, ~1 ms jitter. Their ISP’s path to you is in excellent shape.


What doesn’t: the backhaul from that edge to our dfw machine

40 consecutive requests to /api/health\ – a trivial handler (c.json({ ok: true }), no
database access, no I/O). Run as two sets of 20, each pinned with curl --resolve to a
different one of our ingress IPv4s, so DNS resolution and shared-IP SNI routing are both

eliminated as variables:

    curl -4 --resolve ody-server.fly.dev:443:<IP> \\
         https://ody-server.fly.dev/api/health \\
         -w "%{time_connect} %{time_appconnect} %{time_starttransfer}\\n"

ttfb - tls isolates everything after TLS terminates at your edge: edge → our machine →
back, plus ~15 ms of client<->edge and ~2 ms of application time.

The two ingress addresses behave identically, which is why we believe the fault is downstream
of your edge rather than in how we are reached:

                            shared 66.241.125.197   dedicated 137.66.34.162

      TCP handshake  p50           14.6 ms                 14.8 ms

      TLS handshake  p50           30.7 ms                 29.6 ms

      backhaul       p50          175.5 ms                176.0 ms

      slow-path rate            2/20   (10%)            5/20   (25%)

(The difference in slow-path rate between the two is not something 20 samples each can
distinguish; pooled it is 7/40. The medians are the meaningful comparison and they match.)

Pooled across both, the backhaul is not noisy – it is bimodal, in two tight clusters with a
110 ms void between them:

      FAST  33/40  (82.5%)   151.9 - 185.3 ms

      SLOW   7/40  (17.5%)   295.1 - 349.9 ms



      150-174 ms  15  ###############

      175-199 ms  18  ##################

      200-224 ms   0           <- no samples

      225-249 ms   0           <- no samples

      250-274 ms   0           <- no samples

      275-299 ms   2  ##

      300-324 ms   1  #

      325-349 ms   4  ####

Two things stand out:

  1. Even the fast cluster (~152-185 ms) is high for Sao Paulo ↔ Dallas, where we would expect
    roughly 130 ms.
  2. The distribution is bimodal rather than long-tailed. Congestion produces a continuous tail;
    two discrete clusters separated by a 110 ms void, reproduced independently on two different
    ingress addresses, look like two different paths with the slower one selected some of the
    time.

Control: our machine is not the cause

The same endpoint, 100 consecutive requests from North Carolina (accepted at dfw):

    ttfb - tls:  min 37.6 | p50 39.9 | p95 44.5 | p99 50.3 | max 57.8 ms
                 jitter 1.7 ms; all 100 samples within +24 ms of the minimum

The application answers in ~2 ms with a 13 ms p99 tail, so the variance above isn’t coming from
our side.

Also ruled out

  • IPv6 misrouting (the known Brazilian case) — this player has no IPv6 at all.
  • Ingress/peering — they reach gru in 12 ms with 0% loss.
  • Last mile — 1.26 ms jitter across 74 samples.
  • Our application — see the control above.

What we’re asking

Could someone look at the grudfw path for this app?

  1. Why is the baseline ~150-170 ms rather than the ~130 ms the distance implies?
  2. Why does roughly 17% of traffic take a ~295-350 ms path, with nothing in between?
  3. Is a secondary route being selected, and can it be dropped or retuned?

Context on why the instability matters more than the mean: we hold long-lived WebSockets (10 s
server-side ping sweep) and the game reconciles player movement against measured RTT. A stable
170 ms is far better for us than an average of 240 ms that oscillates — jitter is what actually
breaks gameplay, not latency.

Minor secondary observation: this player’s TLS handshake costs ~30 ms against a ~15 ms RTT,
about two round trips, where our North Carolina connections complete TLS in ~1.2. This is
unchanged between our shared and dedicated IPv4, so it isn’t SNI routing. Possibly a
HelloRetryRequest from a curve mismatch – minor next to the backhaul, but mentioning it in
case it points at an edge config difference.

Happy to run further measurements from the Brazilian connection — we can provide a larger sample of the backhaul timings, or packet captures, on request.

I’m not sure where Claude got this from, but internet latency can’t be estimated just from distance alone.

These tests measure TTFB latency, which is very different from underlying network latency. I would guess this could be a result of TCP layer things like retransmission, or these connections just landed on different “edges” we have since they’re all separate connections, but I don’t think there’s enough information in this thread to make a conclusion. Could you instead share measurement of jitter inside your websocket channel, not on new TCP connections each time? From your description of your architecture I’m not sure TTFB is the right thing to measure here. Within a single TCP stream, as long as the connection is not broken, the route should not change.

In general, though, we do not currently have dedicated backbone network between our regions; we do try our best to make use of the best internet upstreams we could have, but for cross-continent traffic occasional packet loss and jitter is to be expected, unfortunately. We are working on some improvements for the more extreme cases that we’ll hopefully share soon. In your case, our recommendation is usually that you should have a Fly Machine running closer to your users by rearchitecting your app to support this kind of setup. I do understand that this is probably hard for a game that needs to be consistent for all players.

(Also: when was your measurement made? We changed our edge setup globally some time yesterday which should result in better cross-regional latency ~everywhere.)

Hey Peter, thanks for your response and I really appreciate your help! I forwarded what you said to Claude and it had me run some new tests. I ran them for both myself (based in NC, USA) and a user in Brazil, provided them back to Claude, and had it summarize the findings below.


Both of your objections were right, and correcting for them produced a cleaner result than the original measurement.

  • On the ~130 ms estimate. That was a fiber-distance floor, not a measurement. Withdrawing the question.
  • On TTFB across fresh connections. Also right, and it was materially inflating the number. Each of those 40 samples opened its own connection. Re-measured inside a single connection, the slow-path rate drops from 17.5% to about 2%.

So here is the in-stream measurement you asked for. It changes what we’re asking about.


Method

One curl invocation, 300 sequential requests, paced at one per second over five minutes, all on a single connection:

    curl.exe -4 -s -o NUL --rate 60/m --resolve ody-server.fly.dev:443:137.66.34.162 -w '%{time_starttransfer} %{num_connects} %header{fly-request-id} %header{date}\n' 'https://ody-server.fly.dev/api/health?[1-300]' | Out-File -Encoding utf8 output.txt

num_connects reads 1 on the first request and 0 on all 299 that follow, so this is one TCP connection and one TLS session with a fixed 5-tuple for the whole run. The endpoint is
c.json({ ok: true }) with no I/O.

This is HTTP/2 rather than our game WebSocket, chosen so you can reproduce it yourselves against an unauthenticated endpoint. It has the property you asked for: one stream, one 5-tuple, no reconnects. We can supply WebSocket frame timings as well if you’d prefer them.


Result: two discrete modes inside a single connection

São Paulo client, Telefônica Brasil / Vivo (AS27699), IPv4 only. All 300 requests were accepted at gru — the fly-request-id region suffix is constant across the entire connection, with 300 distinct request ids. So separate edges cannot explain what follows.

Excluding the first request, which carries the handshake:

               min  152.13 ms
               p50  175.87 ms
               p95  186.63 ms
               p99  312.30 ms
               max  343.70 ms

That p50 is not a midpoint of a distribution. The 292 sub-250 ms samples fall into two tight clusters with a near-empty valley between them:

      152-153 ms   22  ######################
      154-155 ms   33  #################################
      156-157 ms   30  ##############################
      158-159 ms   14  ##############
      160-161 ms    4  ####
      162-163 ms    6  ######
      164-165 ms    4  ####
      166-167 ms    0
      168-169 ms    1  #
      170-171 ms    2  ##
      172-173 ms    1  #
      174-175 ms   34  ##################################
      176-177 ms   46  ##############################################
      178-179 ms   37  #####################################
      180-181 ms   37  #####################################
      182-183 ms   10  ##########
      184-185 ms    2  ##
                       samples    mean      stdev
       low  mode          113   156.55 ms   2.92 ms
       high mode          179   178.88 ms   3.81 ms
       separation                22.33 ms

The connection alternates between the two modes 141 times across 292 consecutive requests, in runs mostly one to three long. We ran this twice, ten minutes apart, once unpaced as a 53-second burst and once paced over five minutes. The two runs agree to within a millisecond:

                          run 1 (burst)      run 2 (paced)
       low  mode        115 @ 157.19 ms    113 @ 156.55 ms
       high mode        179 @ 178.63 ms    179 @ 178.88 ms
       separation           21.44 ms           22.33 ms

Same two modes, same population split of roughly 39% low and 61% high, on two independently established connections.


The same measurement from North Carolina is unimodal

Identical command, same curl build (8.13.0, Schannel), same endpoint, same server process, also 300 requests on one connection over five minutes. Accepted at dfw on all 300, so there is no inter-region leg in it:

               min   36.47 ms
               p50   38.11 ms
               p95   43.12 ms
               p99   48.57 ms
               max   72.58 ms
             stdev    2.95 ms
      36-37 ms  134  ##############################################
      38-39 ms  126  ###########################################
      40-41 ms   22  ########
      42-43 ms    5  ##
      44-45 ms    4  #
      46-47 ms    4  #
      48-49 ms    1
      52-53 ms    1
      54-55 ms    1
      72-73 ms    1

One continuous peak, no valley anywhere in the distribution. If our application or the measurement method generated 22 ms of structure, it would appear here too. It doesn’t.


The slow path also survives in-stream, and it is discrete as well

7 of 299 samples exceeded 250 ms. They do not spread out — they cluster near 302 ms and near 343 ms. A retransmission timeout would key off the connection’s own RTT estimate rather than landing repeatedly on two fixed values.

One further detail we can’t explain: samples 4, 5 and 7 were slow in both runs, at roughly

342-346 ms each time, at the same positions on two separately established connections. That looks deterministic rather than random, and may be a different phenomenon from the steady-state alternation above.


What we’re asking

Not about the mean any more. A stable 179 ms is entirely workable for us.

Within a single TCP stream to a constant gru edge, with a fixed 5-tuple, response time alternates between two tight modes 22 ms apart, switching roughly every one to three requests. Each mode individually is as tight as our clean North Carolina distribution, so this isn’t congestion or loss-driven variance.

  1. Does the grudfw path spread traffic across more than one route on a per-packet or per-request basis, rather than hashing per flow? Seen from outside, that is exactly what two tight modes alternating inside one connection would look like.
  2. If so, can traffic for this app be pinned to the shorter of the two?

Why the alternation matters more than the latency: we hold long-lived WebSockets and reconcile player movement against a measured round trip. An estimate that is wrong by 22 ms about half the time is worse for us than a consistently slower one, because the correction is applied against whichever mode the last sample happened to land in.

On rearchitecting to run closer to players: understood, and thanks for the suggestion. It isn’t available to us here. Every player shares one authoritative world, so the room has to live in one place; moving it closer to Brazil just relocates the same problem onto everyone else.

Prior measurements from this client, unchanged: 12.35 ms to your gru edge over 74 ICMP samples, 0% loss, 1.26 ms mean deviation, on a 12-hop traceroute that never leaves São Paulo. The ingress is in excellent shape. Happy to run packet captures or supply the raw sample files.

Sorry, this is still not really the same thing as latency jitter within one single Websocket connection. You need to measure that because that is probably what actually matters here. Our edges make balancing decisions on a per HTTP request basis, so it is not surprising that each HTTP request can end up with slightly different latency profiles even if they’re within the same edge->user connection. This does not happen within one Websocket session at all though.

Thanks Peter! That’s good to know. I’m deploying some tooling to allow us to measure the real game websockets. Hopefully should have some more useful data later today.