fly.io instance response times

My app is operating a bit slow so I am looking into my server on Fly.io which I simply use as a poke/pull endpoint.

On the fly.io dashboard, it looks like it’s about 130ms of latency, which seems very high when I expected single-digit millisecond latency, and when I click into the Grafana metrics, it looks like it’s about 1-5ms of latency.

What is the difference between the two and what should I be considering as the true latency of pinging my Fly.io server?

Thanks in advance!

Hey @chris-io

fly_edge_http_response_time_seconds is measured on the edge. It’s the total time it took for a request to get processed including edge <-> worker communication (possibly forwarding to a different region).

HTTP Response Times on fly-metrics.net shows fly_app_http_response_time_seconds. It’s measured on a worker and shows your app response time.

Could you post the output of this command here?

curl -s -I -H "flyio-debug: doit" https://<your app domain> | grep flyio-debug

This will show the edge region you are getting routed to.

{
  "n": "edge-cf-sea1-b64f",
  "nr": "sea",
  "ra": "2604:3d08:817f:24e0::29aa",
  "rf": "Verbatim",
  "sr": "sea",
  "sdc": "sea1",
  "sid": "2874270a666078",
  "st": 0,
  "nrtt": 0,
  "bn": "worker-cf-sea1-6da7",
  "mhn": null,
  "mrtt": null
}

@chris-io Thanks.

You are getting routed to sea and your app is also in sea. 150ms+ response time is rather unexpected. Are you still experiencing the issue?


Fly.io dashboard says 20-100ms and Grafana says 5ms. It’s seems better than yesterday but still slower than usual since the few days before.

Hmm. The request that you’ve done with flyio-debug: doit header set took ~6ms to complete. The proxy received it at 22:26:56.798843000 and responded at 22:26:56.804672000.

Could you try something else, please?

Save the following data to a file, for example, /tmp/curlformat:

   time_namelookup:  %{time_namelookup}s\n
      time_connect:  %{time_connect}s\n
   time_appconnect:  %{time_appconnect}s\n
  time_pretransfer:  %{time_pretransfer}s\n
     time_redirect:  %{time_redirect}s\n
time_starttransfer:  %{time_starttransfer}s\n
----------\n
        time_total:  %{time_total}s\n
\n\n
%{header_json}

And then do:

curl -s -o /dev/null -w @/tmp/curlformat -H "flyio-debug: doit" https://<your app>

The file instructs curl to dump various timings and response headers. Once you see a request that took too much time (time_total), please post fly-request-id response header value here, so I can look it up in the logs to see what took it so long.

Here’s the output from the curl command you provided with fly-request-id 01JHNX6S1BW8JVCCYG28CGQBWN-chi. I just ran this once and seem to get back 450ms from the time_total. I supposed to keep running this command until I get a slower response or is this what you need?

time_namelookup:  0.178569s
time_connect:  0.227958s
time_appconnect:  0.297843s
time_pretransfer:  0.298064s
time_redirect:  0.000000s
time_starttransfer:  0.454230s
----------
time_total:  0.454824s

{
  "access-control-allow-origin": ["*"],
  "access-control-allow-methods": ["GET, POST, PUT, DELETE, OPTIONS"],
  "access-control-allow-headers": [
    "Origin, Content-Type, X-Auth-Token, X-Requested-With, Accept, Authorization, X-CSRF-TOKEN, XSRF-TOKEN, X-Socket-Id"
  ],
  "content-length": ["2"],
  "uwebsockets": ["20"],
  "date": ["Wed, 15 Jan 2025 21:16:43 GMT"],
  "server": ["Fly/1782112b (2025-01-09)"],
  "via": ["2 fly.io"],
  "fly-request-id": ["01JHNX6S1BW8JVCCYG28CGQBWN-chi"],
  "flyio-debug": [
    "{\"n\":\"edge-nac-chi1-ba8d\",\"nr\":\"chi\",\"ra\":\"206.12.14.237\",\"rf\":\"Verbatim\",\"sr\":\"sea\",\"sdc\":\"sea1\",\"sid\":\"2874270a666078\",\"st\":0,\"nrtt\":59,\"bn\":\"worker-cf-sea1-6da7\",\"mhn\":null,\"mrtt\":null}"
  ]
}

This one actually got routed to an edge server in chi and later forwarded to your app in sea.

From the proxy point of view it took ~106ms, which is expected. RTT between chi and sea is ~50ms. If the proxy didn’t have already established connection between the edge server and the worker server for your app it needs to establish one (that’s one RTT) and another RTT to send the request.

Given that your previous request got routed to sea, I assume chi is unexpected. The previous one was over IPv6, this one is over IPv4. Is this a different internet provider?

I’m not sure what internet provider exactly, just on different wi-fi networks at different places throughout the days.

Did we find any actionable information? Let me know if you need anything else from me. Our production usage from users this weekend still show 150ms+ queries. Would love to get back to the expected performant latencies again.

not trying to hijack this thread but I’m seeing extremely high latency in bos. I’m seeing 4sec+ added latency above what my instance is reporting via sentry

1 Like

If you have access to a client that can reproduce high end-to-end latencies, sending us a traceroute (ideally using, mtr -bzo "LSD NABWV" -r -n [ip] for most helpful formatting/info) could help us identify any sub-optimal routing and potentially improve things.

Are we supposed to be running this command from and give the IP of our local computer or the Fly.io server?

Generally, the [ip] is a remote machine, different from the one that you’re running mtr on.

What it’s doing is tracing out the path through the Internet between the two machines. For example…

$ mtr -bzo "LSD NABWV" -r -n debian.org
Start: 2025-01-21T04:59:05+0000
HOST: 2875067b356718              Loss%   Snt Drop   Last   Avg  Best  Wrst StDev
  1. AS30081  2605:4c40:222:bf20:  0.0%    10    0    1.6   1.7   1.5   1.9   0.1
  2. AS30081  2605:4c40:222:100::  0.0%    10    0    1.2   2.2   0.8  12.2   3.5
  3. AS3257   2001:668:0:3:ffff:2  0.0%    10    0    1.1   5.7   0.7  48.1  14.9
  4. AS3257   2001:668:0:2:ffff:0  0.0%    10    0    1.6   7.1   0.7  47.6  14.5
  5. AS1299   2001:2035:0:6cd::1   0.0%    10    0    1.0   1.1   1.0   1.3   0.1
  6. AS1299   2001:2034:1:3::1    90.0%    10    9    1.4   1.4   1.4   1.4   0.0
  7. AS1299   2001:2034:0:249::1   0.0%    10    0    0.7   0.7   0.7   0.8   0.0
  8. AS1299   2001:2035:0:703::2   0.0%    10    0    1.0   1.0   0.9   1.1   0.1
  9. AS54113  2a04:4e42:600::644   0.0%    10    0    1.1   1.0   1.0   1.1   0.0

The AS* field gives Fly.io a company to contact in case one of the hops looks wrong, I believe. (Like sending Seattle → Seattle traffic over to Chicago first.)

If you could find one of those Wi-Fi spots that was giving bad results again and then invoke mtr from your laptop, while pointing it at your .fly.dev address (i.e., as [ip]), that would provide similar details, :mag_right:.

Start: 2025-01-20T21:49:33-0800
HOST: Chriss-MacBook-Pro-3.local  Loss%   Snt Drop   Last   Avg  Best  Wrst StDev
  1. AS812    2605:8d80:4c1:a967:  0.0%    10    0    8.5  23.2   8.2 112.6  32.1
  2. AS40509  2a09:8280:1::39:be9  0.0%    10    0   47.1  55.0  27.0  90.9  22.9

This is what I get currently from the command with my Fly.io instance hostname (without the prefix https:// and trailing slash /) as the [ip] value

Thanks… Does this improve at all if you add the -4 flag (to force IPv4)?

(Residential IPv6 often has glitches, I hear.)

Not the same location and wifi network as yesterday, but here are the commands ran now today with and without the -4 flag

sudo mtr -bzo "LSD NABWV" -r -n soketi-app-my-app-1234.fly.dev   
Password:
Start: 2025-01-21T16:44:01-0800
HOST: Chriss-MBP-3.ht.home        Loss%   Snt Drop   Last   Avg  Best  Wrst StDev
  1. AS???    192.168.0.1          0.0%    10    0    5.0   9.1   4.6  38.2  10.3
  2. AS6327   50.64.96.1           0.0%    10    0   25.4  38.7  19.9  67.8  16.1
  3. AS6327   64.59.152.97         0.0%    10    0   24.2  25.1  16.4  46.5   9.4
  4. AS6327   24.244.59.17         0.0%    10    0   33.8  22.3  15.8  33.8   6.3
  5. AS6327   24.244.58.61         0.0%    10    0   19.4  20.3  16.1  37.9   6.5
  6. AS???    ???                 100.0    10   10    0.0   0.0   0.0   0.0   0.0
  7. AS6327   66.163.78.37         0.0%    10    0   34.7  37.7  28.1  54.9   9.3
  8. AS6327   66.163.65.18         0.0%    10    0   65.7  61.2  58.4  65.7   2.4
  9. AS???    ???                 100.0    10   10    0.0   0.0   0.0   0.0   0.0
 10. AS???    ???                 100.0    10   10    0.0   0.0   0.0   0.0   0.0
 11. AS???    ???                 100.0    10   10    0.0   0.0   0.0   0.0   0.0
 12. AS40509  66.241.124.14        0.0%    10    0   81.7  83.0  60.5 174.3  35.6

sudo mtr -bzo "LSD NABWV" -r -n soketi-app-my-app-1234.fly.dev -4

Start: 2025-01-21T16:44:22-0800
HOST: Chriss-MBP-3.ht.home        Loss%   Snt Drop   Last   Avg  Best  Wrst StDev
  1. AS???    192.168.0.1          0.0%    10    0   17.4  11.1   5.3  22.4   5.8
  2. AS6327   50.64.96.1           0.0%    10    0   20.8  20.6  14.3  25.6   3.8
  3. AS6327   64.59.152.97         0.0%    10    0   44.8  22.3  15.9  44.8   8.4
  4. AS6327   24.244.59.17         0.0%    10    0   20.1  23.7  18.2  36.3   6.9
  5. AS6327   24.244.58.61         0.0%    10    0   18.9  23.2  18.1  38.9   6.4
  6. AS6327   66.163.69.106       80.0%    10    8   18.3  24.7  18.3  31.1   9.1
  7. AS6327   66.163.78.37         0.0%    10    0   31.4  39.9  31.4  53.7   8.3
  8. AS6327   66.163.65.18         0.0%    10    0   56.0  65.1  56.0  82.7   8.8
  9. AS???    ???                 100.0    10   10    0.0   0.0   0.0   0.0   0.0
 10. AS???    ???                 100.0    10   10    0.0   0.0   0.0   0.0   0.0
 11. AS???    ???                 100.0    10   10    0.0   0.0   0.0   0.0   0.0
 12. AS40509  66.241.124.14        0.0%    10    0   64.1  65.3  58.5  93.1  10.4

Were you guys able to derive anything actionable from the data? Is there anything else I can get or do?

The second traceroute gave the info we needed to confirm a routing issue, traffic from Shaw ISP is being incorrectly routed through Chicago edges instead of directly to Seattle. We’re looking into this and I’ll let you know when we manage to apply any routing fixes for this issue.

3 Likes

I ran the commands on-premise of our production user’s venue and wifi-network, where our performance matters. Would you please be able to check for any in-efficient routing to then contact the ISP to resolve as you are currently doing now? It may appear to also incorrectly route through Chicago.

Wifi at front door and lobby

sudo mtr -bzo "LSD NABWV" -r -n soketi-app-my-app-1234.fly.dev   

Start: 2025-01-24T22:47:36-0800
HOST: Chriss-MBP-3                Loss%   Snt Drop   Last   Avg  Best  Wrst StDev
  1. AS1213   193.1.1.1            0.0%    10    0    4.4   6.7   1.8  40.6  11.9
  2. AS6327   96.49.160.1          0.0%    10    0   14.3  15.3  13.1  17.5   1.2
  3. AS6327   64.59.149.253        0.0%    10    0   43.2  23.7  14.0  54.3  13.8
  4. AS6327   66.163.69.97        40.0%    10    4   17.3  41.8  17.3  76.5  26.6
  5. AS6327   66.163.72.69        40.0%    10    4   39.5  33.3  26.8  39.5   5.6
  6. AS6327   66.163.71.118        0.0%    10    0   69.2  34.4  26.1  69.2  12.8
  7. AS6327   66.163.65.18         0.0%    10    0   58.5  63.1  55.8  90.6  10.8
  8. AS???    ???                 100.0    10   10    0.0   0.0   0.0   0.0   0.0
  9. AS???    ???                 100.0    10   10    0.0   0.0   0.0   0.0   0.0
 10. AS???    ???                 100.0    10   10    0.0   0.0   0.0   0.0   0.0
 11. AS40509  66.241.124.14        0.0%    10    0   55.9  65.2  55.3  94.9  14.0

sudo mtr -bzo "LSD NABWV" -r -n soketi-app-my-app-1234.fly.dev -4

Start: 2025-01-24T22:48:04-0800
HOST: Chriss-MBP-3                Loss%   Snt Drop   Last   Avg  Best  Wrst StDev
  1. AS1213   193.1.1.1            0.0%    10    0    4.1   5.8   2.0  26.6   7.4
  2. AS6327   96.49.160.1          0.0%    10    0   12.4  20.6  10.7  49.8  11.0
  3. AS6327   64.59.149.253        0.0%    10    0   18.1  18.3  11.5  26.3   3.9
  4. AS6327   66.163.69.97        80.0%    10    8   16.1  19.7  16.1  23.4   5.2
  5. AS6327   66.163.72.69        20.0%    10    2   29.8  39.3  27.7  71.6  17.5
  6. AS6327   66.163.71.118        0.0%    10    0   25.2  36.2  25.2  61.8  13.1
  7. AS6327   66.163.65.18         0.0%    10    0   98.5  63.8  54.5  98.5  13.7
  8. AS???    ???                 100.0    10   10    0.0   0.0   0.0   0.0   0.0
  9. AS???    ???                 100.0    10   10    0.0   0.0   0.0   0.0   0.0
 10. AS???    ???                 100.0    10   10    0.0   0.0   0.0   0.0   0.0
 11. AS40509  66.241.124.14        0.0%    10    0   57.1  63.3  56.5  83.7  10.3

Wifi at offices

sudo mtr -bzo "LSD NABWV" -r -n soketi-app-my-app-1234.fly.dev   

Start: 2025-01-24T22:47:36-0800
HOST: Chriss-MBP-3                Loss%   Snt Drop   Last   Avg  Best  Wrst StDev
  1. AS1213   193.1.1.1            0.0%    10    0    4.4   6.7   1.8  40.6  11.9
  2. AS6327   96.49.160.1          0.0%    10    0   14.3  15.3  13.1  17.5   1.2
  3. AS6327   64.59.149.253        0.0%    10    0   43.2  23.7  14.0  54.3  13.8
  4. AS6327   66.163.69.97        40.0%    10    4   17.3  41.8  17.3  76.5  26.6
  5. AS6327   66.163.72.69        40.0%    10    4   39.5  33.3  26.8  39.5   5.6
  6. AS6327   66.163.71.118        0.0%    10    0   69.2  34.4  26.1  69.2  12.8
  7. AS6327   66.163.65.18         0.0%    10    0   58.5  63.1  55.8  90.6  10.8
  8. AS???    ???                 100.0    10   10    0.0   0.0   0.0   0.0   0.0
  9. AS???    ???                 100.0    10   10    0.0   0.0   0.0   0.0   0.0
 10. AS???    ???                 100.0    10   10    0.0   0.0   0.0   0.0   0.0
 11. AS40509  66.241.124.14        0.0%    10    0   55.9  65.2  55.3  94.9  14.0

sudo mtr -bzo "LSD NABWV" -r -n soketi-app-my-app-1234.fly.dev -4

Start: 2025-01-24T22:48:04-0800
HOST: Chriss-MBP-3                Loss%   Snt Drop   Last   Avg  Best  Wrst StDev
  1. AS1213   193.1.1.1            0.0%    10    0    4.1   5.8   2.0  26.6   7.4
  2. AS6327   96.49.160.1          0.0%    10    0   12.4  20.6  10.7  49.8  11.0
  3. AS6327   64.59.149.253        0.0%    10    0   18.1  18.3  11.5  26.3   3.9
  4. AS6327   66.163.69.97        80.0%    10    8   16.1  19.7  16.1  23.4   5.2
  5. AS6327   66.163.72.69        20.0%    10    2   29.8  39.3  27.7  71.6  17.5
  6. AS6327   66.163.71.118        0.0%    10    0   25.2  36.2  25.2  61.8  13.1
  7. AS6327   66.163.65.18         0.0%    10    0   98.5  63.8  54.5  98.5  13.7
  8. AS???    ???                 100.0    10   10    0.0   0.0   0.0   0.0   0.0
  9. AS???    ???                 100.0    10   10    0.0   0.0   0.0   0.0   0.0
 10. AS???    ???                 100.0    10   10    0.0   0.0   0.0   0.0   0.0
 11. AS40509  66.241.124.14        0.0%    10    0   57.1  63.3  56.5  83.7  10.3