Abnormally slow SSL handshake resulting in slow server responses across Fly apps?

[uncvrd]

Over the past month I’ve started to experience some really strange & slow response times across my applications. I’ve been digging about, and in chrome dev tools, I see that SSL handshake is taking up about 1 second. I’ve seen numerous threads about this but nothing from Fly regarding a resolution. See:

TLS/SSL handshake very slow

I am trying to understand how reliable Fly.io is, as we are seeing constant multi-second TLS handshake times for our machines. I don’t observe any errors in the logs. [05-08-24 23-57-15] App URL: https://dashboard-fe--staging.fly.dev/ Config: primary_region = 'waw' [build] [http_service] internal_port = 3000 force_https = true auto_stop_machines = "suspend" auto_start_machines = true min_machines_running = 1 processes = ['app'] [[http_service.checks]] grace_period = "10s" …

Really slow responses from Remix JS App on Fly.io (CDG)

I’m still facing the issue, recreating the machines did not work. I deployed the app machine to AMS without any luck either, probably because the DB and Redis are still in CDG. Can we have an update from fly.io’s team?

Application slow for a single user... how? - #16 by PeterCxy (some support but closed…)

SSL & Fly Latency Issues - Tested with https://debug.fly.dev/

On a fresh browser session, it always takes about 1-1.5 seconds for the SSL connection to complete. Below is my https://debug.fly.dev/ output along with several screenshots of the ssl timing. Is there anyway we can improve this as i see the same issue on my deployed apps (i have 4 apps and counting)? === Headers === Host: debug.fly.dev X-Request-Start: t=1689679128847728 Sec-Fetch-Dest: document X-Forwarded-Ssl: on Accept-Encoding: gzip, deflate, br X-Forwarded-For: ******,******* X-Forwarded-…

I (near lax) have been testing an app hosted in iad and I see simple Fastify api response times (returning a boolean value, no computation) taking upwards of 1-2 seconds to respond.

My domain is on Cloudflare so I added A/AAAA and CNAME records pointing to Fly from my Cloudflare dashboard (all with the CF proxy turned off).

Here is an example response waterfall

When doing the following simple request

When I check debug.fly.dev, the response shows that my region is sin?

You can also see my handshakes are occurring in sin too? Why?

When I run traceroute directly from my fly.dev URL I see

traceroute to foundry-staging-trpc-web.fly.dev (66.241.124.45), 64 hops max, 40 byte packets
 1  192.168.1.254 (192.168.1.254)  1.331 ms  1.111 ms  0.970 ms
 2  172.8.144.1 (172.8.144.1)  1.939 ms  2.179 ms  1.965 ms
 3  64.148.105.112 (64.148.105.112)  2.011 ms  3.690 ms  2.422 ms
 4  12.243.128.102 (12.243.128.102)  10.878 ms  8.497 ms  7.964 ms
 5  12.122.128.101 (12.122.128.101)  12.017 ms  6.073 ms  3.879 ms
 6  192.205.37.26 (192.205.37.26)  10.925 ms  4.808 ms  4.303 ms
 7  96.110.44.157 (96.110.44.157)  12.777 ms
    96.110.44.153 (96.110.44.153)  11.958 ms
    96.110.44.149 (96.110.44.149)  11.770 ms
 8  96.110.33.66 (96.110.33.66)  11.479 ms
    96.110.33.78 (96.110.33.78)  12.501 ms
    96.110.33.74 (96.110.33.74)  12.422 ms
 9  75.149.231.130 (75.149.231.130)  11.240 ms  4.684 ms  3.536 ms
10  * * *
11  * * *
12  * * *
13  * * *

And when I use my Cloudflare domain

traceroute trpc-web.foundry-staging.xyz
traceroute to trpc-web.foundry-staging.xyz (66.241.124.45), 64 hops max, 40 byte packets
 1  dsldevice (192.168.1.254)  6.307 ms  7.047 ms  4.535 ms
 2  172-8-144-1.lightspeed.irvnca.sbcglobal.net (172.8.144.1)  5.559 ms  27.707 ms  5.586 ms
 3  64.148.105.112 (64.148.105.112)  6.074 ms  27.640 ms  7.633 ms
 4  12.243.128.102 (12.243.128.102)  12.185 ms  29.933 ms  10.205 ms
 5  ggr2.la2ca.ip.att.net (12.122.128.101)  6.935 ms  11.166 ms  8.003 ms
 6  192.205.37.26 (192.205.37.26)  6.533 ms  7.477 ms  226.267 ms
 7  be-3202-cs02.losangeles.ca.ibone.comcast.net (96.110.44.149)  7.321 ms
    be-3102-cs01.losangeles.ca.ibone.comcast.net (96.110.44.145)  7.867 ms
    be-3302-cs03.losangeles.ca.ibone.comcast.net (96.110.44.153)  8.419 ms
 8  be-3212-pe12.600wseventh.ca.ibone.comcast.net (96.110.33.70)  10.350 ms  11.321 ms
    be-3112-pe12.600wseventh.ca.ibone.comcast.net (96.110.33.66)  10.309 ms
 9  75.149.231.130 (75.149.231.130)  6.918 ms  7.279 ms  10.476 ms
10  * * *
11  * * *
12  * * *

This definitely wasn’t a thing about a month or so ago, any way we can find a way to debug this problem? Let me know what else I can provide…Thanks!

[PeterCxy]

Hi @uncvrd, it looks like your browser was trying to connect to your fly.dev domain over IPv6 and that was routed to sin for some reason. Are you able to perform another traceroute over IPv6? The IPv4 route looks okay, but something funky might be happening over IPv6 that routed you to sin.

[PeterCxy]

Hi @uncvrd, to provide an update here: it turns out that one of the major backbone networks started to (incorrectly) route some of our traffic from the US to APAC. We applied a workaround to this by stopping to advertise our IPs to that problematic network. As of now, it seems that most US traffic is routed correctly, but we have no great visibility into ATT’s network.

If you are still seeing this issue, a traceroute6 from your side would still help a lot. In addition to the IP of your fly.dev domain, could you please also try tracerouting to the following IPs: 2a09:8280:e632:: and 2a09:8280:e621:: – these are (respectively) IPs in the sin and lax region, and although they won’t respond to pings, hops in the middle as shown by traceroute will help us understand why you are being routed to sin.

[uncvrd]

@PeterCxy WOW, thank you. Yes this has cut down my response time from 1.61s to 0.174s. Phew I thought I was going crazy.

For the sake of providing additional information, here is the requested:

traceroute6 foundry-staging-trpc-web.fly.dev
traceroute6 to foundry-staging-trpc-web.fly.dev (2a09:8280:1::24:d816) from 2600:1700:4641:21e0:1c44:db14:d76b:8f11, 64 hops max, 28 byte packets
 1  * * *
 2  2001:506:6000:131:69:235:127:184  10.573 ms  2.312 ms  2.255 ms
 3  2001:506:6000:131:71:156:220:238  10.473 ms  1.895 ms  1.734 ms
 4  * * *
 5  ix-ae-15-0.tcore1.lvw-losangeles.ipv6.as6453.net  11.733 ms  14.858 ms  45.140 ms
 6  *
    if-bundle-33-2.qcore1.lvw-losangeles.ipv6.as6453.net  11.237 ms  3.228 ms
 7  if-ae-35-2.tcore1.sv1-santaclara.ipv6.as6453.net  21.642 ms  14.010 ms  13.749 ms
 8  if-ae-0-2.tcore2.sv1-santaclara.ipv6.as6453.net  21.514 ms  15.341 ms  15.166 ms
 9  if-bundle-20-2.qcore1.sqn-sanjose.ipv6.as6453.net  21.440 ms *  13.180 ms
10  2001:5a0:3e02:130::1  22.173 ms  14.359 ms  14.161 ms
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * *^C

IPv4/6 of fly.dev domain: 66.241.124.45 and 2a09:8280:1::24:d816

Traceroute for 2a09:8280:e632::

traceroute6 2a09:8280:e632::
traceroute6 to 2a09:8280:e632:: (2a09:8280:e632::) from 2600:1700:4641:21e0:1c44:db14:d76b:8f11, 64 hops max, 28 byte packets
 1  * * *
 2  2001:506:6000:131:69:235:127:184  2.498 ms  2.844 ms  2.054 ms
 3  2001:506:6000:131:71:156:220:238  2.357 ms  2.975 ms  2.261 ms
 4  2001:1890:ff:ff06:12:242:115:44  18.643 ms *  8.084 ms
 5  ix-ae-15-0.tcore1.lvw-losangeles.ipv6.as6453.net  2.872 ms  3.251 ms  3.978 ms
 6  if-ae-2-2.tcore2.lvw-losangeles.ipv6.as6453.net  12.024 ms  3.044 ms  3.053 ms
 7  if-ae-28-2.tcore2.av3-toyohashi.ipv6.as6453.net  122.169 ms  111.674 ms  112.998 ms
 8  if-ae-2-2.tcore1.av3-toyohashi.ipv6.as6453.net  119.852 ms  115.369 ms  113.758 ms
 9  if-bundle-55-2.qcore2.esin4-singapore.ipv6.as6453.net  195.356 ms  187.871 ms  188.330 ms
10  if-bundle-2-2.qcore1.esin4-singapore.ipv6.as6453.net  196.854 ms  188.784 ms *
11  * * *
12  2405:2000:2a00:30::12  218.999 ms  193.332 ms  193.504 ms
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *

Traceroute for 2a09:8280:e621::

traceroute6 2a09:8280:e621::
traceroute6 to 2a09:8280:e621:: (2a09:8280:e621::) from 2600:1700:4641:21e0:1c44:db14:d76b:8f11, 64 hops max, 28 byte packets
 1  * * *
 2  2001:506:6000:131:69:235:127:184  3.319 ms  2.644 ms  2.220 ms
 3  2001:506:6000:131:71:156:220:238  1.928 ms  2.561 ms  2.677 ms
 4  la2ca402igs.ipv6.att.net  12.444 ms  5.114 ms  6.569 ms
 5  att-gw.la2ca.comcast.net  13.003 ms  5.230 ms  4.780 ms
 6  be-3202-cs02.losangeles.ca.ibone.comcast.net  13.377 ms
    be-3402-cs04.losangeles.ca.ibone.comcast.net  13.155 ms
    be-3102-cs01.losangeles.ca.ibone.comcast.net  11.642 ms
 7  be-3312-pe12.600wseventh.ca.ibone.comcast.net  13.405 ms
    be-3412-pe12.600wseventh.ca.ibone.comcast.net  12.199 ms
    be-3212-pe12.600wseventh.ca.ibone.comcast.net  12.344 ms
 8  2001:559:0:9::3a  12.916 ms  4.612 ms  4.516 ms
 9  * * *
10  * * *
11  * * *
12  * * *

Thanks a lot for the quick workaround regardless, hope this is helpful