Since about 12:00 UTC on October 2, TLS handshakes for our app at your Frankfurt edge are much slower. From your Prometheus metric fly_edge_tls_handshake_time_seconds, the median went from 350 to 541 ms and p95 from 935 ms to 2.9 s. At the same moment, nrt, iad and gru handshakes got 3–6× faster. Request time at fra is unchanged (~20 ms), and the app answers in 2 ms. Clients are AWS eu-central-1. We use the shared IPv4 66.241.124.200 and a dedicated IPv6. Did anything change in TLS termination or certificate handling at fra around then?
Hi - we serve thousands of apps on each shared IP, so it’s kind of hard to locate an app whose name you didn’t provide based on the IPv4 alone. Can you share the app’s name? (e.g. if it has a public hostname that you set up with a Fly certificate, it’s public anyway and I can look it up by that).
Hi, I also checked our internal p50 times and I don’t see the increase you mention. This is ostensibly the same metric source so I’m wondering why yours differs; are you able to share more data and/or a screenshot of the graph or raw data? Thanks!
Thanks for looking into this, and sorry for leaving out the app. The hostname is conifermatrix.com. In Fly.io land the app is ‘structures’.
My medians were wrong. p50 hasn’t moved, as you said. Most handshakes at fra are still fast. What changed is that a small share now take over a second.
Share of TLS handshakes at fra taking over 1 second:
- Sep 27-29 (UTC): 1,662 handshakes, 6 over 1 second (0.4%)
- Sep 30-Oct 2 12:00: ~1,960 handshakes, 0 over 1 second (0%)
- Oct 3-5: 2,044 handshakes, 112 over 1 second (5.5%)
Over the same Oct 3-5 window, every other edge serving us stays at 0-0.3% (gru 2,701 handshakes, nrt 2,711, iad 2,091).
Timing, in 3-hour buckets: the first slow handshakes appear between 12:00 and 15:00 UTC on Oct 2. They grow to 9-13 per 3 hours by early Oct 3 and keep recurring since.
It also happened before, and not only at fra. From about 00:00 UTC Sep 24, fra had up to about two-thirds of handshakes over 1 second in some 3-hour buckets. nrt only starts appearing in this metric around 21:00 UTC Sep 25, and for its first 9 hours 35–55% of its handshakes took over 1 second (194 of 472). Both edges then dropped to zero at the same time, about 09:00 UTC Sep 26, and fra stayed at zero until Oct 2. Two edges clearing in the same hour makes us think something changed on your side then. Nothing changed on ours: our certificates were issued Sep 6, and there were no IP, DNS or config changes in either window.
Our traffic at fra is low, about 40 handshakes an hour. Most of it is an uptime monitor (AppSignal) making a fresh connection once a minute from AWS eu-central-1. Its total check time went from about 0.8 s to about 2.4 s on Oct 2. Request time at the fra edge is unchanged (~20 ms), and our app answers those requests in ~2 ms (from our own logs).
The graph is two Prometheus queries in Grafana Explore, on our org’s “Prometheus on Fly” data source. Each plots the share of TLS handshakes that took longer than 1 second, per edge, over rolling 3-hour windows. We used the share above 1 s rather than a percentile because fly_edge_tls_handshake_time_seconds has no bucket between 1 s and 10 s, so a p95 inside that range is only an estimate. Edges with 50 or fewer handshakes in a window are hidden to cut out noise.
Query A, fra only:
clamp_min(
100 * (1 - (
sum by (region) (increase(fly_edge_tls_handshake_time_seconds_bucket{app="structures", region="fra", le="1"}[3h]))
/
sum by (region) (increase(fly_edge_tls_handshake_time_seconds_count{app="structures", region="fra"}[3h]))
)),
0)
and on (region)
sum by (region) (increase(fly_edge_tls_handshake_time_seconds_count{app="structures", region="fra"}[3h])) > 50
Query B, every other edge:
clamp_min(
100 * (1 - (
sum by (region) (increase(fly_edge_tls_handshake_time_seconds_bucket{app="structures", region!="fra", le="1"}[3h]))
/
sum by (region) (increase(fly_edge_tls_handshake_time_seconds_count{app="structures", region!="fra"}[3h]))
)),
0)
and on (region)
sum by (region) (increase(fly_edge_tls_handshake_time_seconds_count{app="structures", region!="fra"}[3h])) > 50
Screenshot: Share of TLS handshakes taking over 1 s, per edge, rolling 3-hour windows, app structures. fra is green; nrt (blue) is the Sep 26 spike. Edges with 50 or fewer handshakes per window are hidden. Times are UTC.
Does anything at fra line up with Sep 24–26 and Oct 2 onward? Thanks again for digging in.
two things. Can you run a traceroute from your affected location to your Fly app: mtr -bzo "LSD NABWV" -rw THE_APP_HOST usually does it. And - do you see the same pattern with non-https requests?
TLS is terminated by the Fly proxy at the edges, and none of our metrics shows anything that could explain this slowness on the edges themselves.
That makes sense. We don’t have a box at the location to run mtr from, so we used Globalping. From AWS eu-central-1 (Globalping probes), conifermatrix.com completes the TLS handshake in about 12 ms and answers in about 40 ms, with a clean 4-hop path and no loss ( MTR to conifermatrix.com from aws-eu-central-1 - Globalping ).
On your second question: from AWS eu-central-1 (3 Globalping probes, 30 requests each), plain HTTP and HTTPS were both fast, and none of the 60 requests went over 1 s:
- HTTPS: median 39 ms in total; TLS handshake median 12 ms, worst 16 ms. HTTP HEAD conifermatrix.com from aws-eu-central-1 - Globalping
- HTTP: median 6 ms in total; it’s only the 301 redirect to HTTPS, so there’s no handshake. HTTP HEAD conifermatrix.com from aws-eu-central-1 - Globalping
The roughly 33 ms difference is the TLS handshake plus serving the page itself, against just the redirect. Nothing unusual on either.
Given nearly all our handshakes at fra come from one uptime checker in that region, the slow ones look like that client’s side, not your proxy. We’ll take it up with them. Thank you for helping track this down.
