Fly apps proxied through Cloudflare may route to incorrect regions.

Questions / Help
1

I started noticing increased latency for certain requests so I took a look at the response headers and noticed that some fly-request-id values are passing through regions much further away. I’m requesting from New Jersey so I expect requests to pass through ewr/lga, but I’m seeing requests go through sea:

Request URL: https://faq.careswitch.com/_content/Radzen.Blazor/css/standard-base.css
Request Method: GET
Status Code: 304 
Remote Address: 172.66.42.232:443
Referrer Policy: strict-origin-when-cross-origin

Response Headers:

alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400
cache-control: public, no-cache
cf-cache-status: MISS
cf-ray: 7438613e4b70e708-EWR
content-type: text/css
date: Wed, 31 Aug 2022 20:05:14 GMT
etag: "1d8bb6423da6768"
fly-request-id: 01GBTR40S2AR84S24XRSZXCNKQ-sea
last-modified: Mon, 29 Aug 2022 04:59:38 GMT
server: cloudflare
vary: Accept-Encoding
via: 1.1 fly.io

You can even notice that it passes through EWR for the Cloudflare proxy, but the fly-request-id suggests it goes through sea, which would explain the increased latency.

The websocket request however goes through the expected region:

Request URL: wss://faq.careswitch.com/_blazor?id=v5L3TFl6SxKiD_hQ7Bp6DA
Request Method: GET
Status Code: 101 Switching Protocols

Response Headers:

alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400
CF-Cache-Status: DYNAMIC
CF-RAY: 743867b2698a1845-EWR
Connection: upgrade
Date: Wed, 31 Aug 2022 20:09:38 GMT
fly-request-id: 01GBTRC2WG0FV61MYTN627P1X6-lga
sec-websocket-accept: RY+uAZ8U+ZagbCUuzAH3UwRmuoM=
Server: cloudflare
upgrade: websocket
via: 1.1 fly.io
2

We’re checking this out on our end, but we might want to also rule out Cloudflare’s role here. They’re directing the traffic as it leaves their network in EWR, but the fly-request-id tells me that it’s hitting our network in a different region.

You might want to check how traffic looks on an endpoint that doesn’t go through them for comparison.

What does it look like when you run curl -IL debug.fly.dev from your client? Since this isn’t behind Cloudflare, that’ll help us pinpoint the issue a bit more.

3

I’m experiencing something similar where all my requests to fly.io go through ewr/lga, even though the closest region I am to is yyz.

4

The debug curl is fine, returns lga as expected, but then again so does the Websocket request that also goes through Cloudflare. The weird thing about this is that only some subset of requests appear to route through the distant sea region, even though all of them pass through Cloudflare.

5

If it helps, here’s a sample request for a similar static asset in a different Fly.io app that routes to the expected region:

Request URL: https://web.careswitch.dev/_content/Radzen.Blazor/css/standard-base.css
Request Method: GET
Status Code: 304 
Remote Address: 104.18.11.119:443
Referrer Policy: strict-origin-when-cross-origin

Response Headers:

alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400
cache-control: public, no-cache
cf-cache-status: MISS
cf-ray: 7438ead0bc6517d9-EWR
content-type: text/css
date: Wed, 31 Aug 2022 21:39:09 GMT
etag: "1d8b845b4056068"
fly-request-id: 01GBTXFZM85TZXBGSW3K7CAXVW-lga
last-modified: Thu, 25 Aug 2022 05:44:12 GMT
server: cloudflare
vary: Accept-Encoding
via: 1.1 fly.io

Contrast it with the similar but problematic request in the other fly.io app:

Request URL: https://faq.careswitch.com/_content/Radzen.Blazor/css/standard-base.css
Request Method: GET
Status Code: 304 
Remote Address: 172.66.41.24:443
Referrer Policy: strict-origin-when-cross-origin

Response Headers:

alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400
cache-control: public, no-cache
cf-cache-status: MISS
cf-ray: 7438ecd40dd58c8d-EWR
content-type: text/css
date: Wed, 31 Aug 2022 21:40:32 GMT
etag: "1d8bb6423da6768"
fly-request-id: 01GBTXJG5XYA2YDMZM8656HHR2-sea
last-modified: Mon, 29 Aug 2022 04:59:38 GMT
server: cloudflare
vary: Accept-Encoding
via: 1.1 fly.io
6

This is entirely up to CloudFlare. I do not know why they are routing some requests to Seattle and some to New Jersey. I think it’s worth asking their support folks if they know what’s up.

7

Appreciate your feedback, I’ll look further into it on Cloudflare’s side. It’s particularly puzzling because cf-ray header shows “EWR” so you’d think it got the right region/data center.

8

Can confirm something weird happening on Cloudflare’s side. Took a look at their HTTP logs and noticed this:

"EdgeColoCode": "EWR",
"EdgeServerIP": "172.71.150.25"

Even though the code appears to be correct, that ip address resolves to a data center in Seattle instead of New Jersey.