I’m in YYZ and my requests should be going through YYZ. However, I’m seeing in my responses with the following response header: Fly-Request-Id: 01HP0HD0XRSP3FFEZY5AAVMWJV-syd
Am I getting routed through Sydney?
Seems similar to what happened a few months ago where requests were getting routed through MEL?
Hi,
I am getting consistently slow response times from both my apps since yesterday. Nothing changed on my end so I am puzzled about why this suddenly started happening. I checked my app and the DB are still on the same region (and I am requesting from the same region) and put server timings on a route to check if it was the application but it is reporting 18ms response time. So unsure where this extra 800ms is coming in:
[Screenshot 2023-10-17 at 17.04.35]
My setup is NodeJS to Postgres and…
Also, https://debug.fly.dev is giving me the following
=== Headers ===
Host: debug.fly.dev
Accept-Encoding: gzip, deflate, br
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7
Fly-Tracestate:
X-Forwarded-For: 142.120.19.190, 77.83.140.164
X-Forwarded-Proto: https
Accept-Language: en-US,en;q=0.9,en-GB;q=0.8,en-CA;q=0.7,fr;q=0.6,da;q=0.5
Fly-Forwarded-Ssl: on
Fly-Client-Ip: 142.120.19.190
Fly-Region: syd
User-Agent: Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Mobile Safari/537.36
Sec-Fetch-Mode: navigate
Sec-Fetch-Dest: document
X-Forwarded-Port: 443
Fly-Traceparent: 00-2215343b22d8a1d9d18edd0ba1d0f5f9-df1f7a9aa9e90f8f-00
Sec-Ch-Ua: "Not A(Brand";v="99", "Google Chrome";v="121", "Chromium";v="121"
Sec-Ch-Ua-Mobile: ?1
Sec-Fetch-Site: none
Sec-Fetch-User: ?1
Fly-Forwarded-Proto: https
X-Forwarded-Ssl: on
Fly-Forwarded-Port: 443
Via: 2 fly.io
Sec-Ch-Ua-Platform: "Android"
X-Request-Start: t=1707269087827710
Fly-Request-Id: 01HP0JNRJKFJK5B54FC5QVY4JT-syd
2024-02-07 01:24:47.832287318 +0000 UTC m=+1305323.928637593
Ping @JP_Phillips
Also, if I use my phone over LTE, I get routed properly through YYZ. Even VPN forces YYZ. Without VPN on my home internet, I get routed the SYD. My home internet is Virgin (on Bell fiber)
Thanks!
Yep can confirm seeing the same issue as before again but this time https://debug.fly.dev/ is responding with “Fly-Region: syd” instead of “mel”. I am on Bell in Canada.
Hopefully the fix is the same as what was done last time to correct the network routing.
For reference this was the previous incident report: Fly.io Status - Requests incorrectly routing to mel region
3 Likes
@stephen1 , @tj1 , @cravingsleep can you please attach a traceroute (mtr -4bzo "LSD NABWV" -rw debug.fly.dev).
sudo mtr -4bzo "LSD NABWV" -rw debug.fly.dev
Start: 2024-02-07T07:06:14-0500
HOST: Stephens-MBP Loss% Snt Drop Last Avg Best Wrst StDev
1. AS??? mynetwork.home (192.168.2.1) 0.0% 10 0 2.7 3.0 2.0 6.9 1.4
2. AS??? 64.230.11.214 0.0% 10 0 5.0 8.5 3.8 25.0 6.7
3. AS??? 64.230.97.186 0.0% 10 0 23.7 24.1 23.5 26.7 0.9
4. AS??? 64.230.51.157 0.0% 10 0 28.0 27.4 24.1 31.8 2.6
5. AS??? 142.124.127.197 0.0% 10 0 23.9 24.8 23.9 25.4 0.6
6. AS??? 142.124.127.135 0.0% 10 0 22.8 24.5 22.8 25.6 0.9
7. AS??? 142.124.127.77 0.0% 10 0 23.2 23.4 22.8 24.2 0.4
8. AS46887 ash-eqx-01gw.voxility.net (206.126.236.169) 0.0% 10 0 23.4 23.1 22.4 23.7 0.4
9. AS3223 ash-eqx-01c.voxility.net (5.254.81.22) 0.0% 10 0 25.2 28.9 23.4 62.3 12.1
10. AS3223 5.254.81.130 0.0% 10 0 23.4 23.1 22.0 23.8 0.5
11. AS36236 245.10.225.104.ptr.anycast.net (104.225.10.245) 0.0% 10 0 25.1 25.3 24.9 26.1 0.4
12. AS??? ??? 100.0 10 10 0.0 0.0 0.0 0.0 0.0
13. AS??? ??? 100.0 10 10 0.0 0.0 0.0 0.0 0.0
14. AS??? ??? 100.0 10 10 0.0 0.0 0.0 0.0 0.0
15. AS40509 77.83.140.164 0.0% 10 0 29.2 30.1 28.9 34.1 1.5
FWIW, I’m getting different output on debug.fly.dev this morning (i.e. it’s taking me to CHI…which isn’t nearly as far but still not YYZ as it should be)
Don’t know if it makes a difference, but ran it on my app instead of debug.fly.dev and got the following (including what looks like a trip to Australia):
Start: 2024-02-07T07:09:42-0500
HOST: Stephens-MBP Loss% Snt Drop Last Avg Best Wrst StDev
1. AS??? mynetwork.home (192.168.2.1) 0.0% 10 0 4.4 3.3 1.8 6.6 1.4
2. AS??? 64.230.11.214 0.0% 10 0 3.8 6.8 3.8 12.6 2.9
3. AS??? 64.230.97.186 50.0% 10 5 18.9 19.8 18.7 21.1 1.0
4. AS??? 64.230.51.157 0.0% 10 0 26.9 23.9 18.3 26.9 2.7
5. AS??? 142.124.127.197 0.0% 10 0 20.5 20.3 19.2 22.2 0.8
6. AS??? ??? 100.0 10 10 0.0 0.0 0.0 0.0 0.0
7. AS??? 64.230.79.75 0.0% 10 0 17.6 18.9 16.5 27.7 3.3
8. AS??? ??? 100.0 10 10 0.0 0.0 0.0 0.0 0.0
9. AS??? ??? 100.0 10 10 0.0 0.0 0.0 0.0 0.0
10. AS??? ??? 100.0 10 10 0.0 0.0 0.0 0.0 0.0
11. AS6461 ae15.cs4.slc2.us.zip.zayo.com (64.125.22.176) 80.0% 10 8 73.8 73.5 73.3 73.8 0.4
12. AS6461 ae4.cs2.lax112.us.zip.zayo.com (64.125.23.239) 20.0% 10 2 73.5 74.5 73.0 79.3 2.1
13. AS??? ??? 100.0 10 10 0.0 0.0 0.0 0.0 0.0
14. AS6461 64.125.35.202.ipyx-145641-004-zyo.zip.zayo.com (64.125.35.202) 0.0% 10 0 66.1 65.5 64.6 66.5 0.6
15. AS7473 203.208.171.117 0.0% 10 0 66.8 66.5 65.0 70.1 1.5
16. AS7473 203.208.149.249 0.0% 10 0 66.3 67.3 64.6 81.6 5.1
17. AS7473 203.208.172.233 0.0% 10 0 64.8 65.9 64.4 70.2 1.7
18. AS7473 203.208.190.18 0.0% 10 0 221.7 221.3 220.3 223.0 0.8
19. AS7474 124.19.61.240 0.0% 10 0 219.9 220.8 219.6 224.2 1.3
20. AS7474 cranegrouplimited.22btc76f000.optus.net.au (59.154.3.38) 0.0% 10 0 226.5 226.9 225.8 228.3 0.7
21. AS??? ??? 100.0 10 10 0.0 0.0 0.0 0.0 0.0
22. AS9507 103.66.132.69 0.0% 10 0 225.5 225.6 223.5 234.1 3.1
23. AS40509 213.188.218.30 0.0% 10 0 227.5 226.9 225.9 228.2 0.7
1 Like
tj1
February 7, 2024, 2:51pm
6
Run1
$ mtr -4bzo "LSD NABWV" -rw debug.fly.dev
Start: 2024-02-07T20:16:02+0530
HOST: void0 Loss% Snt Drop Last Avg Best Wrst StDev
1. AS??? 192.168.91.93 0.0% 10 0 2.4 3.0 1.8 9.0 2.1
2. AS??? 192.168.29.10 0.0% 10 0 175.3 194.7 118.8 335.6 70.6
3. AS??? 192.168.28.49 0.0% 10 0 77.8 85.8 31.2 313.8 84.9
4. AS??? 192.168.31.19 0.0% 10 0 41.4 60.8 25.4 283.8 78.9
5. AS??? 192.168.31.33 0.0% 10 0 61.0 61.8 23.7 222.6 57.6
6. AS??? ??? 100.0 10 10 0.0 0.0 0.0 0.0 0.0
7. AS9498 nsg-corporate-237.73.186.122.airtel.in (122.186.73.237) 0.0% 10 0 86.7 88.4 42.6 298.1 75.7
8. AS9498 116.119.112.88 0.0% 10 0 170.0 181.7 151.7 339.6 56.7
9. AS??? fra-eq5-01gw.voxility.net (80.81.195.144) 0.0% 10 0 210.1 221.3 171.7 372.2 66.8
10. AS3223 fra-in3-01gw.voxility.net (5.254.88.26) 0.0% 10 0 193.8 233.5 180.0 531.8 106.2
11. AS3223 fra-in8-01c.voxility.net (5.254.88.2) 0.0% 10 0 189.1 228.9 174.9 465.0 92.3
12. AS36236 178.200.178.185.ptr.anycast.net (185.178.200.178) 0.0% 10 0 205.9 244.1 183.9 423.3 89.3
13. AS??? ??? 100.0 10 10 0.0 0.0 0.0 0.0 0.0
14. AS??? ??? 100.0 10 10 0.0 0.0 0.0 0.0 0.0
15. AS40509 77.83.140.164 0.0% 10 0 464.4 251.4 168.4 464.4 113.6
Run2
Start: 2024-02-07T20:19:40+0530
HOST: void0 Loss% Snt Drop Last Avg Best Wrst StDev
1. AS??? 192.168.91.93 0.0% 10 0 1.8 3.5 1.8 8.4 2.4
2. AS??? 192.168.29.10 10.0% 10 1 118.0 268.7 54.7 1273. 381.5
3. AS??? 192.168.28.53 10.0% 10 1 53.2 185.1 26.3 1162. 369.9
4. AS??? 192.168.31.23 10.0% 10 1 71.7 171.4 30.0 1124. 359.0
5. AS??? 192.168.31.49 20.0% 10 2 40.7 170.8 29.3 1066. 362.3
6. AS??? ??? 100.0 10 10 0.0 0.0 0.0 0.0 0.0
7. AS9498 nsg-corporate-237.73.186.122.airtel.in (122.186.73.237) 10.0% 10 1 30.2 137.4 30.2 948.5 304.2
8. AS9498 116.119.57.158 10.0% 10 1 99.9 208.9 97.2 953.4 280.1
9. AS7473 203.208.148.89 10.0% 10 1 111.8 212.9 111.8 912.5 262.7
10. AS7473 203.208.153.94 10.0% 10 1 107.7 204.2 99.1 841.9 241.4
11. AS4845 7473203.208.158.10 10.0% 10 1 99.8 183.3 94.8 782.8 225.4
12. AS7473 203.208.169.90 10.0% 10 1 163.1 246.5 152.2 782.3 202.9
13. AS7474 124.19.61.63 10.0% 10 1 175.8 224.4 142.0 716.6 184.9
14. AS7474 59.154.58.74 0.0% 10 0 149.5 361.3 149.5 1587. 456.7
15. AS??? 103.251.4.7 0.0% 10 0 255.3 621.2 198.1 2420. 769.2
16. AS9507 103.66.132.69 0.0% 10 0 207.7 591.1 188.7 2361. 750.4
17. AS40509 37.16.21.10 10.0% 10 1 203.4 376.8 186.3 1473. 424.9
Curl seems to bounce between fra and syd.
$ curl -4 -I https://debug.fly.dev -H "flyio-debug: doit"
HTTP/2 200
fly-region: syd
remote-addr: 172.16.0.2:63784
date: Wed, 07 Feb 2024 14:47:41 GMT
content-length: 552
content-type: text/plain; charset=utf-8
server: Fly/ba9e227a (2024-01-26)
via: 2 fly.io
flyio-debug: {"n":"edge-cf-syd1-c895","nr":"syd","ra":"106.208.150.222","rf":"Verbatim","sr":"syd","sdc":"sy4","sid":"24d89d029a6879","st":0,"nrtt":0,"bn":"worker-pkt-sy4-8fbc"}
fly-request-id: 01HP20KWYZ4B0HF2XFA8RJ04EF-syd
$ curl -4 -I https://debug.fly.dev
HTTP/2 200
fly-region: fra
remote-addr: 172.16.129.90:61882
date: Wed, 07 Feb 2024 14:48:38 GMT
content-length: 555
content-type: text/plain; charset=utf-8
server: Fly/ba9e227a (2024-01-26)
via: 2 fly.io
fly-request-id: 01HP20NN3A9BCA57T95APZEZ3J-fra
system
February 14, 2024, 2:52pm
7
This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.