I just moved from Heroku to Fly, and noticed my requests are taking about 10-20x the time — even though my apps are on low CPU and memory usage. Looking at the Grafana dashboard, and following a traceroute for my A record IP, it seems like although all my apps are in Sydney, it gets routed via Singapore
Setting Fly-Prefer-Region: syd doesn’t seem to change this. Can you configure the region of the gateway?
Seems like Fly engs are aware of a possible routing issue affecting AU: Latency > 300ms - on simple health_check actix-web app - #8 by jerome
@amnesthesia if you could provide a traceroute (ideally mtr for a few minutes) to debug.fly.dev, that would help us troubleshoot network issues.
It only hits Singapore when I hit my domain, not when I hit debug.fly.dev, that’s why I thought it was something related to the gateway because it hits Sydney FIRST, then off to Singapore, before reaching my app.
What I find interesting is if I run mtr against fly.io directly, it goes straight to Sydney and does not get routed via Perth → Singapore. This is fly.io:
snake-2.local (192.168.0.106) -> fly.io (37.16.18.81) 2022-11-18T09:19:02+1000
Keys: Help Display mode Restart statistics Order of fields quit
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. 192.168.0.1 77.8% 73 2.1 2.1 1.4 2.7 0.4
2. 192.168.1.1 76.4% 73 1.9 2.6 1.9 3.7 0.5
3. 10.20.26.36 0.0% 73 31.8 31.1 30.0 36.9 1.1
4. bri-apt-wic-wgw1-be-100.tpg.com.au 0.0% 73 32.1 31.5 30.3 34.6 0.8
5. nme-apt-bur-csw1-te1-1.tpgi.com.au 0.0% 73 32.7 32.6 30.6 58.7 3.5
6. bundle-ether23.woo-edge903.brisbane.telstra.net 0.0% 73 32.0 32.1 30.8 40.3 1.3
7. bundle-ether12.woo-core10.brisbane.telstra.net 0.0% 73 32.6 33.2 31.4 35.4 0.9
8. bundle-ether20.chw-core10.sydney.telstra.net 0.0% 73 45.0 45.8 44.0 49.8 1.1
9. bundle-ether19.stl-core30.sydney.telstra.net 0.0% 73 44.3 44.0 42.4 50.8 1.5
10. bundle-ether1.chw-edge903.sydney.telstra.net 0.0% 73 43.6 43.5 41.9 56.4 2.1
11. (waiting for reply)
12. (waiting for reply)
13. 37.16.18.81 0.0% 72 43.2 43.6 42.4 53.9 1.6
Here’s MTR for debug.fly.dev, and then my two domains that both hit Singapore, which debug.fly.dev does not:
snake-2.local (192.168.0.106) -> debug.fly.dev (188.93.151.254) 2022-11-18T09:16:03+1000
Keys: Help Display mode Restart statistics Order of fields quit
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. 192.168.0.1 85.1% 149 1.9 2.4 1.4 7.1 1.2
2. 192.168.1.1 83.8% 149 2.5 2.3 1.6 4.0 0.6
3. 10.20.26.36 0.0% 149 31.3 31.1 30.0 33.8 0.7
4. bri-apt-wic-wgw1-be-200.tpg.com.au 0.0% 149 31.2 31.6 30.2 36.4 1.0
5. nme-apt-bur-csw1-te1-1.tpgi.com.au 0.0% 149 31.8 32.3 30.9 48.8 1.7
6. bundle-ether23.woo-edge903.brisbane.telstra.net 0.0% 149 31.6 31.9 30.5 36.1 0.8
7. bundle-ether12.woo-core10.brisbane.telstra.net 0.0% 149 33.7 33.2 31.2 39.8 1.2
8. bundle-ether20.chw-core10.sydney.telstra.net 0.0% 149 45.5 46.0 43.7 55.9 1.5
9. bundle-ether19.stl-core30.sydney.telstra.net 0.0% 149 43.4 43.9 42.3 49.9 1.0
10. bundle-ether1.chw-edge903.sydney.telstra.net 0.0% 149 43.6 44.0 42.2 93.4 4.7
11. (waiting for reply)
12. (waiting for reply)
13. 188.93.151.254 0.0% 148 43.2 43.8 42.4 57.3 1.9
snake-2.local (192.168.0.106) -> api.openmanifest.org (149.248.207.74) 2022-11-18T09:14:53+1000
Keys: Help Display mode Restart statistics Order of fields quit
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. 192.168.0.1 18.2% 198 1.7 2.3 1.3 7.5 1.0
2. 192.168.1.1 20.7% 198 2.3 2.6 1.7 7.6 1.0
3. 10.20.26.36 0.0% 198 31.9 31.1 29.9 36.9 0.9
4. 60-240-241-130.static.tpgi.com.au 0.0% 198 31.6 31.6 30.4 38.1 1.1
5. syd-gls-har-crt2-be20.tpgi.com.au 0.0% 198 135.7 135.5 134.0 147.5 1.6
6. nme-apt-bur-dom1-as16509.tpgi.com.au 0.0% 198 137.7 137.6 136.3 143.4 1.2
7. per-apt-stg-crt1-be100.tpgi.com.au 0.0% 198 142.1 136.0 134.6 142.1 1.2
8. sin-eqx-aye-int1-hu-0-0-0-20.tpg.com.au 0.0% 198 136.0 137.0 135.1 170.6 3.4
9. (waiting for reply)
10. (waiting for reply)
11. 149.248.207.74 0.0% 197 256.8 257.4 256.3 261.1 0.8
snake-2.local (192.168.0.106) -> devapi.openmanifest.org (137.66.4.76) 2022-11-18T09:15:10+1000
Keys: Help Display mode Restart statistics Order of fields quit
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. 192.168.0.1 86.7% 143 2.0 2.8 1.4 7.1 1.8
2. 192.168.1.1 84.5% 143 2.5 2.8 1.8 6.5 1.2
3. 10.20.26.36 0.0% 143 30.9 31.2 29.9 36.7 1.1
4. 60-240-241-194.static.tpgi.com.au 0.0% 143 31.9 31.6 30.1 42.1 1.2
5. syd-gls-har-crt2-be20.tpgi.com.au 0.0% 142 137.4 135.8 133.8 163.9 2.9
6. syd-gls-har-crt3-be-100.tpg.com.au 0.0% 142 134.8 135.2 133.9 141.1 1.2
7. per-apt-stg-crt1-hu-0-2-0-2.tpg.com.au 0.0% 142 134.1 134.6 133.2 138.9 0.9
8. sin-eqx-aye-int1-hu-0-0-0-20.tpg.com.au 0.0% 142 135.0 135.3 133.9 140.8 1.1
9. (waiting for reply)
10. (waiting for reply)
11. 137.66.4.76 0.0% 142 258.6 256.3 255.0 271.6 1.7
Did you find anything out from this?