Single region but gateway in a different region?

I just moved from Heroku to Fly, and noticed my requests are taking about 10-20x the time — even though my apps are on low CPU and memory usage. Looking at the Grafana dashboard, and following a traceroute for my A record IP, it seems like although all my apps are in Sydney, it gets routed via Singapore

Setting Fly-Prefer-Region: syd doesn’t seem to change this. Can you configure the region of the gateway?

Seems like Fly engs are aware of a possible routing issue affecting AU: Latency > 300ms - on simple health_check actix-web app - #8 by jerome

@amnesthesia if you could provide a traceroute (ideally mtr for a few minutes) to debug.fly.dev, that would help us troubleshoot network issues.

It only hits Singapore when I hit my domain, not when I hit debug.fly.dev, that’s why I thought it was something related to the gateway because it hits Sydney FIRST, then off to Singapore, before reaching my app.

What I find interesting is if I run mtr against fly.io directly, it goes straight to Sydney and does not get routed via Perth → Singapore. This is fly.io:

snake-2.local (192.168.0.106) -> fly.io (37.16.18.81)                                                                                                                                                            2022-11-18T09:19:02+1000
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                                                                                                                                                                                                 Packets               Pings
 Host                                                                                                                                                                                          Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. 192.168.0.1                                                                                                                                                                                77.8%    73    2.1   2.1   1.4   2.7   0.4
 2. 192.168.1.1                                                                                                                                                                                76.4%    73    1.9   2.6   1.9   3.7   0.5
 3. 10.20.26.36                                                                                                                                                                                 0.0%    73   31.8  31.1  30.0  36.9   1.1
 4. bri-apt-wic-wgw1-be-100.tpg.com.au                                                                                                                                                          0.0%    73   32.1  31.5  30.3  34.6   0.8
 5. nme-apt-bur-csw1-te1-1.tpgi.com.au                                                                                                                                                          0.0%    73   32.7  32.6  30.6  58.7   3.5
 6. bundle-ether23.woo-edge903.brisbane.telstra.net                                                                                                                                             0.0%    73   32.0  32.1  30.8  40.3   1.3
 7. bundle-ether12.woo-core10.brisbane.telstra.net                                                                                                                                              0.0%    73   32.6  33.2  31.4  35.4   0.9
 8. bundle-ether20.chw-core10.sydney.telstra.net                                                                                                                                                0.0%    73   45.0  45.8  44.0  49.8   1.1
 9. bundle-ether19.stl-core30.sydney.telstra.net                                                                                                                                                0.0%    73   44.3  44.0  42.4  50.8   1.5
10. bundle-ether1.chw-edge903.sydney.telstra.net                                                                                                                                                0.0%    73   43.6  43.5  41.9  56.4   2.1
11. (waiting for reply)
12. (waiting for reply)
13. 37.16.18.81                                                                                                                                                                                 0.0%    72   43.2  43.6  42.4  53.9   1.6

Here’s MTR for debug.fly.dev, and then my two domains that both hit Singapore, which debug.fly.dev does not:

debug.fly.dev

snake-2.local (192.168.0.106) -> debug.fly.dev (188.93.151.254)                          2022-11-18T09:16:03+1000
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                                                                         Packets               Pings
 Host                                                                  Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. 192.168.0.1                                                        85.1%   149    1.9   2.4   1.4   7.1   1.2
 2. 192.168.1.1                                                        83.8%   149    2.5   2.3   1.6   4.0   0.6
 3. 10.20.26.36                                                         0.0%   149   31.3  31.1  30.0  33.8   0.7
 4. bri-apt-wic-wgw1-be-200.tpg.com.au                                  0.0%   149   31.2  31.6  30.2  36.4   1.0
 5. nme-apt-bur-csw1-te1-1.tpgi.com.au                                  0.0%   149   31.8  32.3  30.9  48.8   1.7
 6. bundle-ether23.woo-edge903.brisbane.telstra.net                     0.0%   149   31.6  31.9  30.5  36.1   0.8
 7. bundle-ether12.woo-core10.brisbane.telstra.net                      0.0%   149   33.7  33.2  31.2  39.8   1.2
 8. bundle-ether20.chw-core10.sydney.telstra.net                        0.0%   149   45.5  46.0  43.7  55.9   1.5
 9. bundle-ether19.stl-core30.sydney.telstra.net                        0.0%   149   43.4  43.9  42.3  49.9   1.0
10. bundle-ether1.chw-edge903.sydney.telstra.net                        0.0%   149   43.6  44.0  42.2  93.4   4.7
11. (waiting for reply)
12. (waiting for reply)
13. 188.93.151.254                                                      0.0%   148   43.2  43.8  42.4  57.3   1.9

api.openmanifest.org

snake-2.local (192.168.0.106) -> api.openmanifest.org (149.248.207.74)                   2022-11-18T09:14:53+1000
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                                                                         Packets               Pings
 Host                                                                  Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. 192.168.0.1                                                        18.2%   198    1.7   2.3   1.3   7.5   1.0
 2. 192.168.1.1                                                        20.7%   198    2.3   2.6   1.7   7.6   1.0
 3. 10.20.26.36                                                         0.0%   198   31.9  31.1  29.9  36.9   0.9
 4. 60-240-241-130.static.tpgi.com.au                                   0.0%   198   31.6  31.6  30.4  38.1   1.1
 5. syd-gls-har-crt2-be20.tpgi.com.au                                   0.0%   198  135.7 135.5 134.0 147.5   1.6
 6. nme-apt-bur-dom1-as16509.tpgi.com.au                                0.0%   198  137.7 137.6 136.3 143.4   1.2
 7. per-apt-stg-crt1-be100.tpgi.com.au                                  0.0%   198  142.1 136.0 134.6 142.1   1.2
 8. sin-eqx-aye-int1-hu-0-0-0-20.tpg.com.au                             0.0%   198  136.0 137.0 135.1 170.6   3.4
 9. (waiting for reply)
10. (waiting for reply)
11. 149.248.207.74                                                      0.0%   197  256.8 257.4 256.3 261.1   0.8

devapi.openmanifest.org:

snake-2.local (192.168.0.106) -> devapi.openmanifest.org (137.66.4.76)                                                                                                                                           2022-11-18T09:15:10+1000
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                                                                                                                                                                                                 Packets               Pings
 Host                                                                                                                                                                                          Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. 192.168.0.1                                                                                                                                                                                86.7%   143    2.0   2.8   1.4   7.1   1.8
 2. 192.168.1.1                                                                                                                                                                                84.5%   143    2.5   2.8   1.8   6.5   1.2
 3. 10.20.26.36                                                                                                                                                                                 0.0%   143   30.9  31.2  29.9  36.7   1.1
 4. 60-240-241-194.static.tpgi.com.au                                                                                                                                                           0.0%   143   31.9  31.6  30.1  42.1   1.2
 5. syd-gls-har-crt2-be20.tpgi.com.au                                                                                                                                                           0.0%   142  137.4 135.8 133.8 163.9   2.9
 6. syd-gls-har-crt3-be-100.tpg.com.au                                                                                                                                                          0.0%   142  134.8 135.2 133.9 141.1   1.2
 7. per-apt-stg-crt1-hu-0-2-0-2.tpg.com.au                                                                                                                                                      0.0%   142  134.1 134.6 133.2 138.9   0.9
 8. sin-eqx-aye-int1-hu-0-0-0-20.tpg.com.au                                                                                                                                                     0.0%   142  135.0 135.3 133.9 140.8   1.1
 9. (waiting for reply)
10. (waiting for reply)
11. 137.66.4.76                                                                                                                                                                                 0.0%   142  258.6 256.3 255.0 271.6   1.7

Did you find anything out from this?