Global anycast routed to AMS

Hello, I am trying to host a DNS app and was testing the latency.

On the anycast network I am routed to AMS on both IPv4 and IPv6.
I am from Sri Lanka and I have better latencies to SIN. (around ~80ms)

Is it possible to change the region?

Below is an MTR to debug.fly.dev

|------------------------------------------------------------------------------------------|
|                                      WinMTR statistics                                   |
|                       Host              -   %  | Sent | Recv | Best | Avrg | Wrst | Last |
|------------------------------------------------|------|------|------|------|------|------|
| 2402:4000:2081:b82a:****:****:****:**** -    0 |    2 |    2 |    1 |    1 |    1 |    1 |
|                      Request timed out. -  100 |    1 |    0 |    0 |    0 |    0 |    0 |
|                        2404:f000:5:4::2 -    0 |    2 |    2 |   43 |   43 |   44 |   44 |
|                          2404:f000:8::a -    0 |    2 |    2 |   45 |   47 |   49 |   45 |
|                        2404:f000:5:1::1 -    0 |    2 |    2 |   31 |   34 |   37 |   31 |
|                      2404:f000:0:200::1 -    0 |    2 |    2 |   37 |   43 |   49 |   49 |
|                       amsix.as36236.net -    0 |    2 |    2 |  182 |  193 |  205 |  182 |
|                       2607:f740:d:10::4 -    0 |    2 |    2 |  200 |  219 |  238 |  238 |
|                       2607:f740:d:16::2 -    0 |    2 |    2 |  187 |  196 |  205 |  187 |
|    2a09:8280:1:763f:8bdd:34d1:c624:78cd -    0 |    2 |    2 |  212 |  214 |  217 |  217 |
|________________________________________________|______|______|______|______|______|______|

Thanks.

Can you try this with IPv4? I’m curious if IPv6 and IPv4 are routing differently.

it looks like the same route.

|------------------------------------------------------------------------------------------|
|                                      WinMTR statistics                                   |
|                       Host              -   %  | Sent | Recv | Best | Avrg | Wrst | Last |
|------------------------------------------------|------|------|------|------|------|------|
|                          homerouter.cpe -    0 |   13 |   13 |    0 |    1 |    5 |    1 |
|                      Request timed out. -  100 |   13 |    0 |    0 |    0 |    0 |    0 |
|                             10.174.66.1 -    0 |   13 |   13 |   23 |   68 |  103 |   68 |
|                      Request timed out. -  100 |   13 |    0 |    0 |    0 |    0 |    0 |
|                         125.214.190.221 -    0 |   13 |   13 |   21 |   30 |   55 |   21 |
|                          125.214.162.65 -    0 |   13 |   13 |   23 |   31 |   45 |   30 |
|                      Request timed out. -  100 |   13 |    0 |    0 |    0 |    0 |    0 |
|                      Request timed out. -  100 |   13 |    0 |    0 |    0 |    0 |    0 |
|                      Request timed out. -  100 |   13 |    0 |    0 |    0 |    0 |    0 |
|                           77.83.140.164 -    0 |   13 |   13 |  182 |  200 |  217 |  217 |
|________________________________________________|______|______|______|______|______|______|
=== Headers ===
Host: 77.83.140.164
Dnt: 1
Fly-Forwarded-Port: 80
Fly-Region: ams
X-Forwarded-Ssl: off
X-Forwarded-Port: 80
Fly-Request-Id: 01FD6PVQA84YNZC9KSR7ZR5RWK
Accept-Language: en-US,en;q=0.9
X-Request-Start: t=1629091650888535
X-Forwarded-For: 116.206.247.***, 77.83.140.164
X-Forwarded-Proto: http
Accept-Encoding: gzip, deflate
Fly-Client-Ip: 116.206.247.179
Fly-Dispatch-Start: t=1629091650888764;instance=bbfc00e2
Via: 1.1 fly.io
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Fly-Forwarded-Proto: http
Fly-Forwarded-Ssl: off

=== ENV ===
FLY_ALLOC_ID=bbfc00e2-e771-47c1-2527-624687341259
FLY_APP_NAME=debug
FLY_PUBLIC_IP=2607:f740:d:26:0:bbfc:e2:1
FLY_REGION=ams
FLY_VM_MEMORY_MB=128
HOME=/root
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
TERM=linux
WS=this
is
a
test
cgroup_enable=memory

2021-08-16 05:27:30.889793945 +0000 UTC m=+652759.872006820

any update?

Looks like we forgot to follow through. I’m looking into this now.

1 Like
Start: 2022-06-02T00:29:58+0530
HOST: miyuru-fedora                           Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 2402:4000:2181:ea38:****:****:****:****  0.0%     1    1.1   1.1   1.1   1.1   0.0
  2.|-- ???                                     100.0     1    0.0   0.0   0.0   0.0   0.0
  3.|-- 2404:f000:4:4::2                         0.0%     1   34.6  34.6  34.6  34.6   0.0
  4.|-- 2404:f000:8:4000::a                      0.0%     1   35.1  35.1  35.1  35.1   0.0
  5.|-- 2404:f000:4:1::1                         0.0%     1   23.6  23.6  23.6  23.6   0.0
  6.|-- 2404:f000:0:100::1                       0.0%     1   36.8  36.8  36.8  36.8   0.0
  7.|-- 2404:f000:0:a01::2                       0.0%     1   19.6  19.6  19.6  19.6   0.0
  8.|-- 2404:f000:0:a19::1                       0.0%     1   27.4  27.4  27.4  27.4   0.0
  9.|-- 2001:de8:4::3:6236:1                     0.0%     1   79.3  79.3  79.3  79.3   0.0
 10.|-- 2403:2500:300:1::5                       0.0%     1   62.9  62.9  62.9  62.9   0.0
 11.|-- 2403:2500:300:5::2                       0.0%     1   63.0  63.0  63.0  63.0   0.0
 12.|-- 2a09:8280:70::5                          0.0%     1  152.4 152.4 152.4 152.4   0.0
Start: 2022-06-02T00:30:22+0530
HOST: miyuru-fedora   Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- _gateway         0.0%     1    4.2   4.2   4.2   4.2   0.0
  2.|-- ???             100.0     1    0.0   0.0   0.0   0.0   0.0
  3.|-- ???             100.0     1    0.0   0.0   0.0   0.0   0.0
  4.|-- ???             100.0     1    0.0   0.0   0.0   0.0   0.0
  5.|-- 125.214.190.148  0.0%     1   29.1  29.1  29.1  29.1   0.0
  6.|-- 125.214.162.65   0.0%     1   33.0  33.0  33.0  33.0   0.0
  7.|-- 80.249.210.142   0.0%     1  183.3 183.3 183.3 183.3   0.0
  8.|-- ???             100.0     1    0.0   0.0   0.0   0.0   0.0
  9.|-- 104.225.98.9     0.0%     1  186.1 186.1 186.1 186.1   0.0
 10.|-- 37.16.16.5       0.0%     1  183.2 183.2 183.2 183.2   0.0
1 Like

@jerome IPv6 traffic now routes singapore :partying_face:, IPv4 is the same as before.

I only care about the IPv6 traffic so, if this is pushed to prod I will consider this issue solved :grin:. that being said if you need any more IPv4 traceroutes to improve your services I will be happy to provide them.

Start: 2022-06-03T15:41:06+0530
HOST: miyuru-fedora                           Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 2402:4000:2382:6765::****::****::****:****  0.0%     1    1.1   1.1   1.1   1.1   0.0
  2.|-- ???                                     100.0     1    0.0   0.0   0.0   0.0   0.0
  3.|-- 2404:f000:8:c005::1                      0.0%     1   55.7  55.7  55.7  55.7   0.0
  4.|-- 2404:f000:8:c100::90f                    0.0%     1   20.8  20.8  20.8  20.8   0.0
  5.|-- 2404:f000:0:2::c6                        0.0%     1   32.6  32.6  32.6  32.6   0.0
  6.|-- 2404:f000:0:a14::1                       0.0%     1   40.6  40.6  40.6  40.6   0.0
  7.|-- 2404:f000:0:a20::1                       0.0%     1   41.3  41.3  41.3  41.3   0.0
  8.|-- 2404:f000:0:a21::1                       0.0%     1   41.9  41.9  41.9  41.9   0.0
  9.|-- 2001:de8:4::3:6236:1                     0.0%     1   60.2  60.2  60.2  60.2   0.0
 10.|-- 2403:2500:300:1::5                       0.0%     1   63.5  63.5  63.5  63.5   0.0
 11.|-- 2403:2500:300:5::2                       0.0%     1   64.9  64.9  64.9  64.9   0.0
 12.|-- 2a09:8280:70::5                          0.0%     1   65.6  65.6  65.6  65.6   0.0
1 Like

Glad this is better now!

An IPv4 trace would be nice, we need to fix this for everyone :slight_smile:

I experience something similar, connecting from Denmark and should be routed through AMS or FRA (or PAR), but from my current IP (in the debug output below) I consistently get routed through Hong Kong (~300 ms latency).

Here’s output from debug.fly.dev:

=== Headers ===
Host: debug.fly.dev
Fly-Request-Id: 01G6ABSE23JXV5ZJPR78K6QRR6-hkg
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Fly-Client-Ip: 185.45.22.146
X-Forwarded-For: 185.45.22.146, 77.83.140.164
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.5 Safari/605.1.15
X-Forwarded-Proto: https
Fly-Forwarded-Ssl: on
Fly-Forwarded-Port: 443
Accept-Language: en-GB,en;q=0.9
Fly-Forwarded-Proto: https
X-Forwarded-Ssl: on
X-Forwarded-Port: 443
Fly-Region: hkg
Via: 2 fly.io
Accept-Encoding: gzip, deflate, br
X-Request-Start: t=1656057804867689

=== ENV ===
FLY_ALLOC_ID=56158aac-fa2c-6a90-2f23-17579210c502
FLY_APP_NAME=debug
FLY_PUBLIC_IP=2605:4c40:95:4eed:0:5615:8aac:1
FLY_REGION=hkg
FLY_VM_MEMORY_MB=128
GPG_KEY=A035C8C19219BA821ECEA86B64E628F8D684696D
HOME=/root
LANG=C.UTF-8
PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PYTHON_GET_PIP_SHA256=01249aa3e58ffb3e1686b7141b4e9aac4d398ef4ac3012ed9dff8dd9f685ffe0
PYTHON_GET_PIP_URL=https://github.com/pypa/get-pip/raw/d781367b97acf0ece7e9e304bf281e99b618bf10/public/get-pip.py
PYTHON_PIP_VERSION=21.2.4
PYTHON_SETUPTOOLS_VERSION=57.5.0
PYTHON_VERSION=3.10.0
TERM=linux
WS=this
is
a
test
cgroup_enable=memory

2022-06-24 08:03:24.870499769 +0000 UTC m=+427011.532288777

Unfortunately I can’t provide traceroute output as the network I’m on blocks it :frowning: Connecting from other networks (e.g. from home) routes correctly.

  • Dan
1 Like

I hoped that this would be fixed as part of the BGP changes of the network maintenance yesterday, but we still get routed through HKG and get 400-500 ms of latency from the network where most of our users access our services from :frowning:

Hey @dansondergaard, can you provide a traceroute to debug.fly.dev please? We can definitely look into this!

@miyuru how is your traceroute looking now? There have been routing changes.

A traceroute from your upstream AS3239 looked right. It routed to Frankfurt. Getting a traceroute and mtr (a few minutes worth) from your end would be useful to debug further.

nothing much changed since the last post.

2a09:8280:70::5 - same mtr as last
interestingly https://debug.fly.dev/ still goes to ams via IPv6.

37.16.16.5

Start: 2022-07-01T20:35:36+0530
HOST: miyuru-fedora               Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- _gateway                   0.0%     1    0.9   0.9   0.9   0.9   0.0
  2.|-- ???                       100.0     1    0.0   0.0   0.0   0.0   0.0
  3.|-- ???                       100.0     1    0.0   0.0   0.0   0.0   0.0
  4.|-- ???                       100.0     1    0.0   0.0   0.0   0.0   0.0
  5.|-- 125.214.190.2              0.0%     1   34.8  34.8  34.8  34.8   0.0
  6.|-- 125.214.162.65             0.0%     1   33.9  33.9  33.9  33.9   0.0
  7.|-- ???                       100.0     1    0.0   0.0   0.0   0.0   0.0
  8.|-- ???                       100.0     1    0.0   0.0   0.0   0.0   0.0
  9.|-- 9.98.225.104.ptr.anycast.  0.0%     1  174.5 174.5 174.5 174.5   0.0
 10.|-- 37.16.16.5                 0.0%     1  174.3 174.3 174.3 174.3   0.0

Unfortunately I can’t traceroute as it’s blocked on the university network (where the problem occurs) :frowning: Is there anything else I can do to help?

traceroute functionality is blocked? That’s interesting.

Does ping work? If so, I wonder if UDP is blocked.

If that’s the case, can you try mtr --tcp debug.fly.dev?

Sorry about the delay, it’s summer vacation time here :slight_smile:

Ping works just fine:

$ ping debug.fly.dev
PING debug.fly.dev (77.83.140.164) 56(84) bytes of data.
64 bytes from 77.83.140.164 (77.83.140.164): icmp_seq=1 ttl=49 time=222 ms
64 bytes from 77.83.140.164 (77.83.140.164): icmp_seq=2 ttl=49 time=222 ms
64 bytes from 77.83.140.164 (77.83.140.164): icmp_seq=3 ttl=49 time=222 ms
64 bytes from 77.83.140.164 (77.83.140.164): icmp_seq=4 ttl=49 time=222 ms
^C

It’s probably all UDP that’s blocked.

Sure, here’s a representative sample (it’s very consistent over time):

Thanks for looking at this!

That’s very helpful, thanks.

Can you do the MTR again? The issue seems to be with NORDU, but we’ve mitigated it from our side. Our upstream network provider will be contacting NORDU to get this fixed at the right spot.

It’s much better now, thanks a lot!

Looks like this has been fixed for me, I am routed to SIN now.

Much better ping now. :smiling_face_with_three_hearts:

PING debug.fly.dev(2a09:8280:1::1:2 (2a09:8280:1::1:2)) 56 data bytes
64 bytes from 2a09:8280:1::1:2 (2a09:8280:1::1:2): icmp_seq=1 ttl=53 time=79.6 ms
64 bytes from 2a09:8280:1::1:2 (2a09:8280:1::1:2): icmp_seq=2 ttl=53 time=80.3 ms

--- debug.fly.dev ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 79.624/79.953/80.282/0.329 ms
1 Like