Routing issue - Berlin gets routed to Boston

1

Hi,

I am based in Berlin and responses from my website got suddenly very slow. I checked the headers and it looks like all requests are now for some reason routed to the BOS region in the US which results in 300ms more latency.

If i connect via VPN to FRA though, then the responses are much faster since they get correctly served by the FRA region in Germany.

The questions is why requests from Berlin are served by BOS in US, but requests from Frankfurt are served from FRA.

Is that a known routing issue?

2

thanks for the heads up!

would you mind running mtr -rwbzc 20/traceroute on both sides of that 300ms connection from Berlin <-> your app?

This might help us figure out what is specifically causing the extra latency, and where it’s happening.

3

Sure, here is the output:

Start: 2022-12-21T17:27:47+0100
HOST: iMac-von-Steffen-8.local                                           Loss%   Snt   Last   Avg  Best  Wrst StDev
  1. AS3209   2a02:8109:9c80:3a20:e228:6dff:fe62:bf4a                    50.0%    20    0.6   0.6   0.5   0.7   0.1
  2. AS3209   2a02:8109:8001:39::1                                       75.0%    20   13.1  12.2   9.9  13.2   1.3
  3. AS3209   2a02:8100:6:2::103:1121                                    45.0%    20   10.9  12.9  10.0  23.2   4.3
  4. AS3209   2a02:8100:6:2::6:38d                                       90.0%    20   10.0  10.9  10.0  11.8   1.3
  5. AS3209   2a00::5fff:1000:c02:c01:11                                 75.0%    20   10.3  12.0  10.3  15.0   2.0
  6. AS3209   2a00::1a0:0:4f                                             95.0%    20   17.1  17.1  17.1  17.1   0.0
  7. AS3209   2a00::1a0:0:4f                                             95.0%    20   14.9  14.9  14.9  14.9   0.0
  8. AS1273   ae9-100-xcr1.hac.cw.net (2001:5001:700::1)                  0.0%    20   15.3  16.7  15.0  26.5   2.4
  9. AS1273   ae40-pcr1.fis.cw.net (2001:5000:0:d1::2)                    0.0%    20   21.5  23.6  19.6  45.2   5.3
 10. AS3257   ae13.cr7-fra2.ip6.gtt.net (2001:668:0:3:ffff:0:9a0e:2259)   0.0%    20   33.6  27.8  20.6  46.7   7.9
 11. AS3257   ae18.cr0-bos1.ip6.gtt.net (2001:668:0:2::1:1a1)             0.0%    20  108.7 108.3 105.7 113.1   2.0
 12. AS3257   2001:668:0:3:ffff:2:0:11e6                                  0.0%    20  105.8 108.9 105.0 130.1   5.5
 13. AS40509  2a09:8280:1::1:67c8                                        35.0%    20  103.9 105.2 102.7 107.3   1.3

I think part of the additional latency is caused by a back and forth between the server and DB which are in different regions. The DB is in FRA and there is currently no read replica in the US.

Generally, there are currently servers running in FRA (eu-west), LHR (eu-west), IAD (us-east) and SEA (us-west) and the DB is in FRA. So it is of course expected that the servers in the US have around 50-100ms extra latency because of the database connection.

That will be fixed by extra read replicas in the future, however I would expect my request in Germany not to be answered by a server in the US, but FRA instead.

I think what currently happens is request from BER goes to BOS, then to the server in IAD, which then queries the DB in FRA, which then sends data back to IAD, which then responds to my request from BER. So it’s a lot of back and forth which results in that latency I guess.

However if I connect via VPN to Frankfurt, I get a response in around 50ms.

Even if I VPN to other countries around, like Belgium for example, I get served much faster in under 100ms (AMS in that case). Just the routing for Berlin seems off.

1 Like
4

Thanks for sending that over! Would you mind confirming the app and db names? (feel free to email this info over to support@fly.io if you’d prefer).

I /think/ I have the right app, but I want to make sure that we have all the right info we need to figure out what’s causing GTT to send traffic to Boston. I’m guessing that you see similar behavior when connecting over IPv4, too?

It definitely seems unlikely that fly proxy would avoid routing to any of the other, closer instances on your app for expected reasons, especially since things are working fine over your FRA VPN.

If you’re inclined, it’d be interesting (but not something I anticipate that we’d immediately need) to compare the two routes, come to think of it.

Looking over the forward route, it does look like loss is a little high, starting from the local ISP’s network. To be clear, I don’t imagine this explains the extra latency you’ve reported, but it is something that you might want to look into for better overall performance should you see it frequently on your traffic.

1 Like
5

Thx for your quick response! The app is called “goodiful”, the DB is hosted on neon.tech.

It looks like the issue is resolved now! My requests are now routed to AMS. I would still expect FRA to be closer, but maybe AMS is actually faster - in any case, much better than US.

2 Likes
6

awesome! glad to hear we’re on the right continent now :slight_smile:

Since my last response, we were able to tweak our configs with one of our ISPs, and it looks like this worked :tada:

Thanks again for calling this out!

1 Like