IPv4 to DFW routed through SJC

1

When I create apps in DFW, my IPv4 traffic is routed through SJC. That adds 70+ ms to each request for me in Austin. IPv6 traffic is not affected.

I set up a test app running nginx at https://dfw-route.fly.dev/.

I’m using Google Fiber and can provide my IP if needed. MTR did not produce anything useful. Is this something that can be fixed?

2

I created another app for anyone to quickly test their IPv4/6 routes: https://my-route.fly.dev

I sent it to several friends on different networks. None had the closest edge location and many differed between v4 and v6. I know anycast doesn’t use physical proximity, but Chicago and San Jose can’t possibly be the fastest route to Austin.

What’s the point of running apps near your users if traffic is routed across the country?

3

Sorry for the inconvenience. Even if you may not find MTR producing anything directly useful, knowing the intermediate systems your traffic is currently getting routed through can help us narrow down what we need to adjust to fix the issue.

4

I’ve tried different flags, but I only ever get that one entry (#8). I also tried while plugging directly into the fiber modem in case my firewall was filtering ICMP. Same result.

% sudo mtr -4 --no-dns my-route.fly.dev

                                       Packets               Pings
 Host                                Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. 10.10.10.1                        0.0%    10    0.3   0.3   0.3   0.4   0.0
 2. 10.26.1.10                        0.0%    10    1.0   1.1   1.0   1.3   0.1
 3. (waiting for reply)
 4. (waiting for reply)
 5. (waiting for reply)
 6. (waiting for reply)
 7. (waiting for reply)
 8. 23.255.225.53                     0.0%    10   36.1  36.5  36.1  37.0   0.3
 9. (waiting for reply)
10. (waiting for reply)
11. (waiting for reply)
12. (waiting for reply)
13. 66.241.124.191                    0.0%     9   36.5  36.8  36.5  37.2   0.3
1 Like
5

Thanks for the report and the MTR details, we’re able to reproduce Google Fiber in Austin entering our network via SJC in our own tests. As you noted, there’s not any public Internet routing that our network provider can address through BGP adjustments on our side, we suspect this may be an issue in Google Fiber’s internal routing they’ll need to fix on their end. We’ll see what we can do but you may have better luck escalating this issue through your own support channels as an ISP customer.

As for your several friends on different networks experiencing other issues, if you have any additional details on specific networks and the specific issues observed, we can follow up on any details you or your friends can provide to improve our overall routing.

6

Over the weekend I ran an experiment on one of my sites that loaded a random resource over IPv4, IPv6, and Cloudflare. For instance, a visitor may generate requests to these three URLs:

https://ipv4.example.com/9e83995c
https://ipv6.example.com/9e83995c
https://cf.example.com/9e83995c

I correlated the requests based on the random ID and set a cookie to prevent duplicates from the same visitor. Cloudflare provided IP geolocation. The requests generated a log similar to this:

id dc_v4 dc_v6 cf_latitude cf_longitude cf_city cf_region cf_country
9e83995c dfw arn 29.4247 -98.4621 San Antonio TX US
6cc15617 ewr arn 42.2937 -89.6354 Freeport IL US
29d1105c lhr lhr 53.5123 -2.3143 Manchester ENG GB

In the end I had about 7k rows, with 4k having IPv4, IPv6, and Cloudflare data.

Based on the data, I found that:

  • The IPv4 and IPv6 routes were the same only 72% of the time.
  • For both IPv4 and IPv6, traffic was routed to the Fly datacenter closest to the user only 57% of the time.
  • The ideal condition, where both IPv4 and IPv6 traffic are routed to the closest datacenter, only occurred 43% of the time.

I know there are caveats to this experiment, but I still find it concerning that the routing is so inconsistent.

7

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.