How is edge routing determined?

Questions / Help
1

Hi there,

Apologies if this has been asked before, I couldn’t find a clear answer. I have apps running in SJC and LAX but the latency to either is ~100ms from San Diego. It seems like Fly is typically routing me to IAD. Why might this be?

2

Routing happens via the complicated world of BGP.

Sometimes, for some ISPs, depending on peering between us and then, users may be routed to a sub-optimal location.

Can you provide a traceroute to your app’s IP?

We can usually improve the situation greatly with a few adjustments.

1 Like
3

I appreciate the quick reply!

Here’s the traceroute to the app in SJC.

traceroute to 66.241.124.247 (66.241.124.247), 30 hops max, 60 byte packets
 1  _gateway (192.168.88.1)  7.259 ms  7.428 ms  7.617 ms
 2  10.131.60.1 (10.131.60.1)  18.369 ms  18.498 ms  18.490 ms
 3  100.120.109.144 (100.120.109.144)  15.017 ms  19.080 ms  19.339 ms
 4  100.120.109.103 (100.120.109.103)  20.123 ms  20.423 ms  20.415 ms
 5  mtc3dsrj01-ge708.103.rd.ok.cox.net (68.1.2.29)  69.155 ms  69.148 ms  69.140 ms
 6  75.102.42.5 (75.102.42.5)  96.095 ms  62.636 ms  62.821 ms

The route is the same to LAX.

4

All IPs are advertised the same way, so that makes sense!

I’ll check with the network team. I can’t guarantee a quick improvement given the period of the year we’re in right now :slight_smile:

1 Like
5

Thanks Jerome :slight_smile: Yeah, I totally understand that this is a slow time of year. It’s not critical at all but I am looking at hosting game servers on Fly so I would certainly appreciate lower latencies :sweat_smile:

6

@jerome I had a friend run the same traceroute with a different San Diego based ISP and they’re also routed to the east coast but to a different edge server.

> tracert 66.241.124.247

Tracing route to 66.241.124.247 over a maximum of 30 hops

  1    <1 ms    <1 ms    <1 ms  router.asus.com [192.168.1.1]
  2     8 ms    11 ms     8 ms  142-254-184-209.inf.spectrum.com [142.254.184.209]
  3    10 ms    11 ms    10 ms  lag-59.crlscaij01h.netops.charter.com [76.167.20.33]
  4    15 ms    15 ms    15 ms  lag-33.sndhcaax01r.netops.charter.com [72.129.3.96]
  5    13 ms    13 ms    13 ms  lag-22.lsancarc01r.netops.charter.com [72.129.1.0]
  6    13 ms    16 ms    13 ms  lag-26.lsancarc0yw-bcr00.netops.charter.com [66.109.3.230]
  7    69 ms    63 ms    71 ms  lag-21.tustca4200w-bcr00.netops.charter.com [107.14.19.36]
  8    63 ms    62 ms    61 ms  lag-16.dllstx976iw-bcr00.netops.charter.com [66.109.6.1]
  9    64 ms    63 ms    61 ms  lag-412.hstqtx0209w-bcr00.netops.charter.com [66.109.6.91]
 10    68 ms    64 ms    61 ms  lag-405.atlngamq46w-bcr00.netops.charter.com [66.109.9.28]
 11    70 ms    74 ms    61 ms  lag-303.pr2.atl20.netops.charter.com [66.109.9.105]
 12    62 ms    63 ms    67 ms  be-201-pe11.56marietta.ga.ibone.comcast.net [50.248.119.57]
 13    62 ms    62 ms    64 ms  be-2411-cs04.56marietta.ga.ibone.comcast.net [96.110.32.33]
 14    62 ms    64 ms    61 ms  be-2403-pe03.56marietta.ga.ibone.comcast.net [96.110.37.94]
 15    64 ms    86 ms    63 ms  75.149.228.162
 16    63 ms    63 ms    67 ms  66.241.124.247

Trace complete.

Their ISP is Spectrum and mine is Cox Communications.

7

Some insights on this from the BGP world:

Anycast mostly works but is not perfect because outside networks would have multiple ways to reach an anycast network like fly.io. Which one they pick depends on their relation with fly.io.

Fly.io anycast routes can be:

  1. Coming from customer side of those networks i.e fly.io is direct or a customer of their customer somewhere. In that case they would prefer that route (highest BGP localpref on customer routes)
  2. Coming from their peering nearby or far away - this would have 2nd highest localpref
  3. Coming from transit of those networks i.e source network has to pay.

Basically you networks want to make money where they can (by egresssing from customer’s port, do it for free when possible via peering and as last resort pay for it).

So in your specific case src network (is that Cox or someone else) is either peering with fly.io in IAD. Post is 3 years old now and hence hard to find what relation was back then. Anycast can perfactly if fly.io picks up same group of upstream ISPs everywhere (which have similar relation wih rest of the internet) but that just may be impractical. They cannot find typical good transit players in all locations they operate in and hence the compromise.

1 Like