wrong coast bad routing/peering to anycast Fly on Charter/Spectrum

I live in NYC, USA, and have a fly VM set up in Newark NJ/NYC/EWR region. ever since joined Fly, I’ve noticed Charter/Spectrum has NO-SANE-PEERING with Fly’s edge HTTP proxy (XXX.fly.dev IPs), in fall 2022, my NYC Charter connection went to California, then back to EWR. 40-60ms ping times to the fly HTTP front end proxy. I didn’t save the trace from that time. From doing pings from different cities, it seems 149.248.223.93 is an anycast IP, its 1-2ms ping anywhere in USA using looking glass tools. (1-2ms ping is appropriate for same city fiber to fiber, no LTE, no cable modems).

Its now spring 2023. I still have 30ms pings instead of 10ms pings to fly on Spectrum/Charter in NYC. Now instead of, I forgot which 3rd party ISP from 2022. CacheNetworks/Fly.io only peers in Atlanta Georgia for probably ALL Spectrum customers in USA with its anycast IPs, and even stranger, COMCAST is acting as a Tier 1 to reach Fly.io. I’m not complaining about Comcast being a Tier 1 ISP. Everyone knows Comcast is big, but its not Cogent/Telia/Zayo/Lumen lol But still it adds onto evidence of very abnormal peering between Spectrum and Fly.

For comparison, in NYC the other cable modem provider Optimum, its has 10-12 ms pings to Fly HTTP proxy. And a proper, intra-city routing. I’ve included both traceroutes. Traceroutes from my Fly VM to my Charter cable modem show no routing problems https://bgp.he.net/AS30081#_peers CacheNetworks lists Charter as a peer, and I doubt any “CDN provider” would be a in a peering dispute with a major USA residential ISP routing all traffic to exactly 1 city vs going over a $$$ Tier 1 link. This problem is Fly’s anycast IP HTTP proxies, not a Fly’s DC’s VM’s connection to public internet. I DO NOT see this routing problem doing a traceroute from my VM to the cable modem.

C:\sources>tracert XXX.fly.dev

Tracing route to XXXX.fly.dev [149.248.223.93]
over a maximum of 30 hops:

  1     1 ms     2 ms     1 ms  RMV
  2    14 ms    14 ms    13 ms  cpe-72-229-160-1.nyc.res.rr.com [72.229.160.1]
  3    19 ms    16 ms    15 ms  lag-62.nyctny5001h.netops.charter.com [68.173.20
1.4]
  4    15 ms    15 ms    15 ms  lag-107.nyclnyrg01r.netops.charter.com [68.173.1
98.160]
  5    18 ms     9 ms    17 ms  lag-29.nwrknjmd67w-bcr00.netops.charter.com [107
.14.19.24]
  6   116 ms    57 ms    40 ms  lag-14.asbnva1611w-bcr00.netops.charter.com [107
.14.17.239]
  7    35 ms    36 ms    32 ms  lag-402.atlngamq46w-bcr00.netops.charter.com [66
.109.6.151]
  8    45 ms    29 ms    32 ms  lag-302.pr2.atl20.netops.charter.com [66.109.9.1
03]
  9    35 ms    31 ms    35 ms  be-201-pe11.56marietta.ga.ibone.comcast.net [50.
248.119.57]
 10    34 ms    34 ms    34 ms  be-2311-cs03.56marietta.ga.ibone.comcast.net [96
.110.32.29]
 11    34 ms    33 ms    34 ms  be-2303-pe03.56marietta.ga.ibone.comcast.net [96
.110.37.90]
 12    32 ms    29 ms    34 ms  75.149.228.162
 13    30 ms    33 ms    36 ms  149.248.223.93

Trace complete.
C:\sources>tracert 149.248.223.93

Tracing route to 149.248.223.93 over a maximum of 30 hops

  1    <1 ms    <1 ms    <1 ms  192.168.1.1
  2    10 ms    17 ms     9 ms  10.240.186.97
  3    13 ms     9 ms     9 ms  67.59.246.116
  4    15 ms    20 ms    11 ms  ool-4353e790.dyn.optonline.net [67.83.231.144]
  5    10 ms    11 ms    14 ms  64.15.8.57
  6    12 ms    11 ms    11 ms  64.15.3.110
  7    12 ms    17 ms    11 ms  be6059.ccr31.jfk10.atlas.cogentco.com [38.142.13
6.193]
  8    12 ms    11 ms    14 ms  be2073.ccr31.jfk05.atlas.cogentco.com [154.54.0.
229]
  9    13 ms    11 ms    11 ms  be2809.rcr21.b007023-2.jfk05.atlas.cogentco.com
[154.54.81.202]
 10     *        *        *     Request timed out.
 11     *        *        *     Request timed out.
 12    13 ms    13 ms    11 ms  149.248.223.93

Trace complete.

C:\sources>

Logging into my EWR region VM, and doing a traceroute to my Spectrum cable modem shows perfect routing.

root@XXX:/# tracepath -4 72.229.163.XXX
 1?: [LOCALHOST]                      pmtu 1420
 1:  172.19.68.97                                          0.153ms 
 1:  172.19.68.97                                          0.088ms 
 2:  66.225.222.253                                       64.524ms 
 3:  lag-1080.pr2.nyc20.netops.charter.com                 8.740ms asymm  4 
 4:  lag-10.nycmny837aw-bcr00.netops.charter.com           0.906ms 
 5:  lag-2.nyquny9101r.netops.charter.com                  7.372ms 
 6:  lag-1.nyctny5002h.netops.charter.com                  2.048ms asymm  8 
 7:  agg2.nyctny5002m.nyc.rr.com                           2.873ms 
 8:  cpe-72-229-163-XXX.nyc.res.rr.com                    18.382ms reached
     Resume: pmtu 1420 hops 8 back 8 
root@XXX:/# 

Optimum NYC Cable modem at https://debug.fly.dev/

Fly-Region: lga

Spectrum NYC cable modem

Fly-Region: atl

Update: I tested some other NYC networks, using some other NYC UAs, TMobile, ATT Wireless, Verizon Wireless , and a commercial VPN on (AS9009 M247 Europe SRL) in NYC, on all say “Fly-Region: lga”, its Spectrum/Charter that is the problem.

1 Like

This is excellent information. We’re looking at what’s going on here, we’ve been adding regions with limited local announcements to fill in some gaps (like ATL) and it seems like that leaked more than it should have.

Will you check again and see if you’re getting better results now?

Problem fixed. Stays inside same city, much lower pings.

C:\sources>tracert 149.248.223.93

Tracing route to 149.248.223.93 over a maximum of 30 hops

  1     2 ms     1 ms    <1 ms  DD-WRT [192.168.1.1]
  2    18 ms     8 ms    12 ms  cpe-72-229-160-1.nyc.res.rr.com [72.229.160.1]
  3    19 ms    29 ms    19 ms  lag-62.nyctny5002h.netops.charter.com [68.173.20
1.6]
  4    16 ms    15 ms    16 ms  lag-107.nyquny9101r.netops.charter.com [68.173.1
98.162]
  5    16 ms    19 ms    19 ms  lag-15.nycmny837aw-bcr00.netops.charter.com [66.
109.6.76]
  6    16 ms    25 ms    15 ms  lag-1.pr2.nyc20.netops.charter.com [66.109.9.5]

  7    14 ms    16 ms    17 ms  nyk-b1-link.ip.twelve99.net [62.115.156.214]
  8    16 ms    15 ms    14 ms  ae-17.a02.nycmny17.us.bb.gin.ntt.net [129.250.8.
229]
  9    13 ms    17 ms    19 ms  ae-15.r20.nwrknj03.us.bb.gin.ntt.net [129.250.3.
8]
 10    21 ms    34 ms    26 ms  ae-0.a01.nycmny17.us.bb.gin.ntt.net [129.250.3.1
53]
 11     *        *        *     Request timed out.
 12     *        *        *     Request timed out.
 13    15 ms    16 ms    15 ms  149.248.223.93

Trace complete.

C:\sources>
1 Like

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.