I live in NYC, USA, and have a fly VM set up in Newark NJ/NYC/EWR region. ever since joined Fly, I’ve noticed Charter/Spectrum has NO-SANE-PEERING with Fly’s edge HTTP proxy (XXX.fly.dev IPs), in fall 2022, my NYC Charter connection went to California, then back to EWR. 40-60ms ping times to the fly HTTP front end proxy. I didn’t save the trace from that time. From doing pings from different cities, it seems 149.248.223.93 is an anycast IP, its 1-2ms ping anywhere in USA using looking glass tools. (1-2ms ping is appropriate for same city fiber to fiber, no LTE, no cable modems).
Its now spring 2023. I still have 30ms pings instead of 10ms pings to fly on Spectrum/Charter in NYC. Now instead of, I forgot which 3rd party ISP from 2022. CacheNetworks/Fly.io only peers in Atlanta Georgia for probably ALL Spectrum customers in USA with its anycast IPs, and even stranger, COMCAST is acting as a Tier 1 to reach Fly.io. I’m not complaining about Comcast being a Tier 1 ISP. Everyone knows Comcast is big, but its not Cogent/Telia/Zayo/Lumen lol But still it adds onto evidence of very abnormal peering between Spectrum and Fly.
For comparison, in NYC the other cable modem provider Optimum, its has 10-12 ms pings to Fly HTTP proxy. And a proper, intra-city routing. I’ve included both traceroutes. Traceroutes from my Fly VM to my Charter cable modem show no routing problems https://bgp.he.net/AS30081#_peers CacheNetworks lists Charter as a peer, and I doubt any “CDN provider” would be a in a peering dispute with a major USA residential ISP routing all traffic to exactly 1 city vs going over a $$$ Tier 1 link. This problem is Fly’s anycast IP HTTP proxies, not a Fly’s DC’s VM’s connection to public internet. I DO NOT see this routing problem doing a traceroute from my VM to the cable modem.
C:\sources>tracert XXX.fly.dev
Tracing route to XXXX.fly.dev [149.248.223.93]
over a maximum of 30 hops:
1 1 ms 2 ms 1 ms RMV
2 14 ms 14 ms 13 ms cpe-72-229-160-1.nyc.res.rr.com [72.229.160.1]
3 19 ms 16 ms 15 ms lag-62.nyctny5001h.netops.charter.com [68.173.20
1.4]
4 15 ms 15 ms 15 ms lag-107.nyclnyrg01r.netops.charter.com [68.173.1
98.160]
5 18 ms 9 ms 17 ms lag-29.nwrknjmd67w-bcr00.netops.charter.com [107
.14.19.24]
6 116 ms 57 ms 40 ms lag-14.asbnva1611w-bcr00.netops.charter.com [107
.14.17.239]
7 35 ms 36 ms 32 ms lag-402.atlngamq46w-bcr00.netops.charter.com [66
.109.6.151]
8 45 ms 29 ms 32 ms lag-302.pr2.atl20.netops.charter.com [66.109.9.1
03]
9 35 ms 31 ms 35 ms be-201-pe11.56marietta.ga.ibone.comcast.net [50.
248.119.57]
10 34 ms 34 ms 34 ms be-2311-cs03.56marietta.ga.ibone.comcast.net [96
.110.32.29]
11 34 ms 33 ms 34 ms be-2303-pe03.56marietta.ga.ibone.comcast.net [96
.110.37.90]
12 32 ms 29 ms 34 ms 75.149.228.162
13 30 ms 33 ms 36 ms 149.248.223.93
Trace complete.
C:\sources>tracert 149.248.223.93
Tracing route to 149.248.223.93 over a maximum of 30 hops
1 <1 ms <1 ms <1 ms 192.168.1.1
2 10 ms 17 ms 9 ms 10.240.186.97
3 13 ms 9 ms 9 ms 67.59.246.116
4 15 ms 20 ms 11 ms ool-4353e790.dyn.optonline.net [67.83.231.144]
5 10 ms 11 ms 14 ms 64.15.8.57
6 12 ms 11 ms 11 ms 64.15.3.110
7 12 ms 17 ms 11 ms be6059.ccr31.jfk10.atlas.cogentco.com [38.142.13
6.193]
8 12 ms 11 ms 14 ms be2073.ccr31.jfk05.atlas.cogentco.com [154.54.0.
229]
9 13 ms 11 ms 11 ms be2809.rcr21.b007023-2.jfk05.atlas.cogentco.com
[154.54.81.202]
10 * * * Request timed out.
11 * * * Request timed out.
12 13 ms 13 ms 11 ms 149.248.223.93
Trace complete.
C:\sources>
Logging into my EWR region VM, and doing a traceroute to my Spectrum cable modem shows perfect routing.
root@XXX:/# tracepath -4 72.229.163.XXX
1?: [LOCALHOST] pmtu 1420
1: 172.19.68.97 0.153ms
1: 172.19.68.97 0.088ms
2: 66.225.222.253 64.524ms
3: lag-1080.pr2.nyc20.netops.charter.com 8.740ms asymm 4
4: lag-10.nycmny837aw-bcr00.netops.charter.com 0.906ms
5: lag-2.nyquny9101r.netops.charter.com 7.372ms
6: lag-1.nyctny5002h.netops.charter.com 2.048ms asymm 8
7: agg2.nyctny5002m.nyc.rr.com 2.873ms
8: cpe-72-229-163-XXX.nyc.res.rr.com 18.382ms reached
Resume: pmtu 1420 hops 8 back 8
root@XXX:/#
Optimum NYC Cable modem at https://debug.fly.dev/
Fly-Region: lga
Spectrum NYC cable modem
Fly-Region: atl
Update: I tested some other NYC networks, using some other NYC UAs, TMobile, ATT Wireless, Verizon Wireless , and a commercial VPN on (AS9009 M247 Europe SRL) in NYC, on all say “Fly-Region: lga”, its Spectrum/Charter that is the problem.