Some insights on this from the BGP world:
Anycast mostly works but is not perfect because outside networks would have multiple ways to reach an anycast network like fly.io. Which one they pick depends on their relation with fly.io.
Fly.io anycast routes can be:
- Coming from customer side of those networks i.e fly.io is direct or a customer of their customer somewhere. In that case they would prefer that route (highest BGP localpref on customer routes)
- Coming from their peering nearby or far away - this would have 2nd highest localpref
- Coming from transit of those networks i.e source network has to pay.
Basically you networks want to make money where they can (by egresssing from customer’s port, do it for free when possible via peering and as last resort pay for it).
So in your specific case src network (is that Cox or someone else) is either peering with fly.io in IAD. Post is 3 years old now and hence hard to find what relation was back then. Anycast can perfactly if fly.io picks up same group of upstream ISPs everywhere (which have similar relation wih rest of the internet) but that just may be impractical. They cannot find typical good transit players in all locations they operate in and hence the compromise.