Higher latency on Fly apps compared to AWS

So i am based in Kathmandu, Nepal. The closest location with good datacenters is in Mumbai. From my location the latency to Mumbai comes at about 35-40ms. Next best location is in Singapore where it comes at about 65-80ms depending on the time of the day (Blame my ISP for this)

I have known this for years. Whenever deploying any application for Nepal market I go for either Mumbai or Singapore.

I have been playing with Fly apps lately in Sin region and couldn’t figure out why the latency is always +20-30 ms. This app (https://liveview-counter.fly.dev/) results in +50-60 ms. Pinging my fly ip or endpoint resolves in Mumbai (40-50ms). From AWS EC2 in mumbai 1-3 ms.

I am aware fly has a mumbai region. Could it be that the Mumbai region receives the request first due to anycast and fly proxy forwards the request to my app in Singapore. This would add unnecessary latency and it would be better off just directly connecting to my app in sin. I did the latency test using k6 and the results i mentioned were median latency.

Are you able to look at your request headers? If so, look for Fly-Region and Fly-Request-Id. Fly-Region should be where the request originally entered our network. The last three bytes of the Fly-Request-Id header also contains this information and if different, is the one you want.

There is no fly-region header present. Here’s the fly-request-id : 01JNNC37PRRHZN7AHXD4Z5P3VP-bom

It is getting routed through mumbai. Why though ? It is not the most optimal path.

Yes, this is how Fly.io networking is intended to work—in general. There’s a distinction between “edge” nodes (Mumbai, in this case) and “worker” (a.k.a. “instance”) ones.

https://community.fly.io/t/what-difference-between-edge-and-instace/8797/2

You control which worker nodes you use, but, by default, all the edge nodes everywhere are turned on for you.

The end user’s own local ISP, plus networks in between, will decide which Fly.io edge is the best choice from their perspective. (This isn’t always minimum latency; it’s also at least a little about minimizing their own costs.) The Fly.io edge then forwards it along to the closest worker, which may well be in a different region.

It looks like there is some ability to tell the platform that you only ever want a single edge region to be used, but I don’t know whether that’s available for Singapore:

https://community.fly.io/t/the-price-of-bandwidth-is-still-confusing/23399/4

Broadly speaking, the default setup is typically good about not crossing entire oceans unnecessarily, but it’s not always going to be optimize down to 10–20ms granularity.

Hope this helps a little!


Aside: The flyio-debug trick and all-pairs round-trip times tables are handy ways to see this more explicitly…

Aside2: Also note that Anycast is not a broadcast of any kind. Only one of the 30+ potential destinations is chosen. (I.e., it’s not trying both Mumbai and Singapore.)

1 Like

It looks like there is some ability to tell the platform that you only ever want a single edge region to be used, but I don’t know whether that’s available for Singapore

Request routes correctly when using dedicated ipv4/6 address but not with shared in singapore region.
Did you mean fly ips allocate-v4 --shared --region sin should get me a shared ip with region of sin instead of global ?

Also when using dedicated ip the app is supposed to do the tls termination as there is no fly proxy involved correct ?

I’m pretty sure that if you pass --shared to fly ips allocate-v4, --region will be ignored.

If you request a dedicated ip, fly proxy is still involved and can take care of tls termination.

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.