Looking for some help troubleshooting an issue with Fly.
One of my client sites is using Fly.io as a reverse proxy to offer whitelabel domains for a Heroku app. This week I began seeing 500ms slower request speeds through the reverse proxy than directly through the Heroku site.
I knew there would be some latency when using a reverse proxy, but I was thinking the latency would be much less. I’ve checked the stats on the Fly app, the Heroku app, everything and it all looks fine. The only thing that looks out of place is the traceroute. Here is an example site if you want to check out the traceroute: https://it.justtakesonecall.com/
Here are two traceoutes. The first goes directly to the Heroku app. The second goes through the Fly.io proxy. The latter takes over a minute to complete.
Hey everyone, I’ll delete this post. I did some additional digging and I realized that I also had DNSimple fronting the Fly.io proxy. I removed DNS simple and things got much faster! Sorry for the false alarm
Oh good, I’m glad it was simple and not tough to track down.
Traceroutes are pretty finicky, if you run into issues with speed the first place to check is https://debug.fly.dev. That’ll show you which region you’re hitting, and you can use it as a baseline for latency tests too.
@kurt Thanks for the quick response on this on a Saturday!
So… I didn’t actually fix the issue the issue this morning. Cutting out DNSimple as a CNAME front helped, but didn’t resolve the massive latency I have been seeing.
BUT I have more clues on what is happening and would love your thoughts.
My Heroku logs are showing these three forwarded ip addresses (excluding my own ip address) 2a09:8280:1:a5df:525:9645:7cb1:93a1 => Fly 2a00:dd80:20:2b:0:c499:8cf3:1 => NetActuate, Frankfurt Germany 162.158.89.15 => Cloudflare
Outbound data transfer shows bandwidh metrics for where the instance is located.
I looked up your app is and it’s only deployed in our Frankfurt region. That’s why all requests are routed there eventually.
Looks like your regions are set to AMS,CDG and FRA. You probably need to add more regions and more instances if you want to cover more geos. Or you can change your regions entirely if you’d rather your app only ran close to you.
To add to this: if you’re running on Heroku in the US, you should run the app in our iad region at first, then expand to more from there. If you’re in Heroku’s EU region, you should run it in lhr. The lag you’re seeing is packets traveling from you to Frankfurt and then to Heroku, vs just going direct to Heroku.
Yep. That did it. Should have checked the region earlier, I just thought where the app was spun up was all automatic, but this totally makes sense. Thank you again for the prompt response.
