I am getting consistently slow response times from both my apps since yesterday. Nothing changed on my end so I am puzzled about why this suddenly started happening. I checked my app and the DB are still on the same region (and I am requesting from the same region) and put server timings on a route to check if it was the application but it is reporting 18ms response time. So unsure where this extra 800ms is coming in:
2023-10-17T16:02:50.830 app[****************] ord [info] 16:02:50.829 [error] Postgrex.Protocol (#PID<0.1998.0>) failed to connect: ** (DBConnection.ConnectionError) tcp recv (idle): closed
2023-10-17T16:02:51.929 app[****************] ord [info] 16:02:51.928 [error] Postgrex.Protocol (#PID<0.1993.0>) timed out because it was handshaking for longer than 15000ms
errors returned from the app (I was able to connect to the db from outside the app just fine throughout). I restarted the DB machine and that went away but the app is still very slow.
I did a bit of digging on this and I am not sure if it is a Fly.io issue specifically.
First of, using my cellular connection I did not have the lagging responses.
I noticed that even my connection (on my ISP) to fly.io was slow, that is the loading of the actual website and assets.
Out of curiousity I did a traceroute to my app and found the packet bounced around Singapore and Australia before it made it back to me! (I am in North America along with my app).
The traceroute to fly.io did exactly the same thing. I think that would explain the long response times I’m getting on my apps. It looks like my ISP thinks the quickest way of getting to any fly subdomain is by going all around the world.
Even firing a simple ping to fly.io gives me 300ms latency whereas any other site is <10ms.
I’m not enough of a network engineer to understand why the route suddenly changed but I’m guessing my ISP messed up.
I’m so glad I’m not the only experiencing this — thought I’d been going crazy seeing these large latency spikes! All the exact same symptoms as described here (>200ms pings to my Fly hosted services, as well as fly.io itself), with really weird routes in traceroute — at one point I was seeing packets routed through China. First started seeing this behaviour about a week ago.
One of my temporary solutions has just been to allocate new IP addresses until I get one with a normal latency (i.e. <10ms), but even after doing that a couple times, they’ve all ended up back on these cross-continental network routes after a couple days.
Best solution thus far has definitely just been hopping on a VPN or cellular connection — usually gets me a good enough route (<50ms) but still not optimal.
I originally thought it would have to be something at my ISP’s level if I was the only one experiencing this, but if others are seeing it to, then… who knows. I’m no network guru.
Though, does kind of defeat the purpose of using Fly to “launch apps near my users” if the network trip is halfway around the world
Mine is currently being returned as ‘mel’ which is Melbourne Australia, although I’m in Orlando FL. I am currently debugging this with support, and we haven’t ruled out my ISP at the moment which is Spectrum.
