Fly machines VERY SLOW - All (10+) apps taking 1-3s per patch/page load.

Phoenix
1

Been using fly for a few years with no problems, and up until a week or so ago everything has been running smoothly, but I’m now seeing ALL of my fly apps, personal and professional across multiple accounts are running quite slow with page load times & patching which is usually instant, is taking around 1-3 seconds per patch (SUPER SLOW). As well as fly ssh, fly proxy etc also being very slow.

They are all in SYD region and some machines are on 32gb ram with x8 CPU with little load, still very slow.

Is anyone else experiencing this? This is quite frustrating and would like to fix ASAP as it is hindering user experience.

Thanks a bunch, please reach out if you have any insight :slight_smile: .

2

We’re not currently aware of any general performance issues in syd, anecdotally my apps there are behaving normally.

Assuming these are HTTP requests, can you catch one with the flyio-debug: doit header set? e.g.

curl -H 'flyio-debug: doit' -I https://<your-app>.fly.dev

And post the fly-request-id here?

3

Yeah sure, here it is:

fly-request-id: 01K8MA5MG6VW2S4SHSCAD4W7VF-syd

4

Also, it may be worth noting, it’s the phx-loading bar at the top that hangs the page loads, due to i assume opening the socket connection

5

Interesting, thanks for that. From when we received that request all looks good, and skimming over all metrics on our side I’m not spotting anything of concern.
One thing to look at would be, if all of these apps are Phoenix and having issues in the same way, if any versions were bumped across them recently. Also, if this is happening mostly for you or your ISP, checking a traceroute or mtr from your machine to your app’s ip address is worthwhile — let me know if that seems abnormal (i.e. it travels out of the country or has particularly high latency).

Beyond that, if these are production apps it might be worthwhile to pick up a support plan, at which point our support folk can dive deeper into tailored debugging & architecture.

6

Turns out the IPv6 Routing decides to have a global round trip…
v4 total=0.090066s
v6 total=1.257628s

For now, I’ve deleted the v6 Record in my DNS settings for my domain, it won’t use fly’s IPv6 and is now back to being snappy.

Here’s the mtr for it:


HOST: 9zggp57qmxm0zt4fmz7y.ip6.superloop.au             Loss%   Snt   Last   Avg  Best  Wrst StDev
  1. AS???    ???                                       100.0   100    0.0   0.0   0.0   0.0   0.0
  2. AS???    ???                                       100.0   100    0.0   0.0   0.0   0.0   0.0
  3. AS38195  14vqf8rfzh82v1pgg73j.ip6.superloop.au     92.0%   100    8.2  12.7   6.7  46.2  13.6
  4. AS38195  2401:d000:10:100::6c2                     34.0%   100    4.7   5.4   4.3  20.0   1.9
  5. AS???    ???                                       100.0   100    0.0   0.0   0.0   0.0   0.0
  6. AS7578   e29.mel-eqxme1-cr3.globalsecurelayer.com   0.0%   100   16.9  17.6  16.5  21.8   0.8
  7. AS7578   po3.mel-eqxme1-bb1.globalsecurelayer.com   0.0%   100   17.2  18.3  16.8  23.2   1.4
  8. AS7578   e53.adl-eqxae1-bb4.globalsecurelayer.com   0.0%   100   16.7  15.8  14.8  19.5   0.9
  9. AS7578   po7.per-eqxpe2-bb5.globalsecurelayer.com   1.0%   100   43.1  43.8  42.9  46.8   0.8
 10. AS7578   po6.per-ndcp2-bb7.globalsecurelayer.com    0.0%   100   44.0  43.7  42.6  47.0   0.7
 11. AS7578   po8.sg-eqxsg3-bb2.globalsecurelayer.com    1.0%   100  123.8 123.4 122.4 127.0   0.8
 12. AS7578   po9.mrs-ixmrs2-cr4.globalsecurelayer.com   0.0%   100  241.2 241.4 240.5 244.7   0.7
 13. AS7578   po8.par-thpa2-cr4.globalsecurelayer.com    0.0%   100  240.6 241.5 240.5 247.3   1.1
 14. AS7578   e13.par-eqxpa3-cr1.globalsecurelayer.com   0.0%   100  241.0 241.6 240.6 245.2   0.9
 15. AS7578   2401:3cc0::4:533                           0.0%   100  234.4 239.9 233.8 292.1  11.5
 16. AS40509  2a09:8280:1::9f:3a74:0                     0.0%   100  233.0 233.0 232.0 242.0   1.3
7

Hi! Thanks muchly for the mtr, we were able to track down this bad route and resolve it. You should be good to add the IPv6 back now :slightly_smiling_face:

8

Awesome! Thanks heaps for your help :slight_smile: