Uploading to the registry is once again very slow.
I have written about this before but never found a reason or solution, uploading speed simply improved suddenly.
The only thing I noticed this time is that community.fly.io is loading very slow as well.
I am in a completely other continent this time, even closer to the EWR deployment location
Hi @jascha,
I’m not aware of any issues.
What region are you in? You can visit this: https://debug.fly.dev/ and find the “FLY_REGION”. It reports where you are connecting to.
Is the EWR region where the application is being deployed to?
It displays FLY_REGION=dfw
The app is deployed to ewr, iad and yyz.
I am currently on a 50Mbit connection which should upload at 1.8Mbps to Dallas:
Yet uploading a 220MB layer to registry.fly.io takes 11 minutes (0.3Mbps) instead of ~2 minutes it should actually take.
Now imagine pushing a 2GB image which takes 2 hours.
@Mark could you kindly advise what’s going on? This is crazy annoying and a major impediment working with fly.io.
Something that should take 10 minutes takes 2 hours and is incredible frustrating.
Just reading back now and I have a few thoughts:
1.8 Mbps (megabits per second) is 0.2 MBps (megabytes per second). 1 Bytes == 8 bits.
Uploading a 220MBytes file should take ~18 minutes at that speed. 11 minutes would mean you were uploading at speeds closer to 0.3MBps.
These speeds are very low. Can you provide a traceroute to debug.fly.dev? A mtr debug.fly.dev for a few minutes might be better to see if there’s packet loss.
1 192.168.0.1 (192.168.0.1) 3.227 ms 5.188 ms 6.863 ms
2 10.155.128.1 (10.155.128.1) 29.547 ms 34.602 ms 21.201 ms
3 * * *
4 * * *
5 10.3.13.33 (10.3.13.33) 35.538 ms 17.108 ms 21.613 ms
6 customer-189-216-3-40.cablevision.net.mx (189.216.3.40) 22.121 ms 21.024 ms 24.434 ms
7 150.189-204-152.bestelclientes.com.mx (189.204.152.150) 29.582 ms 43.416 ms 47.849 ms
8 45.189-204-152.bestelclientes.com.mx (189.204.152.45) 34.349 ms 31.341 ms 42.833 ms
9 216.171.70.209 (216.171.70.209) 46.246 ms 43.126 ms 68.964 ms
10 * * *
11 ustex-elp-sot-dc1.transtelco.net (201.174.253.230) 44.866 ms 85.483 ms 73.886 ms
12 * * *
13 * * *
14 60.15.225.104.ptr.anycast.net (104.225.15.60) 79.426 ms 58.151 ms 44.252 ms
mtr:
Host Loss% Snt Last Avg Best Wrst StDev
1. 192.168.0.1 98.9% 89 13.0 13.0 13.0 13.0 0.0
2. 10.155.128.1 4.5% 88 99.5 96.2 15.2 322.0 42.7
3. (waiting for reply)
4. (waiting for reply)
5. 10.3.13.33 2.3% 88 106.8 98.1 56.5 274.4 25.5
6. customer-189-216-3-40.cablevision.net.mx 73.6% 88 107.6 100.2 65.3 136.9 17.6
7. 150.189-204-152.bestelclientes.com.mx 72.4% 88 116.9 109.7 75.7 146.8 16.0
8. 45.189-204-152.bestelclientes.com.mx 74.7% 88 89.1 110.0 74.6 145.9 20.3
9. 216.171.70.209 3.4% 88 94.7 124.7 48.4 230.7 23.9
10. (waiting for reply)
11. ustex-elp-sot-dc1.transtelco.net 74.7% 88 126.5 121.1 95.1 150.7 12.8
12. (waiting for reply)
13. (waiting for reply)
14. 60.15.225.104.ptr.anycast.net 1.1% 88 123.4 122.8 49.3 167.9 17.4
15. 77.83.140.164 5.7% 88 135.7 119.6 65.9 159.0 16.5
Thank you. Packet loss at the last hop in a MTR is generally a bad sign.
I’m consulting our network team to see if there’s anything we can do about this routing situation.
I think things might be smoother using a remote builder (via --remote-only during fly deploy).
I noticed you had wireguard peers in a pretty far-away region (cdg: Paris). Can you try fly wireguard reset? Either way, if you could get a new peer in a closer region, that will probably help a lot with upload speeds when using remote builders. You can find out your current peers’ regions w/ fly wireguard list or by reading ~/.fly/config.yml.
Ok, interpreting that mtr output: There’s packet loss all the way, on every hop. This would indicate a problem with your ISP (either them as a whole, a misconfiguration or physical link issues? not sure.)
High packet loss usually means a hop is dropping ping replies. That isn’t exactly “packet loss”, more like a measurement artifact.
Low packet loss usually means it’s real packet loss.
High packet loss at your home router (192.168.0.1) is a bit odd, as if it was almost blocking all pings, but not all.
I am also having issues pushing to the registry:
It’s been stuck at “Retrying in …” for a while now
I am on fiberglass:
Did this happen ~35 minutes ago or is it still ongoing? There was a deploy of our registry and it looks like it wasn’t zero-downtime at all.
Seems to be a different kind of issue.
It was about 20-25 mins ago. It was happening when I posted the messages with the screenshots 
Hm sounds like flyctl isn’t recovering when a connection breaks for whatever reason. We’ll need to look into that, but for now we’re not planning on deploying anything more the registry. Hopefully you’re unblocked now.
Ok, I can’t reason about the internet connection or router here as I am traveling. I just remember that I had this issue before at home and it just resolved itself at some point.
I worked around it for now by deploying a VPS server in San Francisco and deploying from there. Everything else was unbearable. I will report back home if I still see issues.
A remote builder should help here. Since your issues a few months ago we’ve added support for WireGuard over web sockets instead of UDP which helps on some networks. If you wanted to try that, run flyctl wg websockets enable then deploy again. Hopefully it works so you don’t need a VPS.
