Pushing to registry very slow

Uploading to the registry is once again very slow.

I have written about this before but never found a reason or solution, uploading speed simply improved suddenly.

The only thing I noticed this time is that community.fly.io is loading very slow as well.

I am in a completely other continent this time, even closer to the EWR deployment location

Hi @jascha,

I’m not aware of any issues.

What region are you in? You can visit this: https://debug.fly.dev/ and find the “FLY_REGION”. It reports where you are connecting to.

Is the EWR region where the application is being deployed to?

It displays FLY_REGION=dfw

The app is deployed to ewr, iad and yyz.

I am currently on a 50Mbit connection which should upload at 1.8Mbps to Dallas:

Yet uploading a 220MB layer to registry.fly.io takes 11 minutes (0.3Mbps) instead of ~2 minutes it should actually take.

Now imagine pushing a 2GB image which takes 2 hours.

@Mark could you kindly advise what’s going on? This is crazy annoying and a major impediment working with fly.io.

Something that should take 10 minutes takes 2 hours and is incredible frustrating.

Just reading back now and I have a few thoughts:

1.8 Mbps (megabits per second) is 0.2 MBps (megabytes per second). 1 Bytes == 8 bits.

Uploading a 220MBytes file should take ~18 minutes at that speed. 11 minutes would mean you were uploading at speeds closer to 0.3MBps.

These speeds are very low. Can you provide a traceroute to debug.fly.dev? A mtr debug.fly.dev for a few minutes might be better to see if there’s packet loss.

 1 (  3.227 ms  5.188 ms  6.863 ms
 2 (  29.547 ms  34.602 ms  21.201 ms
 3  * * *
 4  * * *
 5 (  35.538 ms  17.108 ms  21.613 ms
 6  customer-189-216-3-40.cablevision.net.mx (  22.121 ms  21.024 ms  24.434 ms
 7  150.189-204-152.bestelclientes.com.mx (  29.582 ms  43.416 ms  47.849 ms
 8  45.189-204-152.bestelclientes.com.mx (  34.349 ms  31.341 ms  42.833 ms
 9 (  46.246 ms  43.126 ms  68.964 ms
10  * * *
11  ustex-elp-sot-dc1.transtelco.net (  44.866 ms  85.483 ms  73.886 ms
12  * * *
13  * * *
14 (  79.426 ms  58.151 ms  44.252 ms


 Host                                                                                                                                                                      Loss%   Snt   Last   Avg  Best  Wrst StDev
 1.                                                                                                                                                            98.9%    89   13.0  13.0  13.0  13.0   0.0
 2.                                                                                                                                                            4.5%    88   99.5  96.2  15.2 322.0  42.7
 3. (waiting for reply)
 4. (waiting for reply)
 5.                                                                                                                                                              2.3%    88  106.8  98.1  56.5 274.4  25.5
 6. customer-189-216-3-40.cablevision.net.mx                                                                                                                               73.6%    88  107.6 100.2  65.3 136.9  17.6
 7. 150.189-204-152.bestelclientes.com.mx                                                                                                                                  72.4%    88  116.9 109.7  75.7 146.8  16.0
 8. 45.189-204-152.bestelclientes.com.mx                                                                                                                                   74.7%    88   89.1 110.0  74.6 145.9  20.3
 9.                                                                                                                                                          3.4%    88   94.7 124.7  48.4 230.7  23.9
10. (waiting for reply)
11. ustex-elp-sot-dc1.transtelco.net                                                                                                                                       74.7%    88  126.5 121.1  95.1 150.7  12.8
12. (waiting for reply)
13. (waiting for reply)
14.                                                                                                                                           1.1%    88  123.4 122.8  49.3 167.9  17.4
15.                                                                                                                                                           5.7%    88  135.7 119.6  65.9 159.0  16.5

Thank you. Packet loss at the last hop in a MTR is generally a bad sign.

I’m consulting our network team to see if there’s anything we can do about this routing situation.

I think things might be smoother using a remote builder (via --remote-only during fly deploy).

I noticed you had wireguard peers in a pretty far-away region (cdg: Paris). Can you try fly wireguard reset? Either way, if you could get a new peer in a closer region, that will probably help a lot with upload speeds when using remote builders. You can find out your current peers’ regions w/ fly wireguard list or by reading ~/.fly/config.yml.

1 Like

Ok, interpreting that mtr output: There’s packet loss all the way, on every hop. This would indicate a problem with your ISP (either them as a whole, a misconfiguration or physical link issues? not sure.)
High packet loss usually means a hop is dropping ping replies. That isn’t exactly “packet loss”, more like a measurement artifact.

Low packet loss usually means it’s real packet loss.

High packet loss at your home router ( is a bit odd, as if it was almost blocking all pings, but not all.

I am also having issues pushing to the registry:

It’s been stuck at “Retrying in …” for a while now

I am on fiberglass:


It goes very fast from 0MB to 55.08MB and then just hangs:

eventually going into “Retrying…”

Did this happen ~35 minutes ago or is it still ongoing? There was a deploy of our registry and it looks like it wasn’t zero-downtime at all.

Seems to be a different kind of issue.

It was about 20-25 mins ago. It was happening when I posted the messages with the screenshots :slight_smile:

Hm sounds like flyctl isn’t recovering when a connection breaks for whatever reason. We’ll need to look into that, but for now we’re not planning on deploying anything more the registry. Hopefully you’re unblocked now.

1 Like

Ok, I can’t reason about the internet connection or router here as I am traveling. I just remember that I had this issue before at home and it just resolved itself at some point.

I worked around it for now by deploying a VPS server in San Francisco and deploying from there. Everything else was unbearable. I will report back home if I still see issues.

1 Like

A remote builder should help here. Since your issues a few months ago we’ve added support for WireGuard over web sockets instead of UDP which helps on some networks. If you wanted to try that, run flyctl wg websockets enable then deploy again. Hopefully it works so you don’t need a VPS.

1 Like