Uploading build image to Fly registry times out

Lately, my deploys have been taking longer because of retries or straight up timing out because the upload build image step seems to time out after the image is uploaded (it’s stuck on the progress bar, before it would normally say “pushed”).

I’m running on a remote builder, and my relevant image layer is only 40ish MBs, so I don’t think it’s size related. Does it have something to do with the builder’s region?

We’re looking into registry performance, should have it resolved soon. I don’t think it has anything to do with the region at this point, but will tag you if that changes.

Was lately more than 4 hours ago? We had some over night (US time) capacity problems with our registry. These should have been fine for the last several hours.

Can you paste your logs if this happens again?

Yeah, over the last couple of days, usually around evening in EU time.

The status page should also indicate any issues https://status.flyio.net/ - so that’s a faster way to check what’s going on.

Hmm, to be honest, pushing the image from a remote builder to the registry has always taken a while for me - only recently has it started taking so long that it times out and goes into a retry loop. I’m not sure how logs could help here, it’s just the Docker image push output that hangs around on 40MB/40MB for a minute or so before either reporting pushed or retrying in X seconds.

The fact that is goes to 100% completion and then hangs is interesting information, thanks. Will keep that in mind. Seems clear that there isn’t a problem in pushing the actual layer objects itself, but in one of the finishing-up steps.

1 Like