Docker push failing on 2.8GB image

I get to here:

And it seems to just stall. It then retries, stalls again, retries… I believe it ends with an EOF error but waiting of the third retry again to verify that.

Yea, final result:

image

This was working earlier today, so I’m not sure if maybe it’s not some intermittent issue on the registry end.

Are you using a remote builder or a local docker daemon?

Local docker, I am just pushing the image before doing deploy as it can save a lot fo time (we can do it while running tests). Was working before, not sure why it stopped, all other images work except this one. Running new build now to see.

Confirmed its also failing when doing a fly deploy and not just pushing.

I’m frequently seeing image pushes stalling on smaller images. This has been happening for a few days for me.

They get stuck part way through and won’t budge. Stopping the deployment and starting again they get stuck at the same point.

The only way to move forward is to destroy the app completely and start over, which obviously doesn’t work for apps that have users.

At the moment I’m using the destroy and create approach for experimental deployments, but this seems like a pretty core critical problem.

If the build is happening locally and there’s an error pushing to the Fly registry it would help to get a report. If the push is failing after being built on a builder getting the builder ID should help narrow it down as well.

When the image is being pulled for deployment, reporting an error at that point is helpful as well — we can see which host is having trouble with a pull.

I don’t think destroying apps will help actually — the registry errors tend to be transient when they happen, and aren’t likely to be consistent for smaller images, unless there’s something really odd about that particular image. I’d suggest reporting and retrying instead of recreating the app.

How do I report what’s happening? There’s no error, it stalls during the push indefinitely. Ctrl-Cing and trying again it stalls at the exact same point. Destroying the app andrepushing the same image works a couple times before it is broken again. Logs are the same as others have shared above, although I’m not seeing an EOF error.

I’m seeing it happening again now. Pushing a local docker image, and it’s stalled on the following:

The push refers to repository [registry.fly.io/flydev]
f967def918f6: Pushed 
5f70bf18a086: Pushed 
92d53e225147: Pushed 
be4a40ca44b4: Pushed 
61af446c1ee6: Pushed 
981eaa65af87: Pushing  279.9MB/412.3MB
8d26b7ee8484: Pushed 
98ba3fb41bfb: Pushed 
8f8fad2044b5: Pushed 
b20f606b7e89: Pushed 
c63265d31978: Pushed 
2942c7f5700a: Pushed 
19fba11839f1: Pushed 
146dcb3660bd: Pushed 
6fb4df796221: Pushed 
c728341f8d00: Pushing  29.59MB/142.9MB
12cf420a7219: Pushing  31.52MB

If I CTRL-C and rerun the deploy, it gets stuck pushing the same layer:

The push refers to repository [registry.fly.io/flydev]
f967def918f6: Layer already exists 
5f70bf18a086: Layer already exists 
92d53e225147: Layer already exists 
be4a40ca44b4: Layer already exists 
61af446c1ee6: Layer already exists 
981eaa65af87: Pushing  279.9MB/412.3MB

Not able to deploy normally at the moment before of this.

I’ve managed to continue deploying by using --remote-only and using the remote builder. :grinning_face_with_smiling_eyes:

1 Like

Thanks for the update. The failures do seem related to uploading larger chunks via external networks, so running the builder inside Fly is a good workaround for now. We’re monitoring the registry and experimenting with improvements.

I am seeing the same issue, pushing a new layer of 200mb takes forever and stalls completely at times.

The push refers to repository [registry.fly.io/some-repo]
a972b20365f2: Pushed
9ef0fbcba94a: Pushing [===========>                                       ]  50.85MB/219.1MB
e366b8f7cbe8: Pushed
6d52b3201bcc: Layer already exists
b00ff79f0fab: Layer already exists
798d3440375c: Layer already exists
37d12038f860: Layer already exists
0d61d93f504d: Layer already exists
b342caff342d: Layer already exists

Which region are you in? Also, can you try with fly deploy --remote-only?

IAD, Ashburn, Virginia

But pushing from Europe.

–remote-only works a lot faster

4 posts were split to a new topic: Image not found error