I get to here:
And it seems to just stall. It then retries, stalls again, retries… I believe it ends with an EOF error but waiting of the third retry again to verify that.
Yea, final result:
1 Like
This was working earlier today, so I’m not sure if maybe it’s not some intermittent issue on the registry end.
Are you using a remote builder or a local docker daemon?
Local docker, I am just pushing the image before doing deploy as it can save a lot fo time (we can do it while running tests). Was working before, not sure why it stopped, all other images work except this one. Running new build now to see.
Confirmed its also failing when doing a fly deploy and not just pushing.
I’m frequently seeing image pushes stalling on smaller images. This has been happening for a few days for me.
They get stuck part way through and won’t budge. Stopping the deployment and starting again they get stuck at the same point.
The only way to move forward is to destroy the app completely and start over, which obviously doesn’t work for apps that have users.
At the moment I’m using the destroy and create approach for experimental deployments, but this seems like a pretty core critical problem.
1 Like
If the build is happening locally and there’s an error pushing to the Fly registry it would help to get a report. If the push is failing after being built on a builder getting the builder ID should help narrow it down as well.
When the image is being pulled for deployment, reporting an error at that point is helpful as well — we can see which host is having trouble with a pull.
I don’t think destroying apps will help actually — the registry errors tend to be transient when they happen, and aren’t likely to be consistent for smaller images, unless there’s something really odd about that particular image. I’d suggest reporting and retrying instead of recreating the app.
How do I report what’s happening? There’s no error, it stalls during the push indefinitely. Ctrl-Cing and trying again it stalls at the exact same point. Destroying the app andrepushing the same image works a couple times before it is broken again. Logs are the same as others have shared above, although I’m not seeing an EOF error.
1 Like
I’m seeing it happening again now. Pushing a local docker image, and it’s stalled on the following:
The push refers to repository [registry.fly.io/flydev]
f967def918f6: Pushed
5f70bf18a086: Pushed
92d53e225147: Pushed
be4a40ca44b4: Pushed
61af446c1ee6: Pushed
981eaa65af87: Pushing 279.9MB/412.3MB
8d26b7ee8484: Pushed
98ba3fb41bfb: Pushed
8f8fad2044b5: Pushed
b20f606b7e89: Pushed
c63265d31978: Pushed
2942c7f5700a: Pushed
19fba11839f1: Pushed
146dcb3660bd: Pushed
6fb4df796221: Pushed
c728341f8d00: Pushing 29.59MB/142.9MB
12cf420a7219: Pushing 31.52MB
If I CTRL-C and rerun the deploy, it gets stuck pushing the same layer:
The push refers to repository [registry.fly.io/flydev]
f967def918f6: Layer already exists
5f70bf18a086: Layer already exists
92d53e225147: Layer already exists
be4a40ca44b4: Layer already exists
61af446c1ee6: Layer already exists
981eaa65af87: Pushing 279.9MB/412.3MB
Not able to deploy normally at the moment before of this.
I’ve managed to continue deploying by using --remote-only and using the remote builder. 
2 Likes
Thanks for the update. The failures do seem related to uploading larger chunks via external networks, so running the builder inside Fly is a good workaround for now. We’re monitoring the registry and experimenting with improvements.
I am seeing the same issue, pushing a new layer of 200mb takes forever and stalls completely at times.
The push refers to repository [registry.fly.io/some-repo]
a972b20365f2: Pushed
9ef0fbcba94a: Pushing [===========> ] 50.85MB/219.1MB
e366b8f7cbe8: Pushed
6d52b3201bcc: Layer already exists
b00ff79f0fab: Layer already exists
798d3440375c: Layer already exists
37d12038f860: Layer already exists
0d61d93f504d: Layer already exists
b342caff342d: Layer already exists
Which region are you in? Also, can you try with fly deploy --remote-only?
IAD, Ashburn, Virginia
But pushing from Europe.
–remote-only works a lot faster
Having a similar issue.
Tries pushing, shows pushing all the data 1.193GB in my case, stalls for a while then goes for a retry.
For some reason this has happened for over 30 minutes in multiple attempts. I’m using the remote builder. A different layer of 2.1GB was pushed (in this same deploy, a few minutes ago), and after a couple retries it managed to get pushed. This 1.193GB one has been failing over and over.
I am having difficulties on pushing my image also.
flyctl deploy \
--config config/fly.toml \
--wait-timeout 1200 \
--remote-only
Would anybody help on resolving this error?