Error failed to fetch an image or build from source: error rendering push status stream: EOF

constantlm · January 24, 2023, 4:29am

Is the Docker registry currently giving problems? I’ve had issues deploying from around two hours ago:

First I got the following:

 => => exporting layers                                                                                                                                                                                0.0s
 => => writing image sha256:741b8e823415587cd13c4fbf4f79913552ce55da4b211ca8ddf3757a9833fa2e                                                                                                           0.0s
 => => naming to registry.fly.io/igc:deployment-01GQGSNBWNCVKR9QG5JGZMWH6J                                                                                                                             0.0s
--> Building image done
==> Pushing image to fly
The push refers to repository [registry.fly.io/igc]
93201fa7c74e: Layer already exists
7ba47360dcb3: Layer already exists
d408498b1080: Layer already exists
84a7a998a628: Layer already exists
115f617f2a52: Layer already exists
8acc755903da: Layer already exists
4fffc6f589b2: Layer already exists
29df5dbd1dba: Pushing [==================================================>]  517.2MB
29df5dbd1dba: Pushing [==================================================>]  517.2MB
e33bb45a6387: Layer already exists
e9ba2940f15e: Layer already exists
5f70bf18a086: Layer already exists
d45969274e30: Layer already exists
a1fc54409f4f: Layer already exists
67a4178b7d47: Layer already exists

Error failed to fetch an image or build from source: error rendering push status stream: received unexpected HTTP status: 502 Bad Gateway

Then it changed to:

 => exporting to image                                                                                                                                                                                 0.0s
 => => exporting layers                                                                                                                                                                                0.0s
 => => writing image sha256:6df14bd19c33779744fbc31b5e1976a46d91b12423bc2858bf5ee672c0c3dff5                                                                                                           0.0s
 => => naming to registry.fly.io/igc:deployment-01GQGZNQ7R1CF50J6EX7TQV8R8                                                                                                                             0.0s
--> Building image done
==> Pushing image to fly
The push refers to repository [registry.fly.io/igc]
dc24996a849b: Layer already exists
5c6aad85fef0: Layer already exists
1d0afb040c5e: Layer already exists
9f8fff63d81b: Pushing [==================================================>]    168MB
059f753b44d9: Layer already exists
4a4f95642027: Layer already exists
12c56e92a811: Layer already exists
79d137743a61: Pushing [==================================================>]  517.2MB
af639775e6aa: Pushing [==================================================>]  141.8MB
77a711e74fcd: Layer already exists
b19c49d06e4e: Layer already exists
5f70bf18a086: Layer already exists
30c6a4a242e8: Layer already exists
a1fc54409f4f: Layer already exists
67a4178b7d47: Layer already exists
Error failed to fetch an image or build from source: error rendering push status stream: EOF

I’ve been keeping an eye on https://status.flyio.net/ but there’s no metrics for the Registry (from which this error seems to be coming?)

EDIT:

I’ve tried the following to remedy this:

Destroyed the builder machine
Deploy with --no-cache
--local-only which didn’t work because I’m on an M1 and one of the packages are causing QEMU to segfault.

EDIT 2:
Now receiving error:

--> Building image done
==> Pushing image to fly
The push refers to repository [registry.fly.io/igc]
bc380a192fde: Layer already exists
3a0b4ce84892: Layer already exists
71602499f533: Layer already exists
4bfe473d8033: Layer already exists
3766d6640d70: Layer already exists
4d06568f582e: Layer already exists
2cd80361d0dd: Layer already exists
c156411af27f: Pushing [==================================================>]    514MB
f160f94d3cac: Pushing [==================================================>]  141.8MB
8c9132118576: Layer already exists
793904db9459: Layer already exists
5f70bf18a086: Layer already exists
c2df47dab46e: Layer already exists
a1fc54409f4f: Layer already exists
67a4178b7d47: Layer already exists
Error failed to fetch an image or build from source: error rendering push status stream: write tcp [2605:4c40:33:84a3:0:e3e4:69e:1]:60282->[2a09:8280:1:f132:9b03:c39f:b404:f5d2]:443: write: broken pipe

constantlm · January 24, 2023, 9:03pm

This seems to have been fixed. All good now.

s1q · January 25, 2023, 3:51am

This happens to me on 7/10 deployments with Fly and has done ever since I started using it back in November. When it works its great, but it mostly doesn’t.

andrew-erlanger · January 30, 2023, 1:41am

@s1q same here. Happy with Fly in general but deployment always feels like a roll of the dice, and tends to fail more often than it succeeds.

kurt · January 30, 2023, 1:57am

This is almost entirely related to our registry setup. All registry data is kept in s3 in us-east, and the builders / registry instances in far away regions can struggle trying to talk to s3 in us-east.

Making the registry global is a very big, very slow project. We’re working on it!

andrew-erlanger · January 30, 2023, 3:54am

Thanks for the update @kurt and best of luck with everything. I’m still glad we made the switch to Fly despite the deployment hiccups so keep up the good work

bwal · February 1, 2023, 1:38am

Really struggling with this as well. Originally deploying to SYD. That stopped working yesterday afternoon, so now I am trying MIA. Same outcome:

kurt · February 1, 2023, 1:41am

Changing deploy regions affect this, unfortunately. The problem’s arise trying to push an image from where you (or your builder are) to the registry. That happens the same regardless of where you deploy to.

bwal · February 1, 2023, 1:52am

Hi Kurt, are you suggesting that the reason is because im in Australia?

kurt · February 1, 2023, 9:32pm

Yes! Getting at S3 in us-east from Sydney (and even Europe, sometimes) is surprisingly fraught.

@here we just made a change to our registry that might help with these issues. Will you post here again if things improve? I think it’s only a 50/50 chance, but we’re watching it.

andrew-erlanger · February 2, 2023, 10:48pm

Small sample size, but deployment has been seamless for me since you posted this message. Promising signs

PeterUllrich · April 9, 2023, 11:17am

It’s happening again today and I can’t deploy a new application:

➜ git:(main) ✗ fly deploy
==> Verifying app config
--> Verified app config
==> Building image
Remote builder fly-builder-my-builder-123 ready
==> Creating build context
--> Creating build context done
==> Building image with Docker
--> docker host: 20.10.12 linux x86_64
Sending build context to Docker daemon  157.9kB
[+] Building 0.3s (28/28) FINISHED                                                                                            
 => CACHED [internal] load remote build context                                                                          0.0s
 => CACHED copy /context /                                                                                               0.0s
 => [internal] load metadata for docker.io/library/debian:bullseye-20230227-slim                                         0.2s
 => [internal] load metadata for docker.io/hexpm/elixir:1.14.4-erlang-25.3-debian-bullseye-20230227-slim                 0.2s
 => [builder  1/17] FROM docker.io/hexpm/elixir:1.14.4-erlang-25.3-debian-bullseye-20230227-slim@sha256:631e39aa9e52b52  0.0s
 => [stage-1 1/6] FROM docker.io/library/debian:bullseye-20230227-slim@sha256:77f46c1cf862290e750e913defffb2828c889d291  0.0s
 => CACHED [stage-1 2/6] RUN apt-get update -y && apt-get install -y libstdc++6 openssl libncurses5 locales   && apt-ge  0.0s
 => CACHED [stage-1 3/6] RUN sed -i '/en_US.UTF-8/s/^# //g' /etc/locale.gen && locale-gen                                0.0s
 => CACHED [stage-1 4/6] WORKDIR /app                                                                                    0.0s
 => CACHED [stage-1 5/6] RUN chown nobody /app                                                                           0.0s
 => CACHED [builder  2/17] RUN apt-get update -y && apt-get install -y build-essential git     && apt-get clean && rm -  0.0s
 => CACHED [builder  3/17] WORKDIR /app                                                                                  0.0s
 => CACHED [builder  4/17] RUN mix local.hex --force &&     mix local.rebar --force                                      0.0s
 => CACHED [builder  5/17] COPY mix.exs mix.lock ./                                                                      0.0s
 => CACHED [builder  6/17] RUN mix deps.get --only prod                                                                  0.0s
 => CACHED [builder  7/17] RUN mkdir config                                                                              0.0s
 => CACHED [builder  8/17] COPY config/config.exs config/prod.exs config/                                                0.0s
 => CACHED [builder  9/17] RUN mix deps.compile                                                                          0.0s
 => CACHED [builder 10/17] COPY priv priv                                                                                0.0s
 => CACHED [builder 11/17] COPY lib lib                                                                                  0.0s
 => CACHED [builder 12/17] COPY assets assets                                                                            0.0s
 => CACHED [builder 13/17] RUN mix assets.deploy                                                                         0.0s
 => CACHED [builder 14/17] RUN mix compile                                                                               0.0s
 => CACHED [builder 15/17] COPY config/runtime.exs config/                                                               0.0s
 => CACHED [builder 16/17] COPY rel rel                                                                                  0.0s
 => CACHED [builder 17/17] RUN mix release                                                                               0.0s
 => CACHED [stage-1 6/6] COPY --from=builder --chown=nobody:root /app/_build/prod/rel/my-app ./                            0.0s
 => exporting to image                                                                                                   0.0s
 => => exporting layers                                                                                                  0.0s
 => => writing image sha256:516eebf8afac4dda19e9862a6c4ad9a3172aa6ed31acf336bd6113a574072464                             0.0s
 => => naming to registry.fly.io/my-app:deployment-01GXJVJ7T4Z05RXMQ7R568TY3F                                              0.0s
--> Building image done
==> Pushing image to fly
The push refers to repository [registry.fly.io/my-app]
Error failed to fetch an image or build from source: error rendering push status stream: Get "https://registry.fly.io/v2/": EOF

Time: 0h:00m:05s

The same error occurred the first time when I ran fly launch. This is the second time, so it used the cached image locally.

AlphaJae · April 9, 2023, 11:38am

I am also facing a similar issue:

--> Building image done
==> Pushing image to fly
The push refers to repository [registry.fly.io/[my app]]
Error failed to fetch an image or build from source: error rendering push status stream: Get "https://registry.fly.io/v2/": read tcp [2605:4c40:40:7806:0:2473:a9cb:1]:37764->[2a09:8280:1:f132:9b03:c39f:b404:f5d2]:443: read: connection reset by peer

It was working fine just yesterday

begedin · April 9, 2023, 12:57pm

Changing my region from fra to ams helped for me

PeterUllrich · April 9, 2023, 1:28pm

Deleting my builder and changing region to London worked for me.

Topic		Replies	Views
Cannot deploy application	11	994	November 4, 2023
Error: failed to fetch an image or build from source: error rendering push status stream: received unexpected HTTP status: 500 Internal Server Error	6	493	August 16, 2023
failed to fetch an image or build from source: error rendering push status stream: Get "https://registry.fly.io/v2/": dial tcp: lookup registry.fly.io on [fdaa::3]:53: server misbehaving Questions / Help rails , registry	2	155	February 12, 2024
Pushing images loops through retry then Error: failed to fetch an image or build from source Questions / Help	1	202	July 13, 2023
push failed at preparing image. Build debugging	4	229	November 23, 2023

Error failed to fetch an image or build from source: error rendering push status stream: EOF

Related topics