Phoenix app deploy starts failing randomly

Phoenix
1

Heya, my current app is using GitHub actions to deploy the application on merge. It’s a very straight forward setup with the mostly untouched Dockerfile generated via the CLI.

GitHub action is running:
Run flyctl deploy --remote-only --config ./fly.staging.toml

My last 3 deployment attempts have failed without any changes in configuration. According to the logs the image is successfully built and uploaded. It seems to fail whenever the image is attempted to be pulled from the registry :thinking:

==> Verifying app config
--> Verified app config
==> Building image
Waiting for remote builder fly-builder-restless-pond-5097...
Remote builder fly-builder-restless-pond-5097 ready
==> Creating build context
--> Creating build context done
==> Building image with Docker
--> docker host: 20.10.12 linux x86_64
Sending build context to Docker daemon  74.39kB

#1 [internal] load remote build context
#1 DONE 0.0s

#2 copy /context /
#2 DONE 0.1s

#4 [internal] load metadata for docker.io/hexpm/elixir:1.13.4-erlang-25.0.4-debian-bullseye-20220801-slim
#4 DONE 0.9s

#3 [internal] load metadata for docker.io/library/debian:bullseye-20220801-slim
#3 DONE 0.8s

#10 [builder  1/17] FROM docker.io/hexpm/elixir:1.13.4-erlang-25.0.4-debian-bullseye-20220801-slim@sha256:5cc9ea651f5ca47c053a39e149276f1647aa927a3aac60dfcebdda0f9dd9ff6f
#10 DONE 0.0s

#5 [stage-1 1/6] FROM docker.io/library/debian:bullseye-20220801-slim@sha256:a811e62769a642241b168ac34f615fb02da863307a14c4432cea8e5a0f9782b8
#5 DONE 0.0s

#16 [builder  7/17] RUN mkdir config
#16 CACHED

#6 [stage-1 2/6] RUN apt-get update -y && apt-get install -y libstdc++6 openssl libncurses5 locales   && apt-get clean && rm -f /var/lib/apt/lists/*_*
#6 CACHED

#7 [stage-1 3/6] RUN sed -i '/en_US.UTF-8/s/^# //g' /etc/locale.gen && locale-gen
#7 CACHED

#21 [builder 12/17] COPY assets assets
#21 CACHED

#12 [builder  3/17] WORKDIR /app
#12 CACHED

#26 [builder 17/17] RUN mix release
#26 CACHED

#14 [builder  5/17] COPY mix.exs mix.lock ./
#14 CACHED

#20 [builder 11/17] COPY lib lib
#20 CACHED

#9 [stage-1 5/6] RUN chown nobody /app
#9 CACHED

#15 [builder  6/17] RUN mix deps.get --only prod
#15 CACHED

#22 [builder 13/17] RUN mix assets.deploy
#22 CACHED

#19 [builder 10/17] COPY priv priv
#19 CACHED

#8 [stage-1 4/6] WORKDIR /app
#8 CACHED

#17 [builder  8/17] COPY config/config.exs config/prod.exs config/
#17 CACHED

#23 [builder 14/17] RUN mix compile
#23 CACHED

#13 [builder  4/17] RUN mix local.hex --force &&   mix local.rebar --force
#13 CACHED

#11 [builder  2/17] RUN apt-get update -y && apt-get install -y build-essential git   && apt-get clean && rm -f /var/lib/apt/lists/*_*
#11 CACHED

#25 [builder 16/17] COPY rel rel
#25 CACHED

#18 [builder  9/17] RUN mix deps.compile
#18 CACHED

#24 [builder 15/17] COPY config/runtime.exs config/
#24 CACHED

#27 [stage-1 6/6] COPY --from=builder --chown=nobody:root /app/_build/prod/rel/app ./
#27 CACHED

#28 exporting to image
#28 exporting layers done
#28 writing image sha256:520e2f0ff103c45081347057f8063d6387d231b2edc256c3fc6ad06818ce5dc7 done
#28 naming to registry.fly.io/xxxxx:deployment-01GEF0GC68G7QHK58GVD7EWHN9 done
#28 DONE 0.0s
--> Building image done
==> Pushing image to fly
The push refers to repository [registry.fly.io/xxxxxx]
f0d8037ac249: Preparing
049608b4421d: Preparing
b7fd6450897f: Preparing
1a3eebbb4d98: Preparing
c2b3ff9ff80b: Preparing
92a4e8a3140f: Preparing
92a4e8a3140f: Waiting
b7fd6450897f: Layer already exists
f0d8037ac249: Layer already exists
1a3eebbb4d98: Layer already exists
049608b4421d: Layer already exists
c2b3ff9ff80b: Layer already exists
92a4e8a3140f: Layer already exists
deployment-01GEF0GC68G7QHK58GVD7EWHN9: digest: sha256:bcf64a2ecd3392740899f5646756cd92376a6cd6e4ba038047e9cde01f46e9d4 size: 1576
--> Pushing image done
image: registry.fly.io/xxxxxx:deployment-01GEF0GC68G7QHK58GVD7EWHN9
image size: 126 MB
==> Creating release
--> release v50 created

--> You can detach the terminal anytime without stopping the deployment
==> Release command detected: /app/bin/migrate

--> This release will not be available until the release command succeeds.
	 Starting instance
	 Configuring virtual machine
	 Pulling container image
	 Pull failed, retrying (attempt #0)
	 Pull failed, retrying (attempt #1)
	 Starting instance
	 Configuring virtual machine
	 Pulling container image
	 Pull failed, retrying (attempt #0)
	 Pull failed, retrying (attempt #1)
	 Starting instance
	 Configuring virtual machine
	 Pulling container image
	 Pull failed, retrying (attempt #0)
	 Pull failed, retrying (attempt #1)
	 Pulling image failed
	 Pulling image failed
	 Pulling image failed
Error release command failed, deployment aborted

Not sure if I am the only one running into this :thinking: I am trying to figure out whether there is anything within my reach to do something about this as it seems to be erroring out in getting the just built image :sweat_smile:

2

Seems to have resolved itself :person_shrugging: No changes were made so I guess just a network hiccup within Fly.io itself :sweat_smile: