Image pulling fails constantly

philemon.khor · September 21, 2021, 4:50am

Hi, I am constantly facing errors while deploying to hkg. Please help.

2021-09-21T03:13:57.543313913Z runner[cca08315] hkg [info] Starting instance
2021-09-21T03:13:57.592029913Z runner[cca08315] hkg [info] Configuring virtual machine
2021-09-21T03:13:57.593902747Z runner[cca08315] hkg [info] Pulling container image
2021-09-21T03:14:00.665404925Z runner[cca08315] hkg [info] Pull failed, retrying (attempt #0)
2021-09-21T03:14:01.627750631Z runner[cca08315] hkg [info] Pull failed, retrying (attempt #1)
2021-09-21T03:14:02.605847879Z runner[cca08315] hkg [info] Pull failed, retrying (attempt #2)
2021-09-21T03:14:02.605852460Z runner[cca08315] hkg [info] Pulling image failed
2021-09-21T03:32:39.306260639Z runner[8e144483] hkg [info] Starting instance
2021-09-21T03:32:39.358549657Z runner[8e144483] hkg [info] Pulling container image
2021-09-21T03:32:43.329617416Z runner[8e144483] hkg [info] Pull failed, retrying (attempt #0)
2021-09-21T03:32:44.278406713Z runner[8e144483] hkg [info] Pull failed, retrying (attempt #1)
2021-09-21T03:32:45.221872533Z runner[8e144483] hkg [info] Pulling image failed
2021-09-21T03:59:44.882745330Z runner[8aa96dd5] hkg [info] Starting instance
2021-09-21T03:59:44.926397247Z runner[8aa96dd5] hkg [info] Configuring virtual machine
2021-09-21T03:59:44.928435558Z runner[8aa96dd5] hkg [info] Pulling container image
2021-09-21T03:59:48.272110539Z runner[8aa96dd5] hkg [info] Pull failed, retrying (attempt #0)
2021-09-21T03:59:49.241832033Z runner[8aa96dd5] hkg [info] Pull failed, retrying (attempt #1)
2021-09-21T03:59:50.235267120Z runner[8aa96dd5] hkg [info] Pull failed, retrying (attempt #2)
2021-09-21T03:59:50.235271170Z runner[8aa96dd5] hkg [info] Pulling image failed
2021-09-21T04:00:17.446138224Z runner[e58af7fc] hkg [info] Starting instance
2021-09-21T04:00:17.484387949Z runner[e58af7fc] hkg [info] Configuring virtual machine
2021-09-21T04:00:17.486441383Z runner[e58af7fc] hkg [info] Pulling container image
2021-09-21T04:00:19.080423086Z runner[e58af7fc] hkg [info] Pull failed, retrying (attempt #0)
2021-09-21T04:00:20.037398623Z runner[e58af7fc] hkg [info] Pull failed, retrying (attempt #1)
2021-09-21T04:00:20.979801232Z runner[e58af7fc] hkg [info] Pull failed, retrying (attempt #2)
2021-09-21T04:00:20.979805180Z runner[e58af7fc] hkg [info] Pulling image failed
2021-09-21T04:25:20.891155063Z runner[5b5a31a2] hkg [info] Starting instance
2021-09-21T04:25:20.933175090Z runner[5b5a31a2] hkg [info] Configuring virtual machine
2021-09-21T04:25:20.935118200Z runner[5b5a31a2] hkg [info] Pulling container image
2021-09-21T04:25:21.028389269Z runner[5b5a31a2] hkg [info] Pull failed, retrying (attempt #0)
2021-09-21T04:25:23.824016573Z runner[5b5a31a2] hkg [info] Pull failed, retrying (attempt #1)
2021-09-21T04:25:24.745375393Z runner[5b5a31a2] hkg [info] Pull failed, retrying (attempt #2)
2021-09-21T04:25:24.745380349Z runner[5b5a31a2] hkg [info] Pulling image failed

john · September 21, 2021, 5:51am

Hi there,
I am getting what looks like the same issue - for both my apps, right at the last part of deployment:
(elixir app)
Starting instance
Configuring virtual machine
Pulling container image
Pull failed, retrying (attempt #0)
Pull failed, retrying (attempt #1)
Pulling image failed
They are both in Hong Kong (looks like yours is too).

I tried deleting the builder (cargo culting), no effect. Now I will try (not sure how - assume its in the docs to move my apps to another zone).

jsierles · September 21, 2021, 8:37am

We’re having some trouble today with the hkg region. We’ll update here as soon as we’ve fixed the problem.

Are you running in multiple regions? If so, you should be able to remove the hkg region temporarily using flyctl regions remove hkg.

If you are only running in hkg, you can try adding another region with flyctl regions add sin, then remove hkg with flyctl regions remove hkg.

john · September 21, 2021, 8:49am

Thanks Joshua!

philemon.khor · September 21, 2021, 8:55am

Thanks for the clarification @jsierles

kurt · September 21, 2021, 1:56pm

This should be fixed now, our Docker registry cache in Hong Kong got itself into a state where it couldn’t authenticate people. We think we know why and we’re rolling out some improvements today.

john · September 24, 2021, 4:49am

Seems to be happening again today in hkg, unfortunately.

kurt · September 24, 2021, 1:58pm

@john did you see something in your app logs? I don’t see any errors, just one in Paris that retried successfully.

john · September 24, 2021, 10:02pm

Hi @kurt I had one manual failure (where I fly deploy from the console), I don’t have the log for that - it was the same one Pulling image failed - but I also did a git hub action deploy before that which prompted the manual deploy, logs below . Trying an hour or so later was fine.

Generated wwell app

[1023](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1023)* assembling wwell-0.1.0 on MIX_ENV=prod

[1024](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1024)* using config/runtime.exs to configure the release at runtime

[1025](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1025)* creating _build/prod/rel/wwell/releases/0.1.0/vm.args

[1026](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1026)* creating _build/prod/rel/wwell/releases/0.1.0/env.sh

[1027](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1027)

[1028](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1028)Release created at _build/prod/rel/wwell!

[1029](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1029)

[1030](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1030) # To start your system

[1031](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1031) _build/prod/rel/wwell/bin/wwell start

[1032](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1032)

[1033](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1033)Once the release is running:

[1034](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1034)

[1035](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1035) # To connect to it remotely

[1036](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1036) _build/prod/rel/wwell/bin/wwell remote

[1037](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1037)

[1038](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1038) # To stop it gracefully (you may also send SIGINT/SIGTERM)

[1039](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1039) _build/prod/rel/wwell/bin/wwell stop

[1040](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1040)

[1041](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1041)To list all commands:

[1042](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1042)

[1043](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1043) _build/prod/rel/wwell/bin/wwell

[1044](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1044)

[1045](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1045) ---> 3d9cd7618977

[1046](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1046)Step 20/30 : FROM alpine:3.13.3 AS app

[1047](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1047)3.13.3: Pulling from library/alpine

[1048](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1048)9aae54b2144e: Already exists

[1049](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1049)Digest: sha256:826f70e0ac33e99a72cf20fb0571245a8fee52d68cb26d8bc58e53bfa65dcdfa

[1050](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1050)Status: Downloaded newer image for alpine:3.13.3

[1051](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1051) ---> 302aba9ce190

[1052](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1052)Step 21/30 : RUN apk add --no-cache libstdc++ openssl ncurses-libs

[1053](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1053) ---> Running in cebf1e4209ec

[1054](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1054)fetch https://dl-cdn.alpinelinux.org/alpine/v3.13/main/x86_64/APKINDEX.tar.gz

[1055](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1055)fetch https://dl-cdn.alpinelinux.org/alpine/v3.13/community/x86_64/APKINDEX.tar.gz

[1056](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1056)(1/5) Installing libgcc (10.2.1_pre1-r3)

[1057](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1057)(2/5) Installing libstdc++ (10.2.1_pre1-r3)

[1058](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1058)(3/5) Installing ncurses-terminfo-base (6.2_p20210109-r0)

[1059](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1059)(4/5) Installing ncurses-libs (6.2_p20210109-r0)

[1060](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1060)(5/5) Installing openssl (1.1.1l-r0)

[1061](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1061)Executing busybox-1.32.1-r3.trigger

[1062](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1062)OK: 9 MiB in 19 packages

[1063](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1063) ---> c8ee10605296

[1064](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1064)Step 22/30 : WORKDIR /app

[1065](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1065) ---> Running in 3025494fe608

[1066](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1066) ---> 008dc07569b9

[1067](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1067)Step 23/30 : RUN chown nobody:nobody /app

[1068](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1068) ---> Running in 0c263408d61e

[1069](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1069) ---> 4c3460af0b63

[1070](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1070)Step 24/30 : USER nobody:nobody

[1071](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1071) ---> Running in 8c9436a53d0c

[1072](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1072) ---> 6bfc2f3c96d1

[1073](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1073)Step 25/30 : COPY --from=build --chown=nobody:nobody /app/_build/prod/rel/wwell ./

[1074](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1074) ---> 82f2c9c8df00

[1075](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1075)Step 26/30 : ENV HOME=/app

[1076](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1076) ---> Running in c37816f89d8b

[1077](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1077) ---> 518ff7dfc7c2

[1078](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1078)Step 27/30 : ENV MIX_ENV=prod

[1079](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1079) ---> Running in 66e00a9b1207

[1080](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1080) ---> a0c6e27e4a88

[1081](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1081)Step 28/30 : ENV SECRET_KEY_BASE=nokey

[1082](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1082) ---> Running in 31dfc2fc2e5d

[1083](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1083) ---> 283efae02526

[1084](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1084)Step 29/30 : ENV PORT=4000

[1085](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1085) ---> Running in 07ae1ea516f9

[1086](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1086) ---> 6851125d4b3c

[1087](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1087)Step 30/30 : CMD ["bin/wwell", "start"]

[1088](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1088) ---> Running in 34c04f635d0b

[1089](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1089) ---> 77a33ac9bc69

[1090](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1090)Successfully built 77a33ac9bc69

[1091](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1091)Successfully tagged registry.fly.io/spring-feather-8065:deployment-1632457439

[1092](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1092)--> Building image done

[1093](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1093)==> Pushing image to fly

[1094](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1094)The push refers to repository [registry.fly.io/spring-feather-8065]

[1095](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1095)8792a1158c1c: Preparing

[1096](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1096)1280200545e7: Preparing

[1097](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1097)8a9f72fefa84: Preparing

[1098](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1098)cefee391e7e1: Preparing

[1099](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1099)0f7b3ff8b310: Preparing

[1100](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1100)0f7b3ff8b310: Layer already exists

[1101](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1101)8a9f72fefa84: Pushed

[1102](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1102)1280200545e7: Pushed

[1103](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1103)cefee391e7e1: Pushed

[1104](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1104)8792a1158c1c: Pushed

[1105](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1105)deployment-1632457439: digest: sha256:65ccb07de281c736a94e17a1dc6a92045729f5183831ce0dafa2879dfa146180 size: 1363

[1106](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1106)--> Pushing image done

[1107](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1107)Image: registry.fly.io/spring-feather-8065:deployment-1632457439

[1108](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1108)Image size: 41 MB

[1109](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1109)==> Creating release

[1110](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1110)Release v135 created

[1111](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1111)Release command detected: this new release will not be available until the command succeeds.

[1112](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1112)

[1113](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1113)You can detach the terminal anytime without stopping the deployment

[1114](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1114)==> Release command

[1115](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1115)Command: /app/bin/wwell eval Wwell.Release.migrate

[1116](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1116) Starting instance

[1117](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1117) Configuring virtual machine

[1118](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1118) Pulling container image

[1119](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1119) Unpacking image

[1120](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1120) Preparing kernel init

[1121](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1121) Configuring firecracker

[1122](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1122) Starting virtual machine

[1123](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1123) Preparing to run: `/app/bin/wwell eval Wwell.Release.migrate` as nobody

[1124](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1124) 2021/09/24 04:28:28 listening on [fdaa:0:2f06:a7b:acc:6d8a:635e:2]:22 (DNS: [fdaa::3]:53)

[1125](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1125) 04:28:29.080 [info] Migrations already up

[1126](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1126) Reaped child process with pid: 575 and signal: SIGUSR1, core dumped? false

[1127](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1127) Reaped child process with pid: 577 and signal: SIGUSR1, core dumped? false

[1128](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1128) Main child exited normally with code: 0

[1129](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1129) Starting clean up.

[1130](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1130)

[1131](https://github.com/Wooboard/wwell/runs/3695976383?check_suite_focus=true#step:4:1131)Error Release command failed, deployment aborted

Topic		Replies	Views
Container pull failed on deploy Build debugging	2	245	March 13, 2023
Deployment consistently failing	7	388	February 8, 2021
Pulling image failed on SIN	3	227	January 13, 2023
Deployment stuck at "Pulling container image"	4	833	August 31, 2023
Pulling images failing frequently in cdg	5	261	November 2, 2021

Image pulling fails constantly

Related topics