Fly deploy fails due to machine not pulling the container image correctly.

I have been able to deploy to successfully deploy to fly and it suddenly stopped working today.

Yesterday there were some issues on the which prevented me to deploy but I have been able to deploy yesterday after these issues were resolved.

I’m trying to deploy again today but I’m receiving error:

Error: release command failed - aborting deployment. error running release_command machine: timeout reached waiting for machine to stopped failed to wait for VM e784ee4bd29d28 in stopped state: Get "": net/http: request canceled

It seems like it has to do with my e784ee4bd29d28 machine that does not go into the started state.

While I’m looking at the logs I can see the following output:

2023-09-21T11:59:05Z runner[e784ee4bd29d28] lhr [info]Pulling container image
2023-09-21T11:59:10Z runner[e784ee4bd29d28] lhr [info]Pulling container image
2023-09-21T11:59:17Z runner[e784ee4bd29d28] lhr [info]Pulling container image
2023-09-21T11:59:24Z runner[e784ee4bd29d28] lhr [info]Pulling container image
2023-09-21T11:59:26Z runner[e784ee4bd29d28] lhr [error]Pulling image failed, retrying...

I’m running this command to action the deploy: LOG_LEVEl=debug fly deploy -c fly.production-sa.toml.

I’m attaching the full output of what my deploy command produces:

==> Verifying app config
Validating fly.production-sa.toml
Platform: machines
✓ Configuration is valid
--> Verified app config
==> Building image
Remote builder fly-builder-billowing-smoke-5915 ready
==> Building image with Docker
--> docker host: 20.10.12 linux x86_64
[+] Building 31.8s (32/32) FINISHED                                                                                                                                                                                                                 
 => [internal] load build definition from Dockerfile                                                                                                                                                                                           1.0s
 => => transferring dockerfile: 32B                                                                                                                                                                                                            1.0s
 => [internal] load .dockerignore                                                                                                                                                                                                              1.0s
 => => transferring context: 35B                                                                                                                                                                                                               1.0s
 => resolve image config for                                                                                                                                                                                     1.2s
 => CACHED docker-image://                                                                                                                0.0s
 => [internal] load build definition from Dockerfile                                                                                                                                                                                           0.0s
 => [internal] load metadata for                                                                                                                                                                             0.4s
 => [internal] load .dockerignore                                                                                                                                                                                                              0.0s
 => [base 1/4] FROM                                                                                                                  0.0s
 => [internal] load build context                                                                                                                                                                                                              3.7s
 => => transferring context: 635.91kB                                                                                                                                                                                                          3.7s
 => CACHED [base 2/4] WORKDIR /rails                                                                                                                                                                                                           0.0s
 => CACHED [base 3/4] RUN gem update --system --no-document &&     gem install -N bundler                                                                                                                                                      0.0s
 => CACHED [base 4/4] RUN apt-get update -qq &&     apt-get install --no-install-recommends -y libgeos-dev libgeos-c1v5 &&     rm -rf /var/lib/apt/lists /var/cache/apt/archives                                                               0.0s
 => CACHED [build  1/14] RUN apt-get update -qq &&     apt-get install --no-install-recommends -y build-essential curl libpq-dev libvips node-gyp pkg-config python-is-python3                                                                 0.0s
 => CACHED [build  2/14] RUN apt update && apt -y upgrade                                                                                                                                                                                      0.0s
 => CACHED [build  3/14] RUN apt install -y wget                                                                                                                                                                                               0.0s
 => CACHED [build  4/14] RUN wget                                                                                                      0.0s
 => CACHED [build  5/14] RUN apt install -y     fontconfig     libfreetype6     libjpeg62-turbo     libpng16-16     libx11-6     libxcb1     libxext6     libxrender1     xfonts-75dpi     xfonts-base                                         0.0s
 => CACHED [build  6/14] RUN apt install -y ./wkhtmltox_0.12.6-1.buster_amd64.deb     && rm ./wkhtmltox_0.12.6-1.buster_amd64.deb     && chmod +x /usr/local/bin/wkhtmltopdf                                                                   0.0s
 => CACHED [build  7/14] RUN curl -sL | tar xz -C /tmp/ &&     /tmp/node-build-master/bin/node-build "14.21.3" /usr/local/node &&     npm install -g yarn@1.22.19 &&     rm -rf /t  0.0s
 => CACHED [build  8/14] COPY --link Gemfile Gemfile.lock ./                                                                                                                                                                                   0.0s
 => CACHED [build  9/14] RUN bundle install &&     bundle exec bootsnap precompile --gemfile &&     rm -rf ~/.bundle/ $BUNDLE_PATH/ruby/*/cache $BUNDLE_PATH/ruby/*/bundler/gems/*/.git                                                        0.0s
 => CACHED [build 10/14] COPY --link package.json package-lock.json yarn.lock ./                                                                                                                                                               0.0s
 => CACHED [build 11/14] RUN yarn install --frozen-lockfile                                                                                                                                                                                    0.0s
 => [build 12/14] COPY --link . .                                                                                                                                                                                                              1.5s
 => [build 13/14] RUN bundle exec bootsnap precompile app/ lib/                                                                                                                                                                                0.9s
 => [build 14/14] RUN SECRET_KEY_BASE=DUMMY ./bin/rails assets:precompile                                                                                                                                                                      6.7s
 => CACHED [stage-2 1/5] RUN apt-get update -qq &&     apt-get install --no-install-recommends -y curl imagemagick libvips postgresql-client &&     rm -rf /var/lib/apt/lists /var/cache/apt/archives                                          0.0s 
 => CACHED [stage-2 2/5] COPY --from=build /usr/local/bin/wkhtmltopdf /usr/local/bin/wkhtmltopdf                                                                                                                                               0.0s 
 => CACHED [stage-2 3/5] COPY --from=build /usr/local/bundle /usr/local/bundle                                                                                                                                                                 0.0s 
 => [stage-2 4/5] COPY --from=build /rails /rails                                                                                                                                                                                              2.9s 
 => [stage-2 5/5] RUN useradd rails --create-home --shell /bin/bash &&     chown -R rails:rails db log storage tmp public                                                                                                                      5.4s 
 => exporting to image                                                                                                                                                                                                                         4.5s
 => => exporting layers                                                                                                                                                                                                                        4.5s
 => => writing image sha256:d172bb15106c760a954be341aed6305c4a6dfb8f6e7bb7297854617404b5a374                                                                                                                                                   0.0s
 => => naming to                                                                                                                                                    0.0s
--> Building image done
==> Pushing image to fly
The push refers to repository []
c443262c534a: Pushed 
89d8c249d870: Pushed 
60ef1e54f645: Layer already exists 
76cbaad1b094: Layer already exists 
90f67295dc09: Layer already exists 
066566e9d393: Layer already exists 
536084849616: Layer already exists 
476b5b3df4ce: Layer already exists 
f0fc506f7982: Layer already exists 
46cf0e48c979: Layer already exists 
c3e35b2afb23: Layer already exists 
b7fcac299347: Layer already exists 
ec4a38999118: Layer already exists 
deployment-01HAVS7BHS1CV3Z7HBT8V9CNWY: digest: sha256:971e314b30aadc9547cce202d092f7656d7156a38f95b3344d8ff941ba017e8b size: 3056
--> Pushing image done
image size: 1.2 GB

Watch your deployment at

Running baker-street-prod-sa release_command: bundle exec rake db:migrate
  Waiting for e784ee4bd29d28 to have state: stopped
Error: release command failed - aborting deployment. error running release_command machine: timeout reached waiting for machine to stopped failed to wait for VM e784ee4bd29d28 in stopped state: Get "": net/http: request canceled
You can increase the timeout with the --wait-timeout flag

Hey there,

I just want to say thank you for your patience in all of this. I understand you weren’t able to deploy because of the issue going on in regards to yesterday. I do want to let you know that we are experiencing some more issues regarding image pulling failing. I would keep checking the status page to make sure you stay up to date on what’s going on. We are working on resolving the issue currently.

We appreciate your patience in this!


Good day,

Thanks for the response. I just want to find out if this issue has been resolved? I don’t see any issues on the status page however it still seems like the image pulling is failing.

Same issue now, … so I think this has not completely resolved !

I have been seeing this issue as well, starting from about the time of the first post till now. The odd thing is that I have two processes and one of them is fine and the other is not, consistently, even though they start from the same code repo.

I am getting this error currently on a number of different apps.

Yes, got same issue yesterday. You have to append to your command in order to fix this ( it asks to use a different registry ) as mentioned in fly’s status page.

But I hope they could fix the problem for their primary registry too, …

I tried that after seeing it in the status page and it does not change anything for me. Same exact issue.

EDIT: Interesting, looking again the word “before” in that message is key it seems.

This did not work for me fly deploy

but this did

fly deploy

Custom metrics are still broken for me. Originally I thought that was related to the deploy timeout, but I am not sure

Ah perfect, thanks @thomasgalibert and @robpc, this has also allowed me to deploy successfully.

I was on my way to setup a whole new server :sweat_smile:

1 Like

For anyone seeing my comment about the custom metrics, here is the thread on the change if you are having similar issues. Custom metrics per process group - #3 by robpc

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.