I have been able to deploy to successfully deploy to fly and it suddenly stopped working today.
Yesterday there were some issues on the https://status.flyio.net/ which prevented me to deploy but I have been able to deploy yesterday after these issues were resolved.
I’m trying to deploy again today but I’m receiving error:
Error: release command failed - aborting deployment. error running release_command machine: timeout reached waiting for machine to stopped failed to wait for VM e784ee4bd29d28 in stopped state: Get "https://api.machines.dev/v1/apps/baker-street-prod-sa/machines/e784ee4bd29d28/wait?instance_id=01HAVS31K8MFM9FMHSRDSQM9GA&state=stopped&timeout=60": net/http: request canceled
It seems like it has to do with my e784ee4bd29d28 machine that does not go into the started state.
While I’m looking at the logs I can see the following output:
2023-09-21T11:59:05Z runner[e784ee4bd29d28] lhr [info]Pulling container image registry.fly.io/baker-street-prod-sa:deployment-01HAVS25N2JFJVT2M4HPQE15SR
2023-09-21T11:59:10Z runner[e784ee4bd29d28] lhr [info]Pulling container image registry.fly.io/baker-street-prod-sa:deployment-01HAVS25N2JFJVT2M4HPQE15SR
2023-09-21T11:59:17Z runner[e784ee4bd29d28] lhr [info]Pulling container image registry.fly.io/baker-street-prod-sa:deployment-01HAVS25N2JFJVT2M4HPQE15SR
2023-09-21T11:59:24Z runner[e784ee4bd29d28] lhr [info]Pulling container image registry.fly.io/baker-street-prod-sa:deployment-01HAVS25N2JFJVT2M4HPQE15SR
2023-09-21T11:59:26Z runner[e784ee4bd29d28] lhr [error]Pulling image failed, retrying...
I’m running this command to action the deploy: LOG_LEVEl=debug fly deploy -c fly.production-sa.toml.
I’m attaching the full output of what my deploy command produces:
==> Verifying app config
Validating fly.production-sa.toml
Platform: machines
✓ Configuration is valid
--> Verified app config
==> Building image
Remote builder fly-builder-billowing-smoke-5915 ready
==> Building image with Docker
--> docker host: 20.10.12 linux x86_64
[+] Building 31.8s (32/32) FINISHED
=> [internal] load build definition from Dockerfile 1.0s
=> => transferring dockerfile: 32B 1.0s
=> [internal] load .dockerignore 1.0s
=> => transferring context: 35B 1.0s
=> resolve image config for docker.io/docker/dockerfile:1 1.2s
=> CACHED docker-image://docker.io/docker/dockerfile:1@sha256:ac85f380a63b13dfcefa89046420e1781752bab202122f8f50032edf31be0021 0.0s
=> [internal] load build definition from Dockerfile 0.0s
=> [internal] load metadata for docker.io/library/ruby:3.1.2-slim 0.4s
=> [internal] load .dockerignore 0.0s
=> [base 1/4] FROM docker.io/library/ruby:3.1.2-slim@sha256:5d413e301f37cbe63a54dadded2f347496165c1de0dbe342d1f0f0d47e25b2be 0.0s
=> [internal] load build context 3.7s
=> => transferring context: 635.91kB 3.7s
=> CACHED [base 2/4] WORKDIR /rails 0.0s
=> CACHED [base 3/4] RUN gem update --system --no-document && gem install -N bundler 0.0s
=> CACHED [base 4/4] RUN apt-get update -qq && apt-get install --no-install-recommends -y libgeos-dev libgeos-c1v5 && rm -rf /var/lib/apt/lists /var/cache/apt/archives 0.0s
=> CACHED [build 1/14] RUN apt-get update -qq && apt-get install --no-install-recommends -y build-essential curl libpq-dev libvips node-gyp pkg-config python-is-python3 0.0s
=> CACHED [build 2/14] RUN apt update && apt -y upgrade 0.0s
=> CACHED [build 3/14] RUN apt install -y wget 0.0s
=> CACHED [build 4/14] RUN wget https://github.com/wkhtmltopdf/packaging/releases/download/0.12.6-1/wkhtmltox_0.12.6-1.buster_amd64.deb 0.0s
=> CACHED [build 5/14] RUN apt install -y fontconfig libfreetype6 libjpeg62-turbo libpng16-16 libx11-6 libxcb1 libxext6 libxrender1 xfonts-75dpi xfonts-base 0.0s
=> CACHED [build 6/14] RUN apt install -y ./wkhtmltox_0.12.6-1.buster_amd64.deb && rm ./wkhtmltox_0.12.6-1.buster_amd64.deb && chmod +x /usr/local/bin/wkhtmltopdf 0.0s
=> CACHED [build 7/14] RUN curl -sL https://github.com/nodenv/node-build/archive/master.tar.gz | tar xz -C /tmp/ && /tmp/node-build-master/bin/node-build "14.21.3" /usr/local/node && npm install -g yarn@1.22.19 && rm -rf /t 0.0s
=> CACHED [build 8/14] COPY --link Gemfile Gemfile.lock ./ 0.0s
=> CACHED [build 9/14] RUN bundle install && bundle exec bootsnap precompile --gemfile && rm -rf ~/.bundle/ $BUNDLE_PATH/ruby/*/cache $BUNDLE_PATH/ruby/*/bundler/gems/*/.git 0.0s
=> CACHED [build 10/14] COPY --link package.json package-lock.json yarn.lock ./ 0.0s
=> CACHED [build 11/14] RUN yarn install --frozen-lockfile 0.0s
=> [build 12/14] COPY --link . . 1.5s
=> [build 13/14] RUN bundle exec bootsnap precompile app/ lib/ 0.9s
=> [build 14/14] RUN SECRET_KEY_BASE=DUMMY ./bin/rails assets:precompile 6.7s
=> CACHED [stage-2 1/5] RUN apt-get update -qq && apt-get install --no-install-recommends -y curl imagemagick libvips postgresql-client && rm -rf /var/lib/apt/lists /var/cache/apt/archives 0.0s
=> CACHED [stage-2 2/5] COPY --from=build /usr/local/bin/wkhtmltopdf /usr/local/bin/wkhtmltopdf 0.0s
=> CACHED [stage-2 3/5] COPY --from=build /usr/local/bundle /usr/local/bundle 0.0s
=> [stage-2 4/5] COPY --from=build /rails /rails 2.9s
=> [stage-2 5/5] RUN useradd rails --create-home --shell /bin/bash && chown -R rails:rails db log storage tmp public 5.4s
=> exporting to image 4.5s
=> => exporting layers 4.5s
=> => writing image sha256:d172bb15106c760a954be341aed6305c4a6dfb8f6e7bb7297854617404b5a374 0.0s
=> => naming to registry.fly.io/baker-street-prod-sa:deployment-01HAVS7BHS1CV3Z7HBT8V9CNWY 0.0s
--> Building image done
==> Pushing image to fly
The push refers to repository [registry.fly.io/baker-street-prod-sa]
c443262c534a: Pushed
89d8c249d870: Pushed
60ef1e54f645: Layer already exists
76cbaad1b094: Layer already exists
90f67295dc09: Layer already exists
066566e9d393: Layer already exists
536084849616: Layer already exists
476b5b3df4ce: Layer already exists
f0fc506f7982: Layer already exists
46cf0e48c979: Layer already exists
c3e35b2afb23: Layer already exists
b7fcac299347: Layer already exists
ec4a38999118: Layer already exists
deployment-01HAVS7BHS1CV3Z7HBT8V9CNWY: digest: sha256:971e314b30aadc9547cce202d092f7656d7156a38f95b3344d8ff941ba017e8b size: 3056
--> Pushing image done
image: registry.fly.io/baker-street-prod-sa:deployment-01HAVS7BHS1CV3Z7HBT8V9CNWY
image size: 1.2 GB
Watch your deployment at https://fly.io/apps/baker-street-prod-sa/monitoring
Running baker-street-prod-sa release_command: bundle exec rake db:migrate
Waiting for e784ee4bd29d28 to have state: stopped
Error: release command failed - aborting deployment. error running release_command machine: timeout reached waiting for machine to stopped failed to wait for VM e784ee4bd29d28 in stopped state: Get "https://api.machines.dev/v1/apps/baker-street-prod-sa/machines/e784ee4bd29d28/wait?instance_id=01HAVS31K8MFM9FMHSRDSQM9GA&state=stopped&timeout=60": net/http: request canceled
You can increase the timeout with the --wait-timeout flag
