Flaky deployments from Circle CI

Questions / Help
1

We are using Circle CI workflows to deploy to Fly. The relevant step:

      - run:
          name: "Deploy to Fly"
          command: |
            curl -L https://fly.io/install.sh | sh
            export FLYCTL_INSTALL="/home/circleci/.fly"
            export PATH="$FLYCTL_INSTALL/bin:$PATH"
            flyctl deploy --remote-only -e GIT_SHA="$GIT_SHA" -e GIT_AUTHOR="$GIT_AUTHOR"

The last few weeks, about 50% of deploys fail to complete. Log output in Circle typically shows it being “stuck” on some step, like asset deploy (it varies, sometimes it get stuck on mix.compile, etc.)

=> [builder 18/22] RUN mix assets.deploy                                741.2s

If I peek in the Monitoring tab for the builder in Fly, the logs typically end with something like this:

 2022-09-13T14:49:26.373 app[1781950a509589] iad [info] time="2022-09-13T14:49:26.372907809Z" level=info msg="gracefully stopped\n"

2022-09-13T14:49:26.869 app[1781950a509589] iad [info] time="2022-09-13T14:49:26.869308895Z" level=debug msg="checking docker activity"

2022-09-13T14:49:26.869 app[1781950a509589] iad [info] time="2022-09-13T14:49:26.869771817Z" level=debug msg="Calling GET /v1.41/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0"

2022-09-13T14:49:27.379 app[1781950a509589] iad [info] time="2022-09-13T14:49:27.379648181Z" level=info msg="dockerd has exited"

2022-09-13T14:49:30.772 runner[1781950a509589] iad [info] machine exited with exit code 0, not restarting

Manually firing a deploy (using --remote-only) from my local machine typically succeeds, so I’m not sure if there’s something going on Circle CI-specific here.

2

A shot in the dark: Consider switching to a brand new Fly Builder to see if that alleviates the problem?

# grab the name of the builder
flyctl apps list | grep 'fly-builder' | cut -f1

# destroy (exercise caution!)
flyctl apps destroy <builder-name>

The last few weeks, about 50% of deploys fail to complete

May be: Creating release error when trying to deploy rails app for the first time - #2 by ignoramous?

3

Thanks, I have tried that twice and the problem unfortunately keeps cropping up :confused:

4

@derrickreimer Your problem with CI deploys may be unrelated to peers, however it’s possible this problem may now have been solved. Or at least reduced. Check out:

2 Likes