Deployment Failed from CircleCI Broken Pipe Error

Just ran a deployment for a few apps and got a broken pipe error on about 5/10 of the apps that were deploying from CircleCI.

Here is one of the apps output both during the deployment and after (status)

App
  Name     = better-cart-monster-bash-checkout-api-prod
  Owner    = better-cart
  Version  = 115
  Status   = running
  Hostname = better-cart-monster-bash-checkout-api-prod.fly.dev

Instances
ID       PROCESS VERSION REGION DESIRED STATUS  HEALTH CHECKS      RESTARTS CREATED
af837b9b app     115     ord    run     running 2 total, 2 passing 0        2021-12-15T19:00:04Z
1d4cdb6f app     115     ord    run     running 2 total, 2 passing 0        2021-12-15T18:59:21Z

The errors are consistent on all 5/10 deployments that failed. We are going to retry the deployments and see if everything works, but thought I would drop this here to possibly help/identify any issue that may be at play.

Retrying the deploys on all failed app deployments was successful.

I am starting to get a write: broken pipe error on 80% of my deployments and its really eating up circleci credits to wait for the already long deploys and then have to do it all again.

Is there anything that we can do to resolve this going forward?

Not sure about the underlying issue as I’m guessing it’s a network issue that’s resulting in a failed connection.

Are you doing a local docker build in CircleCI, and then pushing that? If so, normally that would be fine and should indeed work. However since you say it’s failing 80% of the time, perhaps it would be worth trying using Fly’s remote builder and see if you get a better result with one of them? They used to be initially slower because of the time taken for a peer (up to 3 minutes) but Fly have since resolved that issue and so now are much faster to start.

If so, try using --remote-only and that should skip any local docker and force a remote build.

Gotcha, so for more context, here is the job that build the docker image and deploys it to flys registry.

deploy-to-fly-local-build:
    parameters:
      package_name:
        type: string
      fly_app_name:
        type: string
      docker_path:
        type: string
        default: Dockerfile
      config_path:
        type: string
        default: fly.toml

    working_directory: ~/project/<< parameters.package_name >>

    resource_class: medium
    machine:
      image: ubuntu-2004:202104-01
      docker_layer_caching: true

    steps:
      - attach_workspace:
          at: ~/project
      - run:
          name: Docker Build
          command: |
            docker build -t better-cart/<< parameters.fly_app_name >> -t registry.fly.io/<< parameters.fly_app_name >>:$CIRCLE_BUILD_NUM -f ./<< parameters.docker_path >> .
      - run:
          name: Deploy to Fly.io
          command: |
            wget -qO- 'https://getfly.fly.dev/linux-x86-64/flyctl.tgz' | tar xz
            ./flyctl auth docker
            docker push registry.fly.io/<< parameters.fly_app_name >>:$CIRCLE_BUILD_NUM
            ./flyctl deploy -i registry.fly.io/<< parameters.fly_app_name >>:$CIRCLE_BUILD_NUM -a << parameters.fly_app_name >> --config << parameters.config_path >> --detach

If I understand correctly, you are suggesting to remove the Docker Build step and add --remote-only to the Deploy to Fly.io step?

Also, I feel there is optimizations that are missing that can decrease the size of the container, any suggestions you would have for doing so? (this is a monorepo as you can see by the working_directory, etc.)

Ah yes so what you have should work. Like I say, not sure why it seems to randomly fail.

However yes, the remote builder avoids having to use docker locally at all. For now remote builds are free (hopefully always will be!) so it’s worth seeing if it helps.

So yep, this would not be needed

- run:
  name: Docker Build
  command: docker build -t better-cart/<< parameters.fly_app_name >> -t registry.fly.io/<< parameters.fly_app_name >>:$CIRCLE_BUILD_NUM -f ./<< parameters.docker_path >> .

… because that is done by Fly, at their end.

And your deploy command would then be two lines, instead of four (one to install the Fly CLI, as you have got now, and then one to deploy). You would not need the middle two lines.

I don’t want to type an exact command because I don’t want to screw up your build as it’s more complicated than mine with params etc. But basically when you do

fly deploy --remote-only

… it will see you have a Dockerfile and so that will be used to do the remote build. I would remove the --detach maybe and so you’d see what it’s doing in realtime.

That assumes you have the Fly token in the envionment, else that would need passing in to the command too.

As regards the size, that makes less of an issue when doing a remote build but yes, small is always good. The first thing to look at would be the size of the base image. Instead of using a full ubuntu base you could use an alpine or slim image. Depends on your app, language etc, but I use nodejs and so use node slim as a base (because alpine has quirks with dns, but that’s another story).

There are lots of Dockerfile tricks like minimizing layers and using multi-stage builds which I’m no expert at as fortunately my Dockerfiles are super simple so I’ve never had to master them. But step one is getting it working at all, and then step two can be optimising :slight_smile:

1 Like

Well we gave remote builders a shot and it only slowed down the builds on circle even more.

We are also are not getting more reliable deployments as we have started to see the following error message when using remove builders:

Error failed to fetch an image or build from source: error connecting to docker: failed building options: Timeout on AddWireGuardPeer.peerip

Any ideas?

We have seen a lot of issues with CircleCI and remote builders. I think some of their environments won’t allow outbound UDP to the WireGuard port, which is probably why you’re getting that most recent error.

Are you still getting those broken pipe errors? You can see if that’s a CircleCI issue by setting up another Docker registry and pushing there.

Yup, we went back to local builds and now we are still seeing very slow deploys and connection issues:

write tcp 10.128.8.98:50508->77.83.143.221:443: write: connection reset by peer

write tcp 10.142.0.42:49892->77.83.143.221:443: write: broken pipe

That’s the write to the registry failing. But as far as we can tell, the registry is working fine! All our external monitoring, pushes from Github Actions, and pushes from Buildkite are fast and reliable.

If you can SSH into your CircleCI instance, it might be worth installing mtr and then run mtr registry.fly.io. Also curl https://debug.fly.dev. You can paste the output here, but if CircleCI stuff is routed funny that might tell us what’s up.