Flaky deploys on a GitLab

Questions / Help Build debugging
1

Hello. I observe flaky builds on a GitLab CI. It happens on a daily basis, at least 1 of 5 builds fails.

Here is the typical log:

$ /root/.fly/bin/flyctl deploy
==> Verifying app config
--> Verified app config
==> Building image
Waiting for remote builder fly-builder-quiet-paper-6915...
Remote builder fly-builder-quiet-paper-6915 ready
==> Creating build context
--> Creating build context done
==> Building image with Docker
--> docker host: 20.10.12 linux x86_64
time="2022-09-06T19:42:32Z" level=error msg="Can't add file /builds/company/project/.dockerignore to tar: io: read/write on closed pipe"
time="2022-09-06T19:42:32Z" level=error msg="Can't close tar writer: io: read/write on closed pipe"

And another one happens quiet often too:

$ /root/.fly/bin/flyctl deploy
==> Verifying app config
--> Verified app config
==> Building image
Waiting for remote builder fly-builder-quiet-paper-6915...
Remote builder fly-builder-quiet-paper-6915 ready
==> Creating build context
--> Creating build context done
==> Building image with Docker
--> docker host: 20.10.12 linux x86_64
#1 [internal] load remote build context

In both cases job just hangs.

It’s never happen when running fly deploy locally, only from GitLab CI.

My .gitlab-ci.yml file:

image: ruby:3.1.2

stages:
  - test
  - deploy

cache:
  key: $CI_PROJECT_NAME
  paths:
    - vendor/ruby

rubocop:
  stage: test
  script:
    - gem install bundler
    - bundle install -j $(nproc) --path vendor
    - bundle exec rubocop

deploy:
  stage: deploy
  before_script:
    - curl -L https://fly.io/install.sh | sh
  script:
    - /root/.fly/bin/flyctl deploy
  only:
    - main
2

Okay, looks like I’ve identified the problem.

I have cache for rubygems inside project dir (vendor). This cache was a part of a Docker build context (it wasn’t excluded in .dockerignore) and was sent to the buildserver. I’ve added this folder to the .dockerignore and now everything is built much faster.

I’ll keep an eye on deployments for a next few days.