Build instance ran out of space

Topic says it all. Ran out of space while copying to from one build stage to another.

#26 ERROR: failed to mkdir /data/docker/overlay2/rsz9gzctwcktisz4yf8brj5x6/merged/app/vendor/bundle/ruby/3.0.0/gems/unf_ext- mkdir /data/docker/overlay2/rsz9gzctwcktisz4yf8brj5x6/merged/app/vendor/bundle/ruby/3.0.0/gems/unf_ext- no space left on device

I’d guess this instance needs pruning, so may be it would be smart to add a few cron jobs to the build instances, or increase the volume size? Is this something we can do ourselves?

Here’s what I run on another server to keep things tidy:

0 3 * * * docker system prune -f && systemctl restart docker.service
0 3 * * * docker image prune -a --force --filter until=48h

We’ve been meaning to prune when disk space is low. In the mean time you can run the prune command over ssh or delete the builder app and let a new one replace it on the next build.

I removed the app but now see:

Error error connecting to docker: Error waiting for remote builder app: context deadline exceeded

I noticed here that the app came up in ord but a volume was created in dfw. Guessing this caused the problem, I created a volume in ord which seems to have kicked it into action.