We’ve been testing remote builders lately and it seems we always have to manually kill our remote builders, because they run out of space and don’t clear anything up (solution of killing them manually is mentioned here: “no space left on device” when remote building )
We don’t see a practical way to use them in our CI pipeline so we ask ourselves what we’re doing wrong here. Is there something we could miss that prevents remote builders from clearing space?
I don’t think you are doing anything wrong. It’s just that currently remote builders are started with XGB of storage (presumably using a volume, like user apps). That storage can not be auto-resized. So eventually it gets full. As far as I know the only solutions (for now) are:
Keep the size as small as possible. Not always in your control, naturally. But e.g use a small base image etc
Delete the builder (forcing a new one to be started on the next deploy, with an empty volume)
Don’t use remote builders. Some CI have local Docker and if so you could use that to build an image (with --local-only)
Volumes are being improved. I would assume remote builders will then benefit from that too e.g