Remote Builders never clearing space

Questions / Help Build debugging
1

We’ve been testing remote builders lately and it seems we always have to manually kill our remote builders, because they run out of space and don’t clear anything up (solution of killing them manually is mentioned here: “no space left on device” when remote building )

We don’t see a practical way to use them in our CI pipeline so we ask ourselves what we’re doing wrong here. Is there something we could miss that prevents remote builders from clearing space?

2

I don’t think you are doing anything wrong. It’s just that currently remote builders are started with XGB of storage (presumably using a volume, like user apps). That storage can not be auto-resized. So eventually it gets full. As far as I know the only solutions (for now) are:

  • Keep the size as small as possible. Not always in your control, naturally. But e.g use a small base image etc
  • Delete the builder (forcing a new one to be started on the next deploy, with an empty volume)
  • Don’t use remote builders. Some CI have local Docker and if so you could use that to build an image (with --local-only)

Volumes are being improved. I would assume remote builders will then benefit from that too e.g

3

Thanks for the reply! Perhaps we will go that way and just delete every builder after use.

In the thread I linked @jerome wrote

We have logic in place that should clear space in the event that too much has been allocated. It seems like it was unable to reclaim space in this case? Hmm.

Because of that I thought builders should be able to clear space themselves.

1 Like