"no space left on device" when remote building

I’ve been using a remote builder successfully for many builds all day. Suddenly, my build starts failing with:

Error error building: error rendering build status stream: open /data/docker/...: no space left on device

Do you have some kind of space leak maybe?

We have logic in place that should clear space in the event that too much has been allocated. It seems like it was unable to reclaim space in this case? Hmm.

Every organization has its own builder. They come with a 50GB volume.

The easiest way to fix the issue is to destroy your builder. Find the name of your builder for the organization that owns the app you’re building by looking for an app that starts with fly-builder- and destroy it with flyctl apps destroy fly-builder-xyz. The next build will create a new builder with a fresh volume.

The harder way to is start a build, let it fail and flyctl ssh console -a fly-builder-xyz, and docker system prune --all --volumes --force to delete everything. To be honest, it’s probably not worth it to do that. I’d be curious to see what’s taking up so much space though.

Builds have been failing consistently or just the one?

2 Likes

I tried to build several times and it failed with out of space errors at different point in the build every time. I deleted the builder and now it’s working.

1 Like

Just hit this. Will destroy builder and see if it helps.
image

My new builder got created but it has no volume and it’s the minimum spec machine possible.

Error failed to fetch an image or build from source: error connecting to docker: Mounts source volume "vol_" does not exist

Ok ignore me, the volume eventually appeared :slight_smile: