Fly builders running out of disk space causing deploy to fail

Questions / Help Build debugging
#1

Starting late yesterday evening our application began failing to deploy due to its build process failing halfway through from disk space exhaustion.

Snippet from the job:

#21 24.06 error: could not compile `crunchy` due to previous error
#21 24.06 warning: build failed, waiting for other jobs to finish...
#21 24.17 error: failed to write /usr/src/app/target/release/deps/rmetaNWeNef/lib.rmeta: No space left on device (os error 28)
#21 24.17 
#21 24.53 LLVM ERROR: IO failure on output stream: No space left on device
#21 24.55 error: build failed

Is there a published limit on the build environment resources that applications can use with remote builders?

#2

Hello! We don’t really have a limit, but remote builders are created with 50gb volumes by default. There’s a few workarounds:

  • delete the remote builder app so you get a fresh one on the next build
  • ssh into the builder and run docker system prune to purge cache/dangling layers

And this might work, though I haven’t tried it:

  • create a new volume with the same name and a bigger size (eg flyctl volumes create <name> --size 100gb) then delete the old one. next start should use the bigger one

Expanding volumes is on our roadmap, but no timeframe.