Fly builders running out of disk space causing deploy to fail

Starting late yesterday evening our application began failing to deploy due to its build process failing halfway through from disk space exhaustion.

Snippet from the job:

#21 24.06 error: could not compile `crunchy` due to previous error
#21 24.06 warning: build failed, waiting for other jobs to finish...
#21 24.17 error: failed to write /usr/src/app/target/release/deps/rmetaNWeNef/lib.rmeta: No space left on device (os error 28)
#21 24.17 
#21 24.53 LLVM ERROR: IO failure on output stream: No space left on device
#21 24.55 error: build failed

Is there a published limit on the build environment resources that applications can use with remote builders?

Hello! We don’t really have a limit, but remote builders are created with 50gb volumes by default. There’s a few workarounds:

  • delete the remote builder app so you get a fresh one on the next build
  • ssh into the builder and run docker system prune to purge cache/dangling layers

And this might work, though I haven’t tried it:

  • create a new volume with the same name and a bigger size (eg flyctl volumes create <name> --size 100gb) then delete the old one. next start should use the bigger one

Expanding volumes is on our roadmap, but no timeframe.