Keep getting this when doing remote builds - the solution for me has been to delete the builder and then allow fly to generate a new one. Seams like there needs to be consistent pruning for builders or this could be a bug. Happy to debug or offer any insights - would love to get this fixed!
Are there any other logs or information around the traceback of “no space left on device”? Not knowing much but I would assume that there could be an error with the runner but I would need to more about how you build your app. Like if you use a proc file or ci/cd (like github actions) with a toml file.
After a couple builds I get this error when doing a COPY: /data/docker/overlay2/9pdz2k76pdlhpo9cl83ui36x2/merged/app/node_modules/eslint/lib/rules/no-undef-init.js: no space left on device. Its annoying and I really dont like having to kill my fly builder on ever few builds (this error comes from the fly builder because im doing a simple remote build inside of a github action using the fly CLI). The other thing that is interesting is that my fly builder never maxes out on memory:
The only solution i have found is to SSH into the remote builder before my CI/CD runs and prune everything - this feels very hacky and something Fly should automatically handle (It also makes the very next build take forever due to lack of caching). Let me know if there are any better solutions or if there is something I am doing wrong!
I also am using a payed plan for fly so this really shouldnt be happening
Not that long ago, we introduced a feature that cleans up unused images from remote builder apps. I’m curious, how many times can you deploy with the same remote builder app before you get this error? The builder app has a 50GB volume, so it’s surprising that you’d have to delete it every few deploys.
Its about 3 times total - even still, when it throws this error, I never see the storage maxed out in the dashboard and the problem simply continues to happen after a few builds on a freshly created machine! The other thing is that my images are only 500-750mb MAX!
I had been following this issue where an image seemed to bigger than it should be for a rails app. Is your site a rails app?
If not, I’ll try to get some help on figuring out why your builder volume fills up so quickly.
Sorry for the inconvenience and I do agree that this is something Fly.io should handle.
Since you can ssh into your remote builder machine, can you check
docker images and
9185dd7c254383:/# docker images REPOSITORY TAG IMAGE ID CREATED SIZE registry.fly.io/heyfly2 deployment-01HADKMVSB3DF3BN9ZWWDTHWN7 05d7dca773e4 5 minutes ago 849MB registry.fly.io/heyfly2 deployment-01HADKRNQ7VVNFRJK75GSVH3JH e6e6dd37ab79 5 minutes ago 849MB
(05d7dca773e4 is one of the image IDs above)
9185dd7c254383:/# docker history 05d7dca773e4 IMAGE CREATED CREATED BY SIZE COMMENT 05d7dca773e4 5 minutes ago ENTRYPOINT ["/bin/sh" "-c" "/app/helloflyx"] 0B buildkit.dockerfile.v0 <missing> 5 minutes ago RUN /bin/sh -c env > /env.txt # buildkit 144B buildkit.dockerfile.v0 <missing> 5 minutes ago RUN /bin/sh -c cd /app && go build # buildkit 71.4MB buildkit.dockerfile.v0 <missing> 6 minutes ago COPY go.mod go.sum main.go page.html /app # … 2.25kB buildkit.dockerfile.v0 <missing> 6 minutes ago RUN /bin/sh -c mkdir /app # buildkit 0B buildkit.dockerfile.v0 <missing> 5 months ago /bin/sh -c #(nop) WORKDIR /go 0B <missing> 5 months ago /bin/sh -c mkdir -p "$GOPATH/src" "$GOPATH/b… 0B <missing> 5 months ago /bin/sh -c #(nop) ENV PATH=/go/bin:/usr/loc… 0B <missing> 5 months ago /bin/sh -c #(nop) ENV GOPATH=/go 0B <missing> 5 months ago /bin/sh -c set -eux; arch="$(dpkg --print-a… 244MB <missing> 5 months ago /bin/sh -c #(nop) ENV GOLANG_VERSION=1.20.3 0B <missing> 5 months ago /bin/sh -c #(nop) ENV PATH=/usr/local/go/bi… 0B <missing> 5 months ago /bin/sh -c set -eux; apt-get update; apt-g… 228MB <missing> 5 months ago /bin/sh -c apt-get update && apt-get install… 152MB <missing> 5 months ago /bin/sh -c set -ex; if ! command -v gpg > /… 19MB <missing> 5 months ago /bin/sh -c set -eux; apt-get update; apt-g… 10.7MB <missing> 5 months ago /bin/sh -c #(nop) CMD ["bash"] 0B <missing> 5 months ago /bin/sh -c #(nop) ADD file:f2d012660f882f319… 124MB 9185dd7c254383:/#
This will give you per-layer breakdown.
Please feel free to use your support email if you feel uncomfortable sharing the breakdown like above.
This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.