“no space left on device” when remote building

[darrenbaldwin07]

Keep getting this when doing remote builds - the solution for me has been to delete the builder and then allow fly to generate a new one. Seams like there needs to be consistent pruning for builders or this could be a bug. Happy to debug or offer any insights - would love to get this fixed!

[masterjx9]

Are there any other logs or information around the traceback of “no space left on device”? Not knowing much but I would assume that there could be an error with the runner but I would need to more about how you build your app. Like if you use a proc file or ci/cd (like github actions) with a toml file.

Thanks!

[darrenbaldwin07]

After a couple builds I get this error when doing a COPY: /data/docker/overlay2/9pdz2k76pdlhpo9cl83ui36x2/merged/app/node_modules/eslint/lib/rules/no-undef-init.js: no space left on device. Its annoying and I really dont like having to kill my fly builder on ever few builds (this error comes from the fly builder because im doing a simple remote build inside of a github action using the fly CLI). The other thing that is interesting is that my fly builder never maxes out on memory:

The only solution i have found is to SSH into the remote builder before my CI/CD runs and prune everything - this feels very hacky and something Fly should automatically handle (It also makes the very next build take forever due to lack of caching). Let me know if there are any better solutions or if there is something I am doing wrong!

I also am using a payed plan for fly so this really shouldnt be happening

[andie]

hi @darrenbaldwin07

Not that long ago, we introduced a feature that cleans up unused images from remote builder apps. I’m curious, how many times can you deploy with the same remote builder app before you get this error? The builder app has a 50GB volume, so it’s surprising that you’d have to delete it every few deploys.

[darrenbaldwin07]

Its about 3 times total - even still, when it throws this error, I never see the storage maxed out in the dashboard and the problem simply continues to happen after a few builds on a freshly created machine! The other thing is that my images are only 500-750mb MAX!

[andie]

hey @darrenbaldwin07

I had been following this issue where an image seemed to bigger than it should be for a rails app. Is your site a rails app?

App deployment getting stuck at pushing image to fly

I finally found out what the problem was. Or at least part of the problem. The thing was that the tmp/ directory within my local rails app had grown to be very big, about 5.5 GB in size. Files within the tmp/ directory are supposed to be ignored when sending the context to build the docker image. However, there seems to be a bug in the .dockerignore file that is autogenerated by fly during fly launch that is causing all the files in the tmp/ directory to be sent as context to the docker image, …

If not, I’ll try to get some help on figuring out why your builder volume fills up so quickly.

[kaz]

Sorry for the inconvenience and I do agree that this is something Fly.io should handle.

Since you can ssh into your remote builder machine, can you check docker images and docker history?

9185dd7c254383:/# docker images
REPOSITORY                TAG                                     IMAGE ID       CREATED         SIZE
registry.fly.io/heyfly2   deployment-01HADKMVSB3DF3BN9ZWWDTHWN7   05d7dca773e4   5 minutes ago   849MB
registry.fly.io/heyfly2   deployment-01HADKRNQ7VVNFRJK75GSVH3JH   e6e6dd37ab79   5 minutes ago   849MB

(05d7dca773e4 is one of the image IDs above)

9185dd7c254383:/# docker history 05d7dca773e4
IMAGE          CREATED         CREATED BY                                      SIZE      COMMENT
05d7dca773e4   5 minutes ago   ENTRYPOINT ["/bin/sh" "-c" "/app/helloflyx"]    0B        buildkit.dockerfile.v0
<missing>      5 minutes ago   RUN /bin/sh -c env > /env.txt # buildkit        144B      buildkit.dockerfile.v0
<missing>      5 minutes ago   RUN /bin/sh -c cd /app && go build # buildkit   71.4MB    buildkit.dockerfile.v0
<missing>      6 minutes ago   COPY go.mod go.sum main.go page.html /app # …   2.25kB    buildkit.dockerfile.v0
<missing>      6 minutes ago   RUN /bin/sh -c mkdir /app # buildkit            0B        buildkit.dockerfile.v0
<missing>      5 months ago    /bin/sh -c #(nop) WORKDIR /go                   0B        
<missing>      5 months ago    /bin/sh -c mkdir -p "$GOPATH/src" "$GOPATH/b…   0B        
<missing>      5 months ago    /bin/sh -c #(nop)  ENV PATH=/go/bin:/usr/loc…   0B        
<missing>      5 months ago    /bin/sh -c #(nop)  ENV GOPATH=/go               0B        
<missing>      5 months ago    /bin/sh -c set -eux;  arch="$(dpkg --print-a…   244MB     
<missing>      5 months ago    /bin/sh -c #(nop)  ENV GOLANG_VERSION=1.20.3    0B        
<missing>      5 months ago    /bin/sh -c #(nop)  ENV PATH=/usr/local/go/bi…   0B        
<missing>      5 months ago    /bin/sh -c set -eux;  apt-get update;  apt-g…   228MB     
<missing>      5 months ago    /bin/sh -c apt-get update && apt-get install…   152MB     
<missing>      5 months ago    /bin/sh -c set -ex;  if ! command -v gpg > /…   19MB      
<missing>      5 months ago    /bin/sh -c set -eux;  apt-get update;  apt-g…   10.7MB    
<missing>      5 months ago    /bin/sh -c #(nop)  CMD ["bash"]                 0B        
<missing>      5 months ago    /bin/sh -c #(nop) ADD file:f2d012660f882f319…   124MB     
9185dd7c254383:/# 

This will give you per-layer breakdown.

Please feel free to use your support email if you feel uncomfortable sharing the breakdown like above.