Intermittent build failure: "remote builder app unavailable"

Questions / Help Build debugging
1

When running flyctl deploy -c fly.staging.toml (on an app that usually deploys fine), I received this error part way through:

==> Verifying app config
--> Verified app config
==> Building image
Remote builder fly-builder-cold-flower-3697 ready
==> Creating build context
--> Creating build context done
==> Building image with Docker
--> docker host: 20.10.12 linux x86_64
[+] Building 61.7s (0/1)                                                                                                                                                   
 => [internal] load remote build context                                                                                                                             61.7s
ERRO[0067] Can't add file /Users/FOO/node_modules/@popperjs/core/dist/umd/popper-lite.min.js.map to tar: io: read/write on closed pipe 
ERRO[0067] Can't close tar writer: io: read/write on closed pipe 
Error failed to fetch an image or build from source: error building: unexpected EOF

Now, on subsequent deploys, I receive this:

==> Verifying app config
--> Verified app config
==> Building image
WARN Remote builder did not start in time. Check remote builder logs with `flyctl logs -a fly-builder-cold-flower-3697`
Error failed to fetch an image or build from source: error connecting to docker: remote builder app unavailable

Upon running flyctl logs -a fly-builder-cold-flower-3697, I get:

2022-08-08T02:20:16Z runner[73d8dd69f12189] dfw [info]machine exited with exit code 0, not restarting
2022-08-08T03:39:01Z runner[73d8dd69f12189] dfw [info]machine exited with exit code 0, not restarting
2022-08-08T04:57:12Z runner[73d8dd69f12189] dfw [info]machine exited with exit code 0, not restarting
2022-08-09T02:07:30Z runner[73d8dd69f12189] dfw [info]machine exited with exit code 0, not restarting

After a few minutes, it all starts working again. This happens when deploying from both CLI and Github actions. Is there something I need to change on my end to make this more stable? Happy to run my own workers if that’s the best way.

2

(Related Deployment issue -- "remote builder app unavailable" - #20 by srbaker)

3

Generally this means that the builder ran out of space and cannot start back up. The quickest fix is to destroy the builder app with fly apps destroy <appname>. On the next deploy, you’ll get a fresh builder.

This happens because builds accumulate Docker layer cache over time. Cached layers are valuable because they keeps deploys fast. The builder does its best to ‘prune’ unused layers, but it can’t always know which ones won’t be used again.

4 Likes
4

Same issue happened to us today too. But destroying the builder app fixed it.

5

Thanks @jsierles. I assume it started working again because the prune function was able to run between my attempts?

Do you reckon that if I increase the size of the volume, it’s more likely to consistently succeed in future?