Remote builder stopped working, doesn't restart

My remote builder crashed/stopped during a build with this error message:
Error failed to fetch an image or build from source: error building: failed to copy files: failed to open target /data/docker/overlay2/f5i1okz9uepibhypj8585fji9/merged/src/ui/node_modules/es-abstract/2018/SameValueZero.js: open /data/docker/overlay2/f5i1okz9uepibhypj8585fji9/merged/src/ui/node_modules/es-abstract/2018/SameValueZero.js: no space left on device

And hasn’t come back up since when trying to build with fly deploy --remote-only

flyctl logs -a fly-builder-spring-cloud-2626 shows:

2022-08-07T04:50:04Z app[4d89071f666087] sea [info]failed to start daemon: error initializing graphdriver: driver not supported
2022-08-07T04:50:04Z app[4d89071f666087] sea [info]Name:   default
2022-08-07T04:50:04Z app[4d89071f666087] sea [info]Name:     default
2022-08-07T04:50:04Z app[4d89071f666087] sea [info]Error:    Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
2022-08-07T04:50:04Z app[4d89071f666087] sea [info]time="2022-08-07T04:50:04.017886142Z" level=error msg="error waiting on docker: exit status 1"
2022-08-07T04:50:04Z app[4d89071f666087] sea [info]time="2022-08-07T04:50:04.017921452Z" level=info msg="dockerd has exited"
2022-08-07T04:50:04Z app[4d89071f666087] sea [info]time="2022-08-07T04:50:04.017937872Z" level=fatal msg="dockerd exited before we could ascertain its healthyness"
2022-08-07T04:50:08Z runner[4d89071f666087] sea [info]machine exited with exit code 0, not restarting

Is there a way to cleanly reset & bounce or recreate the builder?

Yes you can destroy the builder. And then a new one will be created on the next deploy. That shouldn’t be regularly needed but there are cases like this where it is.

If you know the name of the builder, run fly destroy builder-name-goes-here

If you don’t know its name, see if it’s listed under fly apps list. The builder app should stand out from your own apps.

2 Likes

Ha, that worked, thanks @greg !

1 Like