Remote builder stopped working, doesn't restart

1

My remote builder crashed/stopped during a build with this error message:
Error failed to fetch an image or build from source: error building: failed to copy files: failed to open target /data/docker/overlay2/f5i1okz9uepibhypj8585fji9/merged/src/ui/node_modules/es-abstract/2018/SameValueZero.js: open /data/docker/overlay2/f5i1okz9uepibhypj8585fji9/merged/src/ui/node_modules/es-abstract/2018/SameValueZero.js: no space left on device

And hasn’t come back up since when trying to build with fly deploy --remote-only

flyctl logs -a fly-builder-spring-cloud-2626 shows:

2022-08-07T04:50:04Z app[4d89071f666087] sea [info]failed to start daemon: error initializing graphdriver: driver not supported
2022-08-07T04:50:04Z app[4d89071f666087] sea [info]Name:   default
2022-08-07T04:50:04Z app[4d89071f666087] sea [info]Name:     default
2022-08-07T04:50:04Z app[4d89071f666087] sea [info]Error:    Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
2022-08-07T04:50:04Z app[4d89071f666087] sea [info]time="2022-08-07T04:50:04.017886142Z" level=error msg="error waiting on docker: exit status 1"
2022-08-07T04:50:04Z app[4d89071f666087] sea [info]time="2022-08-07T04:50:04.017921452Z" level=info msg="dockerd has exited"
2022-08-07T04:50:04Z app[4d89071f666087] sea [info]time="2022-08-07T04:50:04.017937872Z" level=fatal msg="dockerd exited before we could ascertain its healthyness"
2022-08-07T04:50:08Z runner[4d89071f666087] sea [info]machine exited with exit code 0, not restarting

Is there a way to cleanly reset & bounce or recreate the builder?

2

Yes you can destroy the builder. And then a new one will be created on the next deploy. That shouldn’t be regularly needed but there are cases like this where it is.

If you know the name of the builder, run fly destroy builder-name-goes-here

If you don’t know its name, see if it’s listed under fly apps list. The builder app should stand out from your own apps.

2 Likes
3

Ha, that worked, thanks @greg !

1 Like
4

This initially worked for me, but while I ran “fly deploy --remote-only,” the builder suddenly failed again with a log like this:

2023-07-08T18:41:58.327 app[e82dd02a765038] dfw [info] time="2023-07-08T18:41:58.326986668Z" level=info msg="Deadline reached without docker build"
2023-07-08T18:41:58.327 app[e82dd02a765038] dfw [info] time="2023-07-08T18:41:58.327053197Z" level=info msg="shutting down"
2023-07-08T18:41:58.327 app[e82dd02a765038] dfw [info] time="2023-07-08T18:41:58.327473688Z" level=info msg="gracefully stopped\n"
2023-07-08T18:41:58.327 app[e82dd02a765038] dfw [info] time="2023-07-08T18:41:58.327528608Z" level=info msg="disk space used: 4.94%"
2023-07-08T18:41:58.327 app[e82dd02a765038] dfw [info] time="2023-07-08T18:41:58.327570277Z" level=info msg="Waiting for dockerd to exit"

What could be the reason for this, and is there any way to prevent the builder machine from shutting down?

5

@clarosilverina I don’t know, I’m afraid.

The builder apps are run by Fly. I’d assume they are only running as long as they need to be, as it would be a waste of resources to keep them running 24/7. It would make sense for them to be started when needed (on a deploy) and pause when not. As indicated by that log.

Maybe try a local build and see if that works? Since if not, that would explain why remote builds aren’t working either. If a local build does work … not sure.

7

Thank you for responding. I just tried --local-only and encountered an error like this:

==> Building image WARN Error connecting to local docker daemon:error during connect: this error may indicate that the docker daemon is not running: Get "http://%2F%2F.%2Fpipe%2Fdocker_engine/_ping": open //./pipe/docker_engine: The system cannot find the file specified. Error: failed to fetch an image or build from source: docker is unavailable to build the deployment image.

8

@clarosilverina Strange :thinking: I guess the obvious cause would be Docker is not actually running? Check you have it installed (Download Docker Desktop | Docker) and running before running that command. Else, yep, it won’t be able to do the build.

If Docker is running, I don’t know. The Fly CLI should be able to see and use it. That would point to a bigger issue with the builds. Perhaps why remote ones aren’t working for you.