I’ve tried deleting my builder app since it used to be on my person and i moved my app to an organization…thought that was the issue. Even after deleting the builder and a new one is created, I’m getting the following error.
sdoxsee@Stephens-MacBook-Pro app % fly deploy
==> Verifying app config
--> Verified app config
==> Building image
WARN Failed to start remote builder heartbeat: You hit a Fly API error with request ID: 01GPY7PSVX1Z55AJW2ZHZ5WW3J-yyz
Error failed to fetch an image or build from source: error connecting to docker: You hit a Fly API error with request ID: 01GPY7PTCYRAFHKK5B1B6EX76C-yyz
sdoxsee@Stephens-MacBook-Pro app % fly apps list
NAME OWNER STATUS PLATFORM LATEST DEPLOY
fly-builder-muddy-sunset-9783 app pending machines
app app running nomad 30m45s ago
app-pg app deployed machines 2023-01-14T12:00:18Z
jory
January 16, 2023, 11:25pm
2
I’m having the same problem (but with slightly different request ID 01GPYE2W3B5FWPEBVZHJMA8ERX-sjc)
1 Like
FWIW, I created a brand new account to test out a clean project and builder seems to be working but I need to this
kurt
January 16, 2023, 11:31pm
4
We rolled out account limits that interfered with some builders today. I believe this might be fixed, will you post here again if it continues erroring?
2 Likes
It looks like we’re having this problem since the last few hours. All our deployments are failing with the below error:
==> Pushing image to fly
The push refers to repository [registry.fly.io/lingering-water-2968]
66ac0f44b7cb: Pushed
a98fb432e25b: Pushed
9387bb1e3054: Pushed
451ed49dbb63: Layer already exists
2c9a7d7f4e87: Layer already exists
67075d032ea5: Layer already exists
81d2ed1f40c7: Layer already exists
ff1044e90868: Layer already exists
80511ac8e6e9: Pushing [==================================================>] 199.1MB
38220b892a60: Layer already exists
cbebcedb252e: Layer already exists
5f70bf18a086: Layer already exists
5505ef242c07: Layer already exists
50f407b42369: Layer already exists
a12586ed027f: Layer already exists
Error failed to fetch an image or build from source: error rendering push status stream: EOF
1 Like
s1q
January 17, 2023, 4:53am
7
i have been using fly for a staging site since November and get this error on probably 7/10 fly deploy attempts. Have tried with --remote-only and without the flag and same thing happens.
Thanks for the reply… yeah I ended up just hammering the remote builder multiple times until it finally deployed. This is the first time its happened for me. Hope it doesn’t happen too often!
Haven’t been able to use a remote builder since yesterday, here’s the error in the monitoring logs:
2023-01-17T14:12:00.081 app[1781e33a504e89] ord [info] time="2023-01-17T14:12:00.080604616Z" level=debug msg="checking docker activity"
2023-01-17T14:12:00.081 app[1781e33a504e89] ord [info] time="2023-01-17T14:12:00.080905170Z" level=debug msg="Calling GET /v1.41/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0"
Tried multiple times. Destroyed the remote builder app and retried several times as well.
The machine (fly remote builder) eventually shuts down with this log message:
2023-01-17T14:32:24.394 app[1781e33a504e89] ord [info] time="2023-01-17T14:32:24.393850009Z" level=debug msg="checking docker activity"
2023-01-17T14:32:24.394 app[1781e33a504e89] ord [info] time="2023-01-17T14:32:24.394148298Z" level=debug msg="Calling GET /v1.41/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0"
2023-01-17T14:32:25.395 app[1781e33a504e89] ord [info] time="2023-01-17T14:32:25.395188783Z" level=debug msg="checking docker activity"
2023-01-17T14:32:25.395 app[1781e33a504e89] ord [info] time="2023-01-17T14:32:25.395515816Z" level=debug msg="Calling GET /v1.41/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0"
2023-01-17T14:32:26.397 app[1781e33a504e89] ord [info] time="2023-01-17T14:32:26.396519802Z" level=debug msg="checking docker activity"
2023-01-17T14:32:26.397 app[1781e33a504e89] ord [info] time="2023-01-17T14:32:26.396970046Z" level=debug msg="Calling GET /v1.41/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0"
2023-01-17T14:32:27.398 app[1781e33a504e89] ord [info] time="2023-01-17T14:32:27.398269917Z" level=debug msg="checking docker activity"
2023-01-17T14:32:27.399 app[1781e33a504e89] ord [info] time="2023-01-17T14:32:27.398528291Z" level=debug msg="Calling GET /v1.41/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0"
2023-01-17T14:32:28.400 app[1781e33a504e89] ord [info] time="2023-01-17T14:32:28.400078562Z" level=debug msg="checking docker activity"
2023-01-17T14:32:28.400 app[1781e33a504e89] ord [info] time="2023-01-17T14:32:28.400356453Z" level=debug msg="Calling GET /v1.41/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0"
2023-01-17T14:32:29.402 app[1781e33a504e89] ord [info] time="2023-01-17T14:32:29.401649331Z" level=debug msg="checking docker activity"
2023-01-17T14:32:29.402 app[1781e33a504e89] ord [info] time="2023-01-17T14:32:29.401940086Z" level=debug msg="Calling GET /v1.41/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0"
2023-01-17T14:32:30.404 app[1781e33a504e89] ord [info] time="2023-01-17T14:32:30.403237993Z" level=debug msg="checking docker activity"
2023-01-17T14:32:30.404 app[1781e33a504e89] ord [info] time="2023-01-17T14:32:30.403518960Z" level=debug msg="Calling GET /v1.41/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0"
2023-01-17T14:32:31.405 app[1781e33a504e89] ord [info] time="2023-01-17T14:32:31.404633994Z" level=debug msg="checking docker activity"
2023-01-17T14:32:31.405 app[1781e33a504e89] ord [info] time="2023-01-17T14:32:31.405009608Z" level=debug msg="Calling GET /v1.41/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0"
2023-01-17T14:32:32.406 app[1781e33a504e89] ord [info] time="2023-01-17T14:32:32.406079337Z" level=debug msg="checking docker activity"
2023-01-17T14:32:32.407 app[1781e33a504e89] ord [info] time="2023-01-17T14:32:32.406328855Z" level=debug msg="Calling GET /v1.41/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0"
2023-01-17T14:32:33.408 app[1781e33a504e89] ord [info] time="2023-01-17T14:32:33.407687276Z" level=debug msg="checking docker activity"
2023-01-17T14:32:33.408 app[1781e33a504e89] ord [info] time="2023-01-17T14:32:33.407945941Z" level=debug msg="Calling GET /v1.41/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0"
2023-01-17T14:32:33.837 app[1781e33a504e89] ord [info] time="2023-01-17T14:32:33.837274376Z" level=info msg="Deadline reached without docker build"
2023-01-17T14:32:33.837 app[1781e33a504e89] ord [info] time="2023-01-17T14:32:33.837346982Z" level=info msg="shutting down"
2023-01-17T14:32:33.837 app[1781e33a504e89] ord [info] time="2023-01-17T14:32:33.837510098Z" level=info msg="gracefully stopped\n"
2023-01-17T14:32:33.837 app[1781e33a504e89] ord [info] time="2023-01-17T14:32:33.837542889Z" level=info msg="disk space used: 0.11%"
2023-01-17T14:32:33.837 app[1781e33a504e89] ord [info] time="2023-01-17T14:32:33.837556575Z" level=info msg="Waiting for dockerd to exit"
2023-01-17T14:32:33.837 app[1781e33a504e89] ord [info] time="2023-01-17T14:32:33.837727566Z" level=info msg="Processing signal 'interrupt'"
2023-01-17T14:32:33.838 app[1781e33a504e89] ord [info] time="2023-01-17T14:32:33.838031967Z" level=debug msg="daemon configured with a 15 seconds minimum shutdown timeout"
2023-01-17T14:32:33.838 app[1781e33a504e89] ord [info] time="2023-01-17T14:32:33.838058496Z" level=debug msg="start clean shutdown of all containers with a 15 seconds timeout..."
2023-01-17T14:32:33.838 app[1781e33a504e89] ord [info] time="2023-01-17T14:32:33.838153595Z" level=debug msg="found 0 orphan layers"
2023-01-17T14:32:33.838 app[1781e33a504e89] ord [info] time="2023-01-17T14:32:33.838672558Z" level=debug msg="Unix socket /var/run/docker/libnetwork/f4f8fce416f6.sock doesn't exist. cannot accept client connections"
2023-01-17T14:32:33.838 app[1781e33a504e89] ord [info] time="2023-01-17T14:32:33.838730597Z" level=debug msg="Cleaning up old mountid : start."
2023-01-17T14:32:33.839 app[1781e33a504e89] ord [info] time="2023-01-17T14:32:33.838849851Z" level=info msg="stopping event stream following graceful shutdown" error="<nil>" module=libcontainerd namespace=moby
2023-01-17T14:32:33.839 app[1781e33a504e89] ord [info] time="2023-01-17T14:32:33.838919772Z" level=debug msg="Cleaning up old mountid : done."
2023-01-17T14:32:33.839 app[1781e33a504e89] ord [info] time="2023-01-17T14:32:33.839012766Z" level=debug msg="unmounting daemon root" mountpoint=/data/docker
2023-01-17T14:32:33.839 app[1781e33a504e89] ord [info] time="2023-01-17T14:32:33.839143321Z" level=debug msg="Clean shutdown succeeded"
2023-01-17T14:32:33.839 app[1781e33a504e89] ord [info] time="2023-01-17T14:32:33.839164150Z" level=info msg="Daemon shutdown complete"
2023-01-17T14:32:33.839 app[1781e33a504e89] ord [info] time="2023-01-17T14:32:33.839229773Z" level=info msg="stopping healthcheck following graceful shutdown" module=libcontainerd
2023-01-17T14:32:33.839 app[1781e33a504e89] ord [info] time="2023-01-17T14:32:33.839332075Z" level=info msg="stopping event stream following graceful shutdown" error="context canceled" module=libcontainerd namespace=plugins.moby
2023-01-17T14:32:33.839 app[1781e33a504e89] ord [info] time="2023-01-17T14:32:33.839392027Z" level=debug msg="received signal" signal=terminated
2023-01-17T14:32:33.839 app[1781e33a504e89] ord [info] time="2023-01-17T14:32:33.839524897Z" level=debug msg="sd notification" error="<nil>" notified=false state="STOPPING=1"
2023-01-17T14:32:34.410 app[1781e33a504e89] ord [info] time="2023-01-17T14:32:34.410013401Z" level=debug msg="checking docker activity"
2023-01-17T14:32:34.410 app[1781e33a504e89] ord [info] time="2023-01-17T14:32:34.410276314Z" level=debug msg="Calling GET /v1.41/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0"
2023-01-17T14:32:34.840 app[1781e33a504e89] ord [info] time="2023-01-17T14:32:34.840110999Z" level=warning msg="grpc: addrConn.createTransport failed to connect to {unix:///var/run/docker/containerd/containerd.sock <nil> 0 <nil>}. Err :connection error: desc = \"transport: Error while dialing dial unix:///var/run/docker/containerd/containerd.sock: timeout\". Reconnecting..." module=grpc
2023-01-17T14:32:34.842 app[1781e33a504e89] ord [info] time="2023-01-17T14:32:34.841996415Z" level=info msg="dockerd has exited"
2023-01-17T14:32:37.419 runner[1781e33a504e89] ord [info] machine exited with exit code 0, not restarting
This doesn’t seem to be the problem that I originally posted. Might be worth its own thread as my issue is solved.
Cheers,
1 Like
