build server dying

[taguniversalmachine]

fly deploy
==> Verifying app config
→ Verified app config
==> Building image
WARN Remote builder did not start in time. Check remote builder logs with flyctl logs -a fly-builder-sparkling-brook-7020
Error failed to fetch an image or build from source: error connecting to docker: remote builder app unavailable

The logs from the builder:

Z" level=debug msg=“Calling GET /v1.41/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0”
2022-06-09T00:35:36Z app[5e67c5e3] dfw [info]time=“2022-06-09T00:35:36.828610567Z” level=debug msg=“checking docker activity”
2022-06-09T00:35:36Z app[5e67c5e3] dfw [info]time=“2022-06-09T00:35:36.841485020Z” level=debug msg=“Calling GET /v1.41/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0”
2022-06-09T00:35:36Z app[5e67c5e3] dfw [info]time=“2022-06-09T00:35:36.866289715Z” level=info msg=“Deadline reached without docker build”
2022-06-09T00:35:36Z app[5e67c5e3] dfw [info]time=“2022-06-09T00:35:36.867534720Z” level=info msg=“shutting down”
2022-06-09T00:35:36Z app[5e67c5e3] dfw [info]time=“2022-06-09T00:35:36.868891897Z” level=info msg=“gracefully stopped\n”
2022-06-09T00:35:36Z app[5e67c5e3] dfw [info]time=“2022-06-09T00:35:36.869269395Z” level=debug msg=“disk space used: 2.31%”
2022-06-09T00:35:36Z app[5e67c5e3] dfw [info]time=“2022-06-09T00:35:36.870340194Z” level=info msg=“Waiting for dockerd to exit”
2022-06-09T00:35:36Z app[5e67c5e3] dfw [info]time=“2022-06-09T00:35:36.870653702Z” level=info msg=“Processing signal ‘interrupt’”
2022-06-09T00:35:36Z app[5e67c5e3] dfw [info]time=“2022-06-09T00:35:36.871058932Z” level=debug msg=“daemon configured with a 15 seconds minimum shutdown timeout”
2022-06-09T00:35:36Z app[5e67c5e3] dfw [info]time=“2022-06-09T00:35:36.871083468Z” level=debug msg=“start clean shutdown of all containers with a 15 seconds timeout…”
2022-06-09T00:35:36Z app[5e67c5e3] dfw [info]time=“2022-06-09T00:35:36.871401425Z” level=debug msg=“found 0 orphan layers”
2022-06-09T00:35:36Z app[5e67c5e3] dfw [info]time=“2022-06-09T00:35:36.871961766Z” level=debug msg=“Unix socket /var/run/docker/libnetwork/0d0fc2d6850d.sock doesn’t exist. cannot accept client connections”
2022-06-09T00:35:36Z app[5e67c5e3] dfw [info]time=“2022-06-09T00:35:36.872116977Z” level=info msg=“stopping event stream following graceful shutdown” error=“” module=libcontainerd namespace=moby
2022-06-09T00:35:36Z app[5e67c5e3] dfw [info]time=“2022-06-09T00:35:36.872300842Z” level=debug msg=“Cleaning up old mountid : start.”
2022-06-09T00:35:36Z app[5e67c5e3] dfw [info]time=“2022-06-09T00:35:36.874422282Z” level=debug msg=“Cleaning up old mountid : done.”
2022-06-09T00:35:36Z app[5e67c5e3] dfw [info]time=“2022-06-09T00:35:36.875348009Z” level=debug msg=“unmounting daemon root” mountpoint=/data/docker
2022-06-09T00:35:36Z app[5e67c5e3] dfw [info]time=“2022-06-09T00:35:36.875634506Z” level=debug msg=“Clean shutdown succeeded”
2022-06-09T00:35:36Z app[5e67c5e3] dfw [info]time=“2022-06-09T00:35:36.875682346Z” level=info msg=“Daemon shutdown complete”
2022-06-09T00:35:36Z app[5e67c5e3] dfw [info]time=“2022-06-09T00:35:36.875725567Z” level=info msg=“stopping healthcheck following graceful shutdown” module=libcontainerd
2022-06-09T00:35:36Z app[5e67c5e3] dfw [info]time=“2022-06-09T00:35:36.875761414Z” level=info msg=“stopping event stream following graceful shutdown” error=“context canceled” module=libcontainerd namespace=plugins.moby
2022-06-09T00:35:36Z app[5e67c5e3] dfw [info]time=“2022-06-09T00:35:36.876187864Z” level=debug msg=“received signal” signal=terminated
2022-06-09T00:35:36Z app[5e67c5e3] dfw [info]time=“2022-06-09T00:35:36.876306146Z” level=debug msg=“sd notification” error=“” notified=false state=“STOPPING=1”
2022-06-09T00:35:37Z app[5e67c5e3] dfw [info]time=“2022-06-09T00:35:37.857181204Z” level=debug msg=“checking docker activity”
2022-06-09T00:35:37Z app[5e67c5e3] dfw [info]time=“2022-06-09T00:35:37.863015840Z” level=debug msg=“Calling GET /v1.41/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0”
2022-06-09T00:35:37Z app[5e67c5e3] dfw [info]time=“2022-06-09T00:35:37.882988439Z” level=info msg=“dockerd has exited”

According to the dashboard, though, the builder is running.

[SebastianSzturo]

Same here, dashboard also seems to be down now.

[sethinnyc]

Same. 503s from most cli requests. Maybe the graphql server is down? I did successfully get a deploy done after trying a few times but it’s mostly down.

[michael]

We’re having a database issue – follow along here Fly.io Status - Elevated API Errors

[SebastianSzturo]

Thank you @michael

Just out of curiosity: Is there a better place to report infrastructure issues directly or is the community forum the way to go?

[michael]

The forum is good. We have enough direct alerts from our monitoring systems (~300 from this incident ) but feedback from real humans helps us identify problems you’re actually seeing.

[taguniversalmachine]

ok mine looks like it’s running again, whew! This was my first deploy in a couple months and it didn’t bode well but glad it’s running now.

[michael]

Sorry about that! We’ve been hitting all the growing pains this week.

[taguniversalmachine]

I am trying to do a new deploy and so far the build has taken over an hour and I am running out time to babysit it, if I disconnect while it is building will it keep going up on your platform or will it abort?

It did seem to choke on a weird filename but seems like it kept going…

fly deploy
==> Verifying app config
→ Verified app config
==> Building image
Remote builder fly-builder-sparkling-brook-7020 ready
==> Creating build context
→ Creating build context done
==> Building image with Docker
→ docker host: 20.10.12 linux x86_64
[+] Building 2077.2s (0/1)
=> [internal] load remote build context 2077.2s
ERRO[1699] Can’t add file /Users/eflores/src/blink_test/priv/static/images/covers/Boris Brejcha • Hozho • Lana Del Rey • Paul Kalkbrenner • Other Artist • Techn[+] Building 4309.9s (0/1)
=> [internal] load remote build context

UPDATE: 5PM CST and the build has been running for over 2 hours, is this normal? Kind of embarrassing as I told my client the update would be pushed out “momentarily” and we are all still waiting.

fly builds list shows nothing running:

fly builds list
ID STATUS

Dashboard shows build server running but no activity

yet console still shows build running, numbers updating continuously, so not sure what is going on:

Final update 5:43PM CST, did a local build and it deployed perfectly. I guess from an end-user standpoint t would be better if the fly utility did not print fake progress numbers, when II see continuously updating numbers it makes me think the software is alive and making progress, but not in this case.