Deployment failing: Error: failed to fetch an image or build from source: error connecting to docker: remote builder app unavailable

Trying to deploy bug fix and receiving this error since this morning.

Tried destroying the fly-builder instance and redeploying with a new builder, same error.

Region nrt, fly v0.1.103.

No outage reported on https://status.flyio.net/.


LOGLEVEL=debug output shows following message repeatedly:

Waiting for remote builder fly-builder-purple-voice-331… :earth_asia:DEBUG Remote builder unavailable, retrying in 50ms (err: Get “http://[fdaa:0:9ced:a7b:fc:28ad:69d2:2]:2375/_ping”: context deadline exceeded)

Docker Hub is having an ongoing outage: Docker system status real-time view

The image the remote builder uses is hosted there. fly deploy --local-only will use a local docker environment on your system to build and push the image. That might work and work around the remote builder issue for now.

I’ll update our status page, too.

We’ll post further updates on our status page: Fly.io Status - Docker Hub Registry Timing Out: Fly Platform Impact

I was able to deploy, presumably after the update to use the caching mirror. Thank you.

:wave: Hey everyone, Docker has marked the issue as resolved on their end, so we’re following suit. Please let us know if you’re still having related issues.

I am absolutely not sure if this is related, but I get this in my dashboard log monitoring:

2023-09-28T19:39:37.753 runner[4d891d69b00218] ams [info] Pulling container image registry.fly.io/cony:deployment-01HBEMQXQZJ20GY5FJ8F5X7ADY

2023-09-28T19:39:44.708 runner[4d891d69b00218] ams [info] Successfully prepared image registry.fly.io/cony:deployment-01HBEMQXQZJ20GY5FJ8F5X7ADY (6.955115967s)

2023-09-28T19:39:45.612 runner[4d891d69b00218] ams [info] Configuring firecracker

could not find a good candidate within 90 attempts at load balancing. last error: no known healthy instances found for route tcp/443. (hint: is your app shutdown? is there an ongoing deployment with a volume or using the ‘immediate’ strategy? has your app’s instances all reached their hard limit?)

And the Terminal Error:

Powershell: → Building image done
==> Pushing image to fly
The push refers to repository [registry.fly.io/cony]
fb71bd9ec6f3: Pushed
75166d7d5b4e: Pushed
4412ca143975: Pushed
34b0c31f5f82: Pushed
a9aa4cfc7b2d: Pushed
0d5e1f8c333d: Pushed
25156ca953a7: Pushed
cc2447e1835a: Pushed
deployment-01HBFY4T5GZXBS9SS8BTG8617E: digest: sha256:e044d8feefffea99e77157657f4eadeefb65c5a8d4eae32be35b4658a15e5ee6 size: 1992
→ Pushing image done
image: registry.fly.io/cony:deployment-01HBFY4T5GZXBS9SS8BTG8617E
image size: 662 MB

Watch your deployment at Sign In · Fly

Updating existing machines in ‘cony’ with rolling strategy
[1/1] Updating 4d891d69b00218 [app]
Error: failed to update VM 4d891d69b00218: request returned non-2xx status, 504
PS C:\Coding\Cony>

The app got suddenly suspended a few days ago and now I am just trying to restart it by redeploying it…

Thanks for any help!

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.