Error waiting for remote builder app: context deadline exceeded

This morning when I try to do fly deploy I am getting the following message:

Error error connecting to docker: Error waiting for remote builder app: context deadline exceeded

1 Like

Hi there. We’re aware. I’ve just created a statuspage incident: Fly.io Status - Remote builders are not detected as "up"

Should now be fixed.

I’ve been seeing this, too. Trying to get a “hello world” Go app up and running and not having any luck. Pretty bummed - this platform seems really cool and would love to try it out

Hey. This kicked off a whole ops thing internally, but we should have posted here right away to make sure you knew we’d seen it and were looking. Remote builders aren’t down (our watchdog builds are succeeding) but we’re digging in to see what could have gone wrong with yours. We’ll keep posting status.

1 Like

Thanks so much! Best of luck and sorry to disrupt the day with an incident :wink: :hugs:

1 Like

Just some status: it looks like your build hit a corner case for us; remote builds create temporary fly-apps running in their own VMs, and yours was scheduled into a host in the DFW region that didn’t have CPUs. We should have caught that; there’s a bunch of stuff happening right now in response.

1 Like

ah! I noticed that - was new to the platform so wasn’t sure if it was some kind of side effect or something. Definitely saw an add’l app spun up every time

builds seem to be working! woohoo! one quick surprise I found was that I didn’t have the gcloud CLI util installed (I did at one point but it’s been a while since I’ve used GCP) so fly deploy failed on gcr.io access - could be something unique to me, idk. Thought I would at least mention though and say thanks for fixing things! :handshake:

(*I worked around this quickly but just a heads up)

Absolutely no problem. If you’re seeing builder failures, other people are too, so you did us a favor here. I’ll kick you some credits for putting up with this. We’re working on monitoring hooks for these kinds of weird builder failures.

A note — you shouldn’t need to do this and it’s definitely worth telling us when you have to — that if your build fails like this, it’s perfectly cromulent to just kill the build and try again. The thing you ran into here was a transient congestion problem on our side.

1 Like

We don’t actually do anything with gcr.io, is it possible it was trying to pull an image from there for you? The deploy process will pull what it needs to build, then push the result to us. If your app is using a base image off gcr it may not be able to auth to pull it down.

1 Like

Oh thanks! That’s very kind of you. Always happy to help. Seems like a very cool platform and I’ve been looking around for one for a side project that’s getting close to production. And good to know for the future!

re: the gcr thing — has to have been a local thing, probably related to how they do auth for pulling from their container reg. nowadays; definitely nothing on the Fly side. Just happened to be in the paketo buildpacks image gcr.io/paketo-buildpacks/go :+1:

1 Like

Oh the paketo buildpack might be us. I wonder what was up with that.

1 Like

Those are publicly accessible without auth. Does it always fail or just sometimes?

1 Like

Weird! it failed ~consistently (tried 2-3 times) before I installed a gcloud component. Could just be me though