This morning when I try to do fly deploy
I am getting the following message:
Error error connecting to docker: Error waiting for remote builder app: context deadline exceeded
This morning when I try to do fly deploy
I am getting the following message:
Error error connecting to docker: Error waiting for remote builder app: context deadline exceeded
Hi there. We’re aware. I’ve just created a statuspage incident: Fly.io Status - Remote builders are not detected as "up"
Should now be fixed.
I’ve been seeing this, too. Trying to get a “hello world” Go app up and running and not having any luck. Pretty bummed - this platform seems really cool and would love to try it out
Hey. This kicked off a whole ops thing internally, but we should have posted here right away to make sure you knew we’d seen it and were looking. Remote builders aren’t down (our watchdog builds are succeeding) but we’re digging in to see what could have gone wrong with yours. We’ll keep posting status.
Thanks so much! Best of luck and sorry to disrupt the day with an incident
Just some status: it looks like your build hit a corner case for us; remote builds create temporary fly-apps running in their own VMs, and yours was scheduled into a host in the DFW region that didn’t have CPUs. We should have caught that; there’s a bunch of stuff happening right now in response.
ah! I noticed that - was new to the platform so wasn’t sure if it was some kind of side effect or something. Definitely saw an add’l app spun up every time
builds seem to be working! woohoo! one quick surprise I found was that I didn’t have the gcloud
CLI util installed (I did at one point but it’s been a while since I’ve used GCP) so fly deploy
failed on gcr.io
access - could be something unique to me, idk. Thought I would at least mention though and say thanks for fixing things!
(*I worked around this quickly but just a heads up)
Absolutely no problem. If you’re seeing builder failures, other people are too, so you did us a favor here. I’ll kick you some credits for putting up with this. We’re working on monitoring hooks for these kinds of weird builder failures.
A note — you shouldn’t need to do this and it’s definitely worth telling us when you have to — that if your build fails like this, it’s perfectly cromulent to just kill the build and try again. The thing you ran into here was a transient congestion problem on our side.
We don’t actually do anything with gcr.io
, is it possible it was trying to pull an image from there for you? The deploy
process will pull what it needs to build, then push the result to us. If your app is using a base image off gcr it may not be able to auth to pull it down.
Oh thanks! That’s very kind of you. Always happy to help. Seems like a very cool platform and I’ve been looking around for one for a side project that’s getting close to production. And good to know for the future!
re: the gcr thing — has to have been a local thing, probably related to how they do auth for pulling from their container reg. nowadays; definitely nothing on the Fly side. Just happened to be in the paketo buildpacks image gcr.io/paketo-buildpacks/go
Oh the paketo buildpack might be us. I wonder what was up with that.
Those are publicly accessible without auth. Does it always fail or just sometimes?
Weird! it failed ~consistently (tried 2-3 times) before I installed a gcloud
component. Could just be me though