Endless depot builder waiting

✓ Configuration is valid
→ Verified app config
==> Building image
Waiting for depot builder… :backhand_index_pointing_left:t2: stops here

1 Like

Same here. Though it finally finished after about 5 mins..

Yeah. Did Fly send out a half-baked release? The last few days have been horrendous – time to fall back, retool and re-release at a later date! I’ve been hanging in “Waiting for depot builder…” for about 40 minutes.

Also – the status page shows no issues. Probably should update that as well.

Oh! And just got this error while I was re-editing:
Error: failed to fetch an image or build from source: error building: context deadline exceeded

Tried again, and back to “Waiting for the depot builder”.

Looks like the earlier incident entry was just reopened:

https://status.flyio.net/incidents/3j0hrt7wgv0t

We are again seeing elevated latency provisioning depot builders on new deploys. Users may see deploys using Depot builders hang or timeout at the “Waiting for Depot Builder” step. We are working on a fix.

We are switching all deploys to use the default Fly builders in the meantime.

If desired users can manually switch back to depot builders using fly deploy --depot=true but may continue to see latency issues at this time.

That incident page says resolved, but that may have been a bit premature… i’m seeing this issue now.

edit: I debated posting this as I knew it would spring to into life after I did, which of course it dutifully has.

1 Like

In case it helps any future similar incidents, during this incident I was seeing builds running very slowly and running out of memory. Suspect it was due to differences with the fly builders?

I couldn’t get a build out at all so had to build and deploy locally as a workaround.

1 Like

Probably… I like the freebie legacy builders, myself, but many people do find them too slow for what they themselves compile…

In the past, it was possible to manually upgrade the (now-)legacy ones, since those are just apps in your organization with their own, malleable Machines. These days, I suspect that might lead to extra charges, though.

Either way, local Docker was a good third option to mention…

This was an issue for me yesterday and is an issue right now, Im trying to recover my Production app and cant release a fix :face_with_raised_eyebrow:

We spend £300+ pm on Fly

I am having the same issue. Deploying is just stopping on “Waiting for Depot Builder”

Sorry to hear that, :adhesive_bandage:… Usually, fly deploy --depot=false works around this, although there are some caveats as noted upthread.

Another technique that sometimes helps is to move the builder over to a different region:

https://community.fly.io/t/region-cannot-host-your-machine-fra/23071/31

(These days, you might also need --recreate-builder.)

1 Like

I am also facing this issue, is this not an official incident yet again?

It appears to be getting worse and worse over the past 90 mins

Hi folks, we’re aware of this and are taking steps to remedy the situation. For now, to avoid deploys getting blocked, I’d recommend going to Sign in to Your Account · Fly, then settings, “app builders”, and selecting a region for your builder that’s neither Dallas, TX nor Ashburn, VA. The issue is limited to those two regions.

  • Daniel
2 Likes

Worth giving this issue a look for folks using this GHA, given the guidance of overiding the builder options Add deploy options by sjdonado · Pull Request #73 · superfly/fly-pr-review-apps · GitHub

merged and release in progress Release 1.6.0 · superfly/fly-pr-review-apps · GitHub

Not sure if it related issue but I’ve been stuck at Waiting for depot builder last few hours

tried ‘reset’ing the builder machine, change its location to sydney, but no luck.

anything else I can do on my end?

Update: after second reset on Singapore it seems to be working now.

1 Like

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.