Waiting for depot builder (Mar 27)

Hi folks - My builds run in GitHub actions and they are hanging on “waiting for depot builder” then timeout. This has been happening for the past ~2 hours consistently. This is the second time this week this broke for a sustained period, I was unable to deploy for most of the day on 3/21. App and builder are (now) both in ORD, which I see had issues yesterday but currently I see nothing on https://status.flyio.net/. Is anyone else having an issue? Do I just need to wait it out? Is there somewhere other than here I should report something like this?

Thanks!

Validating api/fly.toml
✓ Configuration is valid
--> Verified app config
==> Building image
Waiting for depot builder...
==> Building image
Waiting for depot builder...
Error: failed to fetch an image or build from source: error building: context deadline exceeded

P.S. This is my first post here, not sure what the etiquette is on re-using relevant old threads. There are already a couple from earlier in the week. There is also a current thread on deploy failing but I’m not sure if it is related.

I had to switch back again from ORD to DFW as I saw ORD was having machine issues the other day.

1 Like

I see there is an incident now, but my last couple builds actually succeeded :slight_smile: :partying_face:

1 Like

Same issue here since this morning. AMS region, all deploys failing consistently for 6+ hours now. Tried everything:

  • –remote-only (default) — hangs on “Waiting for depot builder”, then times out
  • –depot=false — fails with “insufficient memory available to fulfill request” for the remote builder VM
  • –recreate-builder — still hangs on depot builder
  • –local-only — not an option without Docker Desktop on this machine

Three separate apps (API, staging, cronjob), all the same result. Last successful deploy was ~18 hours ago, no changes
to infra since.

Also unable to scale existing machines — getting governor policy blocked start: hard gate failed:
cpu_util_exceeds_target when trying to update memory via flyctl machine update.

Nothing on status.flyio.net. Would be great if these builder capacity issues were reflected there.

2 Likes

Did you try moving the builder? That’s the official workaround lately…

https://community.fly.io/t/fly-deploy-has-been-failing-for-over-12-hours-when-pushing-to-registry-fly-io/27461/4

(--recreate-builder on its own just gives you a new builder in the same region.)