Depot remote builders becoming the default

Fresh Produce
21

Thanks, can you try once more?

22

We were pretty caught off guard by depot becoming default and our builds suddenly started failing from out of memory issues during our next js builds. Setting depot to false gets our build working again. Do arguments like build.vm still get passed to depot?

[build]
  [build.args]
    NODE_VERSION = "21"
  [build.environment]
    NODE_OPTIONS = "--max-old-space-size=16384"
    DOCKER_BUILDKIT = "1" 
  [build.vm]
    cpu_kind = "performance"
    cpus = 2
    memory = "16gb"
1 Like
23

I was just able to get the process to start without --depot=false, but it did take a while waiting for Depot before continuing.

24

That vm setting isn’t supported, but it looks like you may have manually scaled up your builder instance memory to 8GB. That’s never officially been supported. But, we do plan to allow using larger builder sizes with the new build system.

For now, you can continue using your Fly builder until we expose this option.

25

Same experience for me. The deployment worked but it hung out at “Waiting for Depot builder” for a few minutes and then, subjectively, the overall process felt about 1.5-2x slower than normal. But it did deploy.

1 Like
26

Are you seeing wait times like this on every deploy? It looks like you only have one build, so the first one would have taken a while to start up with a fresh machine and cache volume. Subsequent deploys should be fast.

27

The switch to Depot seems to have broken our build.

Using fly deploy --depot=false works as expected but using depot we see an error…


#21 [builder 13/20] RUN mix assets.deploy
#21 1.271 
#21 1.271 10:07:33.728 [debug] Downloading esbuild from https://registry.npmjs.org/@esbuild/linux-x64/0.23.0
#21 2.141 
#21 2.141   ../priv/static/assets/js/app.js       179.8kb
#21 2.141   ../priv/static/assets/css/app.css       7.3kb
#21 2.141   ../priv/static/assets/css/scenes.css    891b 
#21 2.141 
#21 2.141 ⚡ Done in 73ms
#21 2.220 Compiling 240 files (.ex)
#21 ERROR: process "/bin/sh -c mix assets.deploy" did not complete successfully: exit code: 137

This is a Phoenix project and it’s failing on (or just after) the mix assets.deploy step.

I assume is OOM due to the exit code: 137 so I assume the Depot builders maybe have less ram available?

How can we fix this and increase the Depot resources?

28

@andykent @sg391 Hey, my mistake - you both have larger builders because you’re on one of our higher paid plans. We’ll make sure you get your correct builder size.

1 Like
29

I’m in fra and just seemingly stuck on " Waiting for depot builder…". and then it fails:

==> Building image
Waiting for depot builder...

2024/09/18 12:36:22 error releasing builder: unavailable: 503 Service Unavailable
Error: failed to fetch an image or build from source: error building: unavailable: 503 Service Unavailable
error reporting health: unavailable: 503 Service Unavailable
1 Like
30

ARN also failing. “Waiting for Depot builder…” for many minutes before crashing:
error reporting health: unauthenticated: Invalid tokenError: failed to fetch an image or build from source: error building: unavailable: Unable to acquire machine, please retry

Luckily I can still do --depot=false to deploy.

31

Hey all! Your builds are about to become more performant and reliable, thanks to our partnership with Depot .

That hasn’t been our experience at all. Our builds are waiting for Depot builder for minutes. :frowning:

1 Like
32

Ah, there’s an ongoing issue. It sounds like you can opt out, at least for now:

1 Like
33

Hey folks, sorry for the trouble here. We’ve been slowly rolling out this change, but the final step to roll it out to everyone pushed some limits of our API.

For now, as a precaution, everyone has been moved back to our old builders, and status of the Depot builders can be tracked here: Fly.io Status - Depot builders failing

We’ll post again here when we start the rollout again.

2 Likes
34

We’ve started rolling out Depot builders again today. Let us know if you have any issues!

2 Likes
35

I just ran into a deploy issue where I was waiting for a Depot builder for several minutes. I had to cancel the deploy and re-run it before it started working again. Happy to provide details privately if it’ll help debug!

36

ditto, restarting didn’t help… had to deploy with --depot=false

37

Thanks for the reports. We’ve rolled Depot builders back once again, while we investigate. The problems are related to using our machines API at scale.

Before we bring Depot builders back, we’ll introduce a failsafe to fall back to Fly builders after a reasonable timeout or when errors occur.

3 Likes
38

I initially switched to BuildJet since Fly builders can’t seem to handle building 3 apps at once and would fail very consistently. Is there a way to force a different builder per app? I just tried again after about 6 months and my builds failed again. Seems like I run out of memory…looks like it is possible to do with Depot builders with the scope flag, can I do something similar with Fly builders in the mean time?

1 Like
39

It’s not possible with Fly builders, but Depot is back online now, so you can try it with --depot=true and the scope flag. Another option we’ll soon expose is the ability to use a builder with a larger memory footprint.

1 Like
40

Alright, thank you Joshua