Thanks for following up. I appreciate your attention, but I believe there’s still some misunderstanding about the nature of the problem we’ve experienced with Depot builders. Let me clarify a few points:
-
Not just resource contention: While resource contention may exacerbate the issue, we experienced a consistent 4X slowdown with Depot builders even when running single builds. This indicates a fundamental performance difference between Depot and standard Fly builders. I’ve detailed this in my previous post. And resource contention is really a consequence of the slowdown as we’re far more likely to have parallel builds when things are 4X slower.
-
Systemic issue and framing: The performance degradation isn’t limited to specific steps in our CI process, as suggested in this earlier response. As I demonstrated, all steps of our build process are affected, with the overall build time increasing from about 9 minutes to over 34 minutes. This consistent slowdown across all steps indicates a systemic issue on Depot’s end, not a problem with our specific CI setup. Our pipeline has been working fine for years without significant changes. The sudden onset of a massive slowdown when switching to Depot builders is, by definition, an issue with the new system. The responses I’ve received seem to be premised on the idea that there’s an issue on our end, which is frustrating given the clear evidence that this is a Depot-specific problem. Even if it affects a small percentage of customers, when you’re the customer facing this, it’s quite disruptive.
-
Default behavior change: While
--depot-scope appmight help with resource contention, it doesn’t address the core performance issue. Moreover, I would suggest that the default behavior for Depot should match the previous non-Depot behavior to avoid disrupting existing workflows. I’ve also mentioned this concern in my forum post. -
Concern about future deprecation: The last official public statement indicated that Fly would “likely stop supporting the standard Fly remote builder.” While I understand this may not be immediate, it’s concerning for those of us who can’t currently use Depot due to these performance issues. As I mentioned, I was told via email support that you “may” continue to support the standard builder, but that’s not very reassuring.
I appreciate that you’re willing to investigate this further. I’d be willing to devote some time to helping, but with the understanding that you’re asking us to spend our time assisting you in identifying and resolving an issue with your new system, not that you’re helping to debug an issue on our end, as things are currently framed.
BTW regarding this:
I suspect there was some confusion/ambiguity stemming from the parallel responses in the community forum thread.
Right, I posted publicly after not getting any follow-up on the support thread. And then I didn’t receive a response to my forum post. Posting here is what finally got a response.
We can keep our future communication regarding this in the support ticket.