Cannot deploy to the syd region

Questions / Help Build debugging
1

I cannot seem to deploy, I tried a few different strategies, either hangs waiting for deploy, or if I build locally, hangs trying to push to the registry. Eventually I noticed this message:

WARN Failed to start remote builder heartbeat: failed to create volume: no capacity available in syd

.. and then after destroying the build instance and trying again:

WARN Failed to start remote builder heartbeat: failed to create volume: failed_precondition: machine capacity hold failed: insufficient memory available to fulfill request

EDIT: I should note that this doesn’t seem to be a problem local to my machine, I normally deploy via Github actions and only tried locally because I saw that they were failing.

Github also gets stuck on:

Waiting for depot builder…

3

Whatever was going on, this issue seems to have resolved itself - deployments are working again from Github actions.

5

This issue has surfaced again today.

6

Hi… If you’re seeing this a lot, you can try moving the builder to a less congestion-prone region. There aren’t great docs for doing so at the moment, but the following forum thread has the main idea:

https://community.fly.io/t/fly-deploy-has-been-failing-for-over-12-hours-when-pushing-to-registry-fly-io/27461/4

There’s also an env var you can set instead:

https://community.fly.io/t/region-cannot-host-your-machine-fra/23071/31

(These days, you might need to add --recreate-builder and/or --depot=false, on that command line.)

The underlying problem, going by the warning messages you posted earlier, is a transient capacity crunch in that specific region, so it would be best to choose one of the large ones as the alternative…

7

Brilliant, thanks! I didn’t know you could move just the builder to a new region, I’ll give this a go

8

I hadn’t deployed for a couple of months and then this today. Was a bit confused. Glad this thread is here but a status incident wouldn’t go astray…

Unfortunately my apps need to run in Australia for regulatory reasons.

9

Same issue here - and moving regions isn’t an option.

Urgent solution needed!

10

We have also been experiencing this consistently over the past 24 hours. I have seen it impact our deployments intermittently over the past several months too.

11

Does --depot=false work for you?

I just got a build to go through in SYD with that variant…

(This flag gives you one of the freebie legacy builders.)

12

I’ve tried all these options, and nothing works. I’ve tried build regions in Asia, NA and Europe. I’ve built locally, I’ve built without depot.
Yesterday it just eventually randomly worked, so I guess I’ll just have to try until it does.

13

No didn’t work for me.

Hopefully someone from the staff can give an update - based on their status page they aren’t even aware that there is an issue:

14

@flyio-support Tagging you here just in case support is unaware.

FWIW - I’ve just tried changing build region and still stuck indefinitely on ‘Waiting for depot builder…’

15

Hey all, thanks for tagging us. It looks like we need a bit more capacity in our syd region and I’ve bumped our infra team to have a closer look. We’re monitoring how many placement failures are happening in the region.

I’ll keep you-all updated.

16

Managed to get it working as a one off with:

  1. On my local rather than GH actions (doubt this had any effect)
  2. Changing build region
  3. depot = false
17

Yup, I couldn’t get it to work earlier but this just did the trick:

FLY_REMOTE_BUILDER_REGION=iad fly deploy --depot=false --verbose

18

We added some more capacity in syd about 11 hours ago so the root cause of deployment/building issues in syd should be solved now! If you’re still running into issues there may be something else at play. Feel free to reach out to us if there’s anything else.

19

Thanks for the heads up. For me, this would have been quite bad if I was trying to get a critical fix out or something. Hopefully Fly will not under provision the Sydney region going forward.