WARN Failed to start remote builder heartbeat: Couldn’t allocate volume, not enough compute capacity left in mad
Error: failed to fetch an image or build from source: Couldn’t allocate volume, not enough compute capacity left in mad
Hi @KshMlg that’s unfortunate.
Right now I can see you already have a new volume started for your builder, can you confirm if it’s working now?
If not, you can try creating your remote builder in another region:
- Make sure you have a recent flyctl
- Find your builder machine ID via:
fly machine list -a <your-fly-builder-app-name>- Clone the machine in a new region:
fly machine clone <machine-id> --region <NEARBY_REGION> -a <your-fly-builder-app-name>- Destroy your previous machine (
fly machine remove <old-machine-id>)- Destroy your old volume (
fly volume delete <old-volume-id>)
Hi Lubien,
Thanks! I waited for a while and now its working again, i didnt change nothing. 
Now updating flyctl just in case 
1 Like
Bad news, again its coming back.This is getting weird dont know whats going on with “mad” instances but a few days where everything looks laggish
Can you try moving your volume out of that region using the steps described above?
Yes, next time directly follow your instructions. Thanks
We’re having the same problem with mad. Trying to move to Paris (cdg) but getting the same error:
Error: Couldn't allocate volume, not enough disk space left in gdc.
Any recommendations?
Now it’s working again with no changes 
.
We are still seeing this.
WARN Failed to start remote builder heartbeat: Couldn't allocate volume, not enough compute capacity left in mad
Error: failed to fetch an image or build from source: error connecting to docker: Couldn't allocate volume, not enough compute capacity left in mad
Running with the latest flyctl version (v0.1.65)
Having the same problem 
Is there a way to check zones availability before moving? I don’t want to be rebuilding to find the same situation elsewhere!
Hi @stakindotcom can you try moving your builder to a different region?
Another possible workaround is to use fly deploy --local-only if you have docker locally to build your app as we investigate this.
1 Like
Building locally is not the best option for those of us using ARM (M1/M2) machines.
@lubien could you please confirm if there is any outage going on in the mad region? Thanks!
Hi @javiercr
One of our mad hosts is at capacity. The last 3 people that posted here indeed have/had their builders at that same host. My suggestion would be to pick a region close to you to clone your remote builder for now.
Thank you for the quick reply!
As a suggestion, it would be great if the status page would reflect these issues, as they impact the ability to deploy for users on those regions.
3 Likes
I managed to deploy @lubien. Can we expect this being stabilized? If this is gonna be fixed I prefer not risking messing up by destroying my machine
Thanks for the suggestion @javiercr!
I’m glad it worked @pablogiralt. As a note: destroying remote builders is harmless, we will always provide a new one when needed. The workarounds I’ve sent above are only needed if you want to manually migrate a builder from one region to another which you all honestly shouldn’t have to do at all!
If you have spare time, would you all be able to share what other things you tried before to make it work so we can gather more insights on this issue?
Actually the workaround you suggested didn’t work for me. I can’t list the builder machines using
fly machine list -a <your-fly-builder-app-name>
I always get
Error: machines could not be retrieved
The problem is that even if I go to the dashboard and manually delete the builder machine, I still can’t deploy because I get the same error that started this thread:
WARN Failed to start remote builder heartbeat: Couldn't allocate volume, not enough compute capacity left in mad
Error: failed to fetch an image or build from source: error connecting to docker: Couldn't allocate volume, not enough compute capacity left in mad
Is there a way to start a new builder in a new region while deploying to the mad region? (i.e. keeping the app and database machines in mad).
@javiercr would you be able to use LOG_LEVEL=debug fly machine list -a <builder app> so we get more info on that error, this is definitely not great. What you’d look for is something like “Fly Request ID abc-1234asd”, we can use that on our error tracker. If not, look for other possible issues in there and if possible paste in here or DM me.
There is a way of rearranging a remote builder on another region but it’s very manual and requires listing the builder machines so I’d recommend you to use our dashboard for now.
tl;dr: we are gonna clone the builder volume in a new zone in the same region then clone the machine to attach to it.
- Look for your current volume name
$ fly vol list -a fly-builder-name
ID STATE NAME SIZE REGION ZONE ENCRYPTED ATTACHED VM CREATED AT
vol_n0l9vl22gymr635d created machine_data 50GB gru 692f false 17811666c9e308 6 days ago
My volume ID is vol_n0l9vl22gymr635d and it’s on zone 692f.
- Clone to a new zone. The command above uses
gruregion because that’s where my builder lives, you’d change that tomad.
$ fly vol create machine_data -s 50 -r gru --no-encryption -a fly-builder-name
ID: vol_0o6d423kno7r87gy
Name: machine_data
App: fly-builder-name
Region: gru
Zone: c6ed
Size GB: 50
Encrypted: false
Created at: 28 Jul 23 11:05 UTC
As you can see my new volume vol_0o6d423kno7r87gy (we will use this ID later) uses zone c6ed. By default volumes on the same app and same region will be on new zones.
- Find the machine ID you’re going to clone
$ fly m list -a fly-builder-name
1 machines have been retrieved from app fly-builder-name.
View them in the UI here
fly-builder-name
ID NAME STATE REGION IMAGE IP ADDRESS VOLUME CREATED LAST UPDATED APP PLATFORM PROCESS GROUP SIZE
17811666c9e308 twilight-tree-8836 stopped gru flyio/rchab:sha-a4467b8 fdaa:0:3335:a7b:fa:5e1:1b5c:2 vol_n0l9vl22gymr635d 2023-07-21T14:56:12Z 2023-07-21T15:08:58Z shared-cpu-4x:4096MB
We will be cloning 17811666c9e308!
- Clone your machine and attach the new volume
$ fly m clone 17811666c9e308 --attach-volume vol_0o6d423kno7r87gy -a fly-builder-name
Cloning machine 17811666c9e308 into region gru
Attaching existing volume vol_0o6d423kno7r87gy
Provisioning a new machine with image registry-1.docker.io/flyio/rchab:sha-a4467b8@sha256:2075bb4872cf81bd8a52eb7a418ac1883b620ea0eca9c796ffb556b8c036fdc5...
Machine 3d8d9779f2d138 has been created...
Waiting for machine 3d8d9779f2d138 to start...
No health checks found
Machine has been successfully cloned!
- Destroy your old machine so our proxy will not be confused routing to the old one
$ fly m destroy 17811666c9e308 -a fly-builder-name
machine 17811666c9e308 was found and is currently in stopped state, attempting to destroy...
17811666c9e308 has been destroyed
If your builder is running add --force to destroy that machine.
At this point, you already have a builder on a new zone. Volumes dictate where the machine should go.
- Destroy your old volume
This command doesn’t require the -a fly-builder-name part.
$ fly vol destroy vol_n0l9vl22gymr635d
Warning! Individual volumes are pinned to individual hosts. You should create two or more volumes per application. Deleting this volume will leave you with 1 volume(s) for this application, and it is not reversible. Learn more at https://fly.io/docs/reference/volumes/
? Are you sure you want to destroy this volume? Yes
Error: failed destroying volume: This volume is attached to a machine(17811666c9e308) and cannot be deleted.
If it says the volume is still attached it’s fine! Distributed system state sync are hard 
$ fly vol destroy vol_n0l9vl22gymr635d
Warning! Individual volumes are pinned to individual hosts. You should create two or more volumes per application. Deleting this volume will leave you with 1 volume(s) for this application, and it is not reversible. Learn more at https://fly.io/docs/reference/volumes/
? Are you sure you want to destroy this volume? Yes
Destroyed volume vol_n0l9vl22gymr635d from fly-builder-name
What if I break my builder?
You can always delete your builder app. Always. Fly will create one in the next deploy.
Let me know if this How-to works for you!
We’ve provisioned new hosts in MAD last week. Let us know if you still have this issue.