fly deploys are failing. Remote builder failing to build.

1

fly deploy is failing, below is the stack trace:

❯ fly deploy
==> Verifying app config
--> Verified app config
==> Building image
WARN Failed to start remote builder heartbeat: Couldn't allocate volume, no disks available (region: vin, size_gb: 50)
Error failed to fetch an image or build from source: error connecting to docker: Couldn't allocate volume, no disks available (region: vin, size_gb: 50)

I have upgraded to the latest fly CLI.

fly status doesn’t show any reported outages.

fly doctor seems fine:

Testing authentication token... PASSED
Testing flyctl agent... PASSED
Testing local Docker instance... PASSED
Pinging WireGuard gateway (give us a sec)... PASSED

App specific checks for blockade:
Checking that app has ip addresses allocated... PASSED
Checking A record for blockade.fly.dev... PASSED
Checking AAAA record for blockade.fly.dev... PASSED

Build checks for blockade:
Checking docker context size (this may take little bit)... PASSED (6.2 MB)

Even tried pruning my local docker volumes, just in case to free up local memory. No luck.

The remote builder is not working for deploying my existing app.

2

I am having the same problem deploying a Flask application

==> Verifying app config
--> Verified app config
==> Building image
Remote builder fly-builder-hidden-darkness-4256 ready
Error failed to fetch an image or build from source: app does not have a Dockerfile or buildpacks configured. See https://fly.io/docs/reference/configuration/#the-build-section

In the builder logs I see a lot of lines like that:

2023-01-05T13:50:42.064 app[3d8d3e1ce499e8] fra [info] time="2023-01-05T13:50:42.057507479Z" level=debug msg="Calling GET /v1.41/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0"
3

Hello! @ctrlaltdylan Please try deleting your remote builder instance - Fly will recreate it and hopefully resolve that issue. vin might be out of capacity, in theory a builder shouldn’t have been created there.

@liutprand Try the same thing! That’s a different error in your case, but deleting the builder (resulting in Fly recreating a new one) might clear that up.

If both/either of you have Docker installed locally (and are on Intel CPU’s instead of ARM - e.g. Mac M1/M2 computers), you can also use the --local-only flag to use your local Docker to build the image.

4

Hi @fideloper-fly

Thanks for the tips. But can you be more specific about how to delete the remote builder instance?

I’m not sure how to do that? Is that a dashboard operation or a CLI command? If so what is it?

5

You can do that in either place - the builder should appear in your dashboard. Click into it, head to Settings, and you’ll find a place to delete it.

Else you can use flyctl:

fly apps list
fly apps destroy <app-name>

If you can’t find a builder instance, it’s possible it was cleaned up in an automated process and will be re-created by itself in the next deploy.

1 Like
6

Thanks @fideloper-fly I didn’t realize that a remote builder was literally a temporary app instance.

Deleting the auto-generated builder app looks like that did the trick.

But just anecdotally, I did wipe all other apps from my dashboard earlier today, only my production app was left and I did still see this deployment memory error.

So for anyone else visiting this thread in the future, a builder fly app is instantiated when you start a deploy, and it will have an autogenerated name like fly-builder-white-shadow-9057 for example.

When your deployment fails, this temporary app persists. You’ll need to delete these temporary apps from your account in order to free up space on the remote builder.

Thanks @fideloper-fly - P.S. I think I read your blog back in the day on hosting PHP Laravel apps on linux, super helpful stuff. Now we’re in the world of containers, but those foundations have helped me so much. Much appreciated!

7

I think the vin region is “special” and shouldn’t be getting builders currently (it looks like the new one created for you is in ORD).

In that case, a new volume couldn’t be created for that builder (rather than an existing builder running out of disk space - that’s possible too, however!)

I’m glad you found that past articles and other stuff helpful! It’s helping me here too :joy:

1 Like
8

I had already tried to delete the builder app and relaunch the deploy, but it doesn’t change. Still getting the same error…

Any other ideas?

9

I have deleted my free builder several times and am still getting the same error. The new builders are also in the pending state and have no volumes so perhaps there really is an issue with where they are getting deployed to (vin - in my case).

10

Hey!
In your case, the error was app does not have a Dockerfile or buildpacks configured - do you not have a Dockerfile, or perhaps it’s not located in the same directory as your fly.toml?

11

vin is an issue, yes. That shouldn’t be getting new builders right now.

12

FWIW I’m seeing the same. Deleting the app and retrying fly deploy just attempts to create the builder in “vin” again, which fails.

ntalbott@nevin doubloon % fly apps destroy fly-builder-polished-wave-4335
Destroying an app is not reversible.
? Destroy app fly-builder-polished-wave-4335? Yes
Destroyed app fly-builder-polished-wave-4335

ntalbott@nevin doubloon % fly deploy
==> Verifying app config
--> Verified app config
==> Building image
WARN Failed to start remote builder heartbeat: Couldn't allocate volume, no disks available (region: vin, size_gb: 50)
Error failed to fetch an image or build from source: error connecting to docker: Couldn't allocate volume, no disks available (region: vin, size_gb: 50)

ntalbott@nevin doubloon % fly apps list
NAME                             	OWNER  	STATUS 	PLATFORM	LATEST DEPLOY
...
fly-builder-empty-wildflower-2863	loor-tv	pending	machines
13

Would you mind trying again? We’ve just made a configuration change that should prevent vin from being chosen.

14

I think you fixed it. At least it’s making it a lot further than it was. Thanks!

15

Seems to be working now. Thanks!

1 Like
16

I am currently experiencing the vin issue

fly deploy
==> Verifying app config
--> Verified app config
==> Building image
WARN Failed to start remote builder heartbeat: Couldn't allocate volume, no disks available (region: vin, size_gb: 50)
Error failed to fetch an image or build from source: error connecting to docker: Couldn't allocate volume, no disks available (region: vin, size_gb: 50)

I am on latest fly version (fly v0.0.446 darwin/arm64 Commit: 352bd3e3 BuildDate: 2023-01-11T19:05:35Z)

I have attempted to delete the temporary builder instance with fly apps destroy a few times and the error persists

17

I’d suggest you to build your app locally using a Dockerfile then when deploying use --local-only
flag,

fly deploy --local-only

it will build your app using the Dockerfile and deploy without the need of a remote builder, i was having the same issue over and over, solved when did the above.

18

Thank you. Unfortunately I believe local building does not work on the Mac M1 / M2 chips.

19

I’m also in this position. I have an x86 windows machine, and I’m having a different issue over there.

20

It started happening with me today.
Somehow hkg region appears to be in bad shape.

I posted my troubleshooting here: Failed to start remote builder heartbeat: Couldn't allocate volume, not enough compute capacity left in hkg - #2 by heliocola

Somehow my account (or builder app) is stuck with hkg and even an older app in another region is not deploying.

I’ve managed to install docker, fix a symlink issue, and use --local-only.
Hopefully this will return to normal soon.

:hug-to-fly.io-ops-team: in case they are in need!