My remote builder with machine id 9185517b45d983 (app fly-builder-green-pond-1760 ) is stuck in created state. It seems it is not able to pull the container image to get itself started? Logs -
2023-01-29T05:51:08.084 runner[9185517b45d983] sin [info] Pulling container image
2023-01-29T05:51:16.642 runner[9185517b45d983] sin [info] Pulling container image
2023-01-29T05:51:22.712 runner[9185517b45d983] sin [info] Pulling container image
2023-01-29T05:51:25.247 runner[9185517b45d983] sin [info] Pulling container image
I cannot stop or destroy this machine using the API either, as it is still in created state.
I saw a similar report on the forum here (Fly API error....). Nothing seems to be off in the status dashboard. Created a new thread as I thought the subject can be more indicative.
Perhaps. the builder is still stuck in created state though. And I can’t get remote builds done (on which I depend). Someone from Fly.io team please help!
2023-01-30T03:24:26.674 runner[1781031f9e9689] sin [info] Pulling container image
2023-01-30T03:24:28.306 runner[1781031f9e9689] sin [info] Pulling container image
2023-01-30T03:24:29.596 runner[1781031f9e9689] sin [info] Pulling container image
This time while running fly deploy I specified the region as maa but the builder still seems to be created in sin. Here’s the command I am using to trigger the remote build
fly deploy --region=maa --image-label v0.0.1 --build-only --push --remote-only -a <app_name>
output
==> Verifying app config
--> Verified app config
==> Building image
WARN Failed to start remote builder heartbeat: You hit a Fly API error with request ID: 01GR0B3S260GHKD4YJTFFM527X-maa
WARN Remote builder did not start in time. Check remote builder logs with `flyctl logs -a fly-builder-autumn-sea-3191`
Error failed to fetch an image or build from source: error connecting to docker: remote builder app unavailable
@hi.kanily It turns out that you can’t set the builder’s region. They spawn in the region closest to you geographically. You also can’t set the app’s region because it’s tied to the region of the volume.
I think allowing region override for builders should be an option for cases like this.
Anyway, I came up with a hack you can try to workaround this:
Trigger a build:
fly deploy ... --region sjc --build-only
Once it shows “WARN Failed to start remote builder heartbeat” or “waiting for remote builder…”, CTRL+C it. You only want the builder to spawn. (I’m using sjc region but you can use any, remember to replace it in the steps below if you change it!)
Grab the builder app name:
fly apps list
The builder app starts with fly-builder-....
Grab the builder app’s machine ID:
fly machine list -a $BUILDER_APP_NAME
There should only be one machine, in the sin region. Copy that ID.
Clone the machine into a different region:
fly machine clone $MACHINE_ID_FROM_ABOVE --region sjc -a $BUILDER_APP_NAME
This will clone the machine and its volume into the sjc region.
Verify that sjc is in the app’s region pool:
fly regions list -a $BUILDER_APP_NAME
Region Pool:
sin
sjc
Backup Region:
(Don’t bother with fly regions set .., it won’t go through because of the attached volume in sin.)
Monitor the logs of your builder, it should hint that sjc is up and running:
Trigger your build again, watch the logs. sjc should be the one performing the build. The sin builder will be stuck, but sjc should be proceeding normally.
To clean this up, fly apps destroy $BUILDER_APP_NAME will remove the apps/machines/volumes for you.
Hope this helps!
And just out of curiosity, why are you depending on remote-only builds?
The issue seems to stem from an upstream network; I’ve had poor connectivity to lots of services outside of Fly. I believe a fix is in progress (Fly status page).
PS. Regarding the dependency on a remote builder…
So you’re using Fly as a build server? That’s interesting.
I’m trying that out now, will report back! It probably won’t be charged because the machine doesn’t seem to know that it’s a builder, but the app does, so the billing probably skips resources connected to a builder app.
More fundamentally, if builders are free and ssh-able, is there anything preventing abuse of builder-spawned machines?
