remote builder machine stuck in created state

6

@hi.kanily It turns out that you can’t set the builder’s region. They spawn in the region closest to you geographically. You also can’t set the app’s region because it’s tied to the region of the volume.

I think allowing region override for builders should be an option for cases like this.

Anyway, I came up with a hack you can try to workaround this:

  1. Trigger a build:

    fly deploy ... --region sjc --build-only

    Once it shows “WARN Failed to start remote builder heartbeat” or “waiting for remote builder…”, CTRL+C it. You only want the builder to spawn. (I’m using sjc region but you can use any, remember to replace it in the steps below if you change it!)

  2. Grab the builder app name:

    fly apps list

    The builder app starts with fly-builder-....

  3. Grab the builder app’s machine ID:

    fly machine list -a $BUILDER_APP_NAME

    There should only be one machine, in the sin region. Copy that ID.

  4. Clone the machine into a different region:

    fly machine clone $MACHINE_ID_FROM_ABOVE --region sjc -a $BUILDER_APP_NAME

    This will clone the machine and its volume into the sjc region.

  5. Verify that sjc is in the app’s region pool:

    fly regions list -a $BUILDER_APP_NAME
Region Pool:
sin
sjc
Backup Region:

    (Don’t bother with fly regions set .., it won’t go through because of the attached volume in sin.)

  6. Monitor the logs of your builder, it should hint that sjc is up and running:

    fly logs -a $BUILDER_APP_NAME
2023-01-30T06:14:00Z app[732870d7be5585] sjc [info]time="2023-01-30T06:14:00.825682582Z" level=debug msg="checking docker activity"

  7. Trigger your build again, watch the logs. sjc should be the one performing the build. The sin builder will be stuck, but sjc should be proceeding normally.

To clean this up, fly apps destroy $BUILDER_APP_NAME will remove the apps/machines/volumes for you.

Hope this helps!

And just out of curiosity, why are you depending on remote-only builds?

edit: phrasing + question

3 Likes