"remote builder app unavailable" when launching

Failed to launch when following the tutorial: Build, Deploy And Run A Go Application

WARN Remote builder did not start in time. Check remote builder logs with `flyctl logs -a fly-builder-aged-breeze-3280`
Error remote builder app unavailable

The region I chose is Hong Kong.

Hi just so we can get some more details can you run flyctl logs -a fly-builder-aged-breeze-3280 and attempt a redeploy with the following command: ‘LOG_LEVEL=debug flyctl deploy’

There’s no extra log in the output of flyctl logs when I run flyctl deploy, but the original output is pasted below:

2022-05-23T13:23:26Z app[4e36f123] nrt [info]time="2022-05-23T13:23:26.739846588Z" level=debug msg="Calling GET /v1.41/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0"
2022-05-23T13:23:27Z app[4e36f123] nrt [info]time="2022-05-23T13:23:27.742622315Z" level=debug msg="checking docker activity"
2022-05-23T13:23:27Z app[4e36f123] nrt [info]time="2022-05-23T13:23:27.743277469Z" level=debug msg="Calling GET /v1.41/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0"
2022-05-23T13:23:28Z app[4e36f123] nrt [info]time="2022-05-23T13:23:28.746981292Z" level=debug msg="checking docker activity"
2022-05-23T13:23:28Z app[4e36f123] nrt [info]time="2022-05-23T13:23:28.747781208Z" level=debug msg="Calling GET /v1.41/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0"
2022-05-23T13:23:29Z app[4e36f123] nrt [info]time="2022-05-23T13:23:29.750814148Z" level=debug msg="checking docker activity"
2022-05-23T13:23:29Z app[4e36f123] nrt [info]time="2022-05-23T13:23:29.751447937Z" level=debug msg="Calling GET /v1.41/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0"
2022-05-23T13:23:30Z app[4e36f123] nrt [info]time="2022-05-23T13:23:30.755852292Z" level=debug msg="checking docker activity"
2022-05-23T13:23:30Z app[4e36f123] nrt [info]time="2022-05-23T13:23:30.756432150Z" level=debug msg="Calling GET /v1.41/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0"
2022-05-23T13:23:31Z app[4e36f123] nrt [info]time="2022-05-23T13:23:31.760777074Z" level=debug msg="checking docker activity"
2022-05-23T13:23:31Z app[4e36f123] nrt [info]time="2022-05-23T13:23:31.764445561Z" level=debug msg="Calling GET /v1.41/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0"
2022-05-23T13:23:32Z app[4e36f123] nrt [info]time="2022-05-23T13:23:32.779880508Z" level=debug msg="checking docker activity"
2022-05-23T13:23:32Z app[4e36f123] nrt [info]time="2022-05-23T13:23:32.780417165Z" level=debug msg="Calling GET /v1.41/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0"
2022-05-23T13:23:33Z app[4e36f123] nrt [info]time="2022-05-23T13:23:33.783728096Z" level=debug msg="checking docker activity"
2022-05-23T13:23:33Z app[4e36f123] nrt [info]time="2022-05-23T13:23:33.784132344Z" level=debug msg="Calling GET /v1.41/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0"
2022-05-23T13:23:34Z app[4e36f123] nrt [info]time="2022-05-23T13:23:34.788159234Z" level=debug msg="checking docker activity"
2022-05-23T13:23:34Z app[4e36f123] nrt [info]time="2022-05-23T13:23:34.788662567Z" level=debug msg="Calling GET /v1.41/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0"
2022-05-23T13:23:35Z app[4e36f123] nrt [info]time="2022-05-23T13:23:35.792195812Z" level=debug msg="checking docker activity"
2022-05-23T13:23:35Z app[4e36f123] nrt [info]time="2022-05-23T13:23:35.792767355Z" level=debug msg="Calling GET /v1.41/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0"
2022-05-23T13:23:36Z app[4e36f123] nrt [info]time="2022-05-23T13:23:36.796265191Z" level=debug msg="checking docker activity"
2022-05-23T13:23:36Z app[4e36f123] nrt [info]time="2022-05-23T13:23:36.797021724Z" level=debug msg="Calling GET /v1.41/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0"
2022-05-23T13:23:37Z app[4e36f123] nrt [info]time="2022-05-23T13:23:37.800655809Z" level=debug msg="checking docker activity"
2022-05-23T13:23:37Z app[4e36f123] nrt [info]time="2022-05-23T13:23:37.801459960Z" level=debug msg="Calling GET /v1.41/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0"
2022-05-23T13:23:38Z app[4e36f123] nrt [info]time="2022-05-23T13:23:38.808040359Z" level=debug msg="checking docker activity"
2022-05-23T13:23:38Z app[4e36f123] nrt [info]time="2022-05-23T13:23:38.808502737Z" level=debug msg="Calling GET /v1.41/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0"
2022-05-23T13:23:39Z app[4e36f123] nrt [info]time="2022-05-23T13:23:39.811552344Z" level=debug msg="checking docker activity"
2022-05-23T13:23:39Z app[4e36f123] nrt [info]time="2022-05-23T13:23:39.812628325Z" level=debug msg="Calling GET /v1.41/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0"
2022-05-23T13:23:40Z app[4e36f123] nrt [info]time="2022-05-23T13:23:40.816319982Z" level=debug msg="checking docker activity"
2022-05-23T13:23:40Z app[4e36f123] nrt [info]time="2022-05-23T13:23:40.816835818Z" level=debug msg="Calling GET /v1.41/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0"
2022-05-23T13:23:41Z app[4e36f123] nrt [info]time="2022-05-23T13:23:41.819077051Z" level=debug msg="checking docker activity"
2022-05-23T13:23:41Z app[4e36f123] nrt [info]time="2022-05-23T13:23:41.820273120Z" level=debug msg="Calling GET /v1.41/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0"
2022-05-23T13:23:42Z app[4e36f123] nrt [info]time="2022-05-23T13:23:42.824849753Z" level=debug msg="checking docker activity"
2022-05-23T13:23:42Z app[4e36f123] nrt [info]time="2022-05-23T13:23:42.825477511Z" level=debug msg="Calling GET /v1.41/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0"
2022-05-23T13:23:43Z app[4e36f123] nrt [info]time="2022-05-23T13:23:43.828716074Z" level=debug msg="checking docker activity"
2022-05-23T13:23:43Z app[4e36f123] nrt [info]time="2022-05-23T13:23:43.829221336Z" level=debug msg="Calling GET /v1.41/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0"
2022-05-23T13:23:44Z app[4e36f123] nrt [info]time="2022-05-23T13:23:44.832641305Z" level=debug msg="checking docker activity"
2022-05-23T13:23:44Z app[4e36f123] nrt [info]time="2022-05-23T13:23:44.833392087Z" level=debug msg="Calling GET /v1.41/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0"
2022-05-23T13:23:45Z app[4e36f123] nrt [info]time="2022-05-23T13:23:45.839990467Z" level=debug msg="checking docker activity"
2022-05-23T13:23:45Z app[4e36f123] nrt [info]time="2022-05-23T13:23:45.840494410Z" level=debug msg="Calling GET /v1.41/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0"
2022-05-23T13:23:46Z app[4e36f123] nrt [info]time="2022-05-23T13:23:46.845129190Z" level=debug msg="checking docker activity"
2022-05-23T13:23:46Z app[4e36f123] nrt [info]time="2022-05-23T13:23:46.846046581Z" level=debug msg="Calling GET /v1.41/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0"
2022-05-23T13:23:47Z app[4e36f123] nrt [info]time="2022-05-23T13:23:47.849675470Z" level=debug msg="checking docker activity"
2022-05-23T13:23:47Z app[4e36f123] nrt [info]time="2022-05-23T13:23:47.850288828Z" level=debug msg="Calling GET /v1.41/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0"
2022-05-23T13:23:48Z app[4e36f123] nrt [info]time="2022-05-23T13:23:48.854021309Z" level=debug msg="checking docker activity"
2022-05-23T13:23:48Z app[4e36f123] nrt [info]time="2022-05-23T13:23:48.854582530Z" level=debug msg="Calling GET /v1.41/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0"
2022-05-23T13:23:49Z app[4e36f123] nrt [info]time="2022-05-23T13:23:49.858358865Z" level=debug msg="checking docker activity"
2022-05-23T13:23:49Z app[4e36f123] nrt [info]time="2022-05-23T13:23:49.858944080Z" level=debug msg="Calling GET /v1.41/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0"
2022-05-23T13:23:50Z app[4e36f123] nrt [info]time="2022-05-23T13:23:50.863115753Z" level=debug msg="checking docker activity"
2022-05-23T13:23:50Z app[4e36f123] nrt [info]time="2022-05-23T13:23:50.863623169Z" level=debug msg="Calling GET /v1.41/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0"
2022-05-23T13:23:51Z app[4e36f123] nrt [info]time="2022-05-23T13:23:51.869289412Z" level=debug msg="checking docker activity"
2022-05-23T13:23:51Z app[4e36f123] nrt [info]time="2022-05-23T13:23:51.869906154Z" level=debug msg="Calling GET /v1.41/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0"
2022-05-23T13:23:52Z app[4e36f123] nrt [info]time="2022-05-23T13:23:52.873381100Z" level=debug msg="checking docker activity"
2022-05-23T13:23:52Z app[4e36f123] nrt [info]time="2022-05-23T13:23:52.874116731Z" level=debug msg="Calling GET /v1.41/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0"
2022-05-23T13:23:53Z app[4e36f123] nrt [info]time="2022-05-23T13:23:53.877460032Z" level=debug msg="checking docker activity"
2022-05-23T13:23:53Z app[4e36f123] nrt [info]time="2022-05-23T13:23:53.878027357Z" level=debug msg="Calling GET /v1.41/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0"
2022-05-23T13:23:54Z app[4e36f123] nrt [info]time="2022-05-23T13:23:54.881003116Z" level=debug msg="checking docker activity"
2022-05-23T13:23:54Z app[4e36f123] nrt [info]time="2022-05-23T13:23:54.882029964Z" level=debug msg="Calling GET /v1.41/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0"
2022-05-23T13:23:55Z app[4e36f123] nrt [info]time="2022-05-23T13:23:55.887299638Z" level=debug msg="checking docker activity"
2022-05-23T13:23:55Z app[4e36f123] nrt [info]time="2022-05-23T13:23:55.888036585Z" level=debug msg="Calling GET /v1.41/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0"
2022-05-23T13:23:56Z app[4e36f123] nrt [info]time="2022-05-23T13:23:56.892336295Z" level=debug msg="checking docker activity"
2022-05-23T13:23:56Z app[4e36f123] nrt [info]time="2022-05-23T13:23:56.893041790Z" level=debug msg="Calling GET /v1.41/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0"
2022-05-23T13:23:57Z app[4e36f123] nrt [info]time="2022-05-23T13:23:57.897782041Z" level=debug msg="checking docker activity"
2022-05-23T13:23:57Z app[4e36f123] nrt [info]time="2022-05-23T13:23:57.898805613Z" level=debug msg="Calling GET /v1.41/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0"
2022-05-23T13:23:58Z app[4e36f123] nrt [info]time="2022-05-23T13:23:58.902542619Z" level=debug msg="checking docker activity"
2022-05-23T13:23:58Z app[4e36f123] nrt [info]time="2022-05-23T13:23:58.903338579Z" level=debug msg="Calling GET /v1.41/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0"
2022-05-23T13:23:59Z app[4e36f123] nrt [info]time="2022-05-23T13:23:59.907639007Z" level=debug msg="checking docker activity"
2022-05-23T13:23:59Z app[4e36f123] nrt [info]time="2022-05-23T13:23:59.908129240Z" level=debug msg="Calling GET /v1.41/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0"
2022-05-23T13:24:00Z app[4e36f123] nrt [info]time="2022-05-23T13:24:00.911119896Z" level=debug msg="checking docker activity"
2022-05-23T13:24:00Z app[4e36f123] nrt [info]time="2022-05-23T13:24:00.911750100Z" level=debug msg="Calling GET /v1.41/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0"
2022-05-23T13:24:01Z app[4e36f123] nrt [info]time="2022-05-23T13:24:01.915224113Z" level=debug msg="checking docker activity"
2022-05-23T13:24:01Z app[4e36f123] nrt [info]time="2022-05-23T13:24:01.917841297Z" level=debug msg="Calling GET /v1.41/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0"
2022-05-23T13:24:02Z app[4e36f123] nrt [info]time="2022-05-23T13:24:02.921071222Z" level=debug msg="checking docker activity"
2022-05-23T13:24:02Z app[4e36f123] nrt [info]time="2022-05-23T13:24:02.921443530Z" level=debug msg="Calling GET /v1.41/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0"
2022-05-23T13:24:03Z app[4e36f123] nrt [info]time="2022-05-23T13:24:03.924090587Z" level=debug msg="checking docker activity"
2022-05-23T13:24:03Z app[4e36f123] nrt [info]time="2022-05-23T13:24:03.924738678Z" level=debug msg="Calling GET /v1.41/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0"
2022-05-23T13:24:04Z app[4e36f123] nrt [info]time="2022-05-23T13:24:04.928547332Z" level=debug msg="checking docker activity"
2022-05-23T13:24:04Z app[4e36f123] nrt [info]time="2022-05-23T13:24:04.929196904Z" level=debug msg="Calling GET /v1.41/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0"
2022-05-23T13:24:05Z app[4e36f123] nrt [info]time="2022-05-23T13:24:05.835938635Z" level=info msg="Deadline reached without docker build"
2022-05-23T13:24:05Z app[4e36f123] nrt [info]time="2022-05-23T13:24:05.836086219Z" level=info msg="shutting down"
2022-05-23T13:24:05Z app[4e36f123] nrt [info]time="2022-05-23T13:24:05.837895990Z" level=info msg="gracefully stopped\n"
2022-05-23T13:24:05Z app[4e36f123] nrt [info]time="2022-05-23T13:24:05.838162363Z" level=debug msg="disk space used: 1.03%"
2022-05-23T13:24:05Z app[4e36f123] nrt [info]time="2022-05-23T13:24:05.838247670Z" level=info msg="Waiting for dockerd to exit"
2022-05-23T13:24:05Z app[4e36f123] nrt [info]time="2022-05-23T13:24:05.839044238Z" level=info msg="Processing signal 'interrupt'"
2022-05-23T13:24:05Z app[4e36f123] nrt [info]time="2022-05-23T13:24:05.840928053Z" level=debug msg="daemon configured with a 15 seconds minimum shutdown timeout"
2022-05-23T13:24:05Z app[4e36f123] nrt [info]time="2022-05-23T13:24:05.841004898Z" level=debug msg="start clean shutdown of all containers with a 15 seconds timeout..."
2022-05-23T13:24:05Z app[4e36f123] nrt [info]time="2022-05-23T13:24:05.841341421Z" level=debug msg="found 0 orphan layers"
2022-05-23T13:24:05Z app[4e36f123] nrt [info]time="2022-05-23T13:24:05.843517570Z" level=debug msg="Unix socket /var/run/docker/libnetwork/efdad381dd7b.sock doesn't exist. cannot accept client connections"
2022-05-23T13:24:05Z app[4e36f123] nrt [info]time="2022-05-23T13:24:05.844656717Z" level=debug msg="Cleaning up old mountid : start."
2022-05-23T13:24:05Z app[4e36f123] nrt [info]time="2022-05-23T13:24:05.845001891Z" level=info msg="stopping event stream following graceful shutdown" error="<nil>" module=libcontainerd namespace=moby
2022-05-23T13:24:05Z app[4e36f123] nrt [info]time="2022-05-23T13:24:05.845193367Z" level=debug msg="Cleaning up old mountid : done."
2022-05-23T13:24:05Z app[4e36f123] nrt [info]time="2022-05-23T13:24:05.845548357Z" level=debug msg="unmounting daemon root" mountpoint=/data/docker
2022-05-23T13:24:05Z app[4e36f123] nrt [info]time="2022-05-23T13:24:05.846256231Z" level=debug msg="Clean shutdown succeeded"
2022-05-23T13:24:05Z app[4e36f123] nrt [info]time="2022-05-23T13:24:05.846302014Z" level=info msg="Daemon shutdown complete"
2022-05-23T13:24:05Z app[4e36f123] nrt [info]time="2022-05-23T13:24:05.846525260Z" level=info msg="stopping event stream following graceful shutdown" error="context canceled" module=libcontainerd namespace=plugins.moby
2022-05-23T13:24:05Z app[4e36f123] nrt [info]time="2022-05-23T13:24:05.846662964Z" level=info msg="stopping healthcheck following graceful shutdown" module=libcontainerd
2022-05-23T13:24:05Z app[4e36f123] nrt [info]time="2022-05-23T13:24:05.847268890Z" level=debug msg="received signal" signal=terminated
2022-05-23T13:24:05Z app[4e36f123] nrt [info]time="2022-05-23T13:24:05.847892209Z" level=debug msg="sd notification" error="<nil>" notified=false state="STOPPING=1"
2022-05-23T13:24:05Z app[4e36f123] nrt [info]time="2022-05-23T13:24:05.933388387Z" level=debug msg="checking docker activity"
2022-05-23T13:24:05Z app[4e36f123] nrt [info]time="2022-05-23T13:24:05.933926041Z" level=debug msg="Calling GET /v1.41/containers/json?filters=%7B%22status%22%3A%7B%22running%22%3Atrue%7D%7D&limit=0"
2022-05-23T13:24:06Z app[4e36f123] nrt [info]time="2022-05-23T13:24:06.853843981Z" level=info msg="dockerd has exited"

And flyctl deploy fails with the following output at the end:

===> BUILDING
Starting build
Running build for buildpack paketo-buildpacks/ca-certificates@3.2.1
Looking up buildpack
Finding plan
Running build for buildpack Paketo CA Certificates Buildpack 3.2.1
Creating plan directory
Preparing paths
Running build command

Paketo CA Certificates Buildpack 3.2.1
  https://github.com/paketo-buildpacks/ca-certificates
  Launch Helper: Contributing to layer
    Creating /layers/paketo-buildpacks_ca-certificates/helper/exec.d/ca-certificates-helper
Processing layers
Updating environment
Reading output files
ERROR: failed to build: bom table isn't supported in this buildpack api version. The BOM should be written to <layer>.sbom.<ext>, launch.sbom.<ext>, or build.sbom.<ext>
DEBUG result image:<nil> error:executing lifecycle: failed with status code: 52
Error failed to fetch an image or build from source: executing lifecycle: failed with status code: 52

It’s working now.
I guess all the errors I saw were caused by the GFW preventing me to access the builders because when I use a VPN server in HK everything works.
The build step is not as fast as I expected for a Hello World Go example, though.