Remote builder timing out when pulling docker images

Questions / Help
1

This may be related to Remote builder connection timeout but creating a new thread as I can connect to the remote builder but it’s timing out connecting to the Docker Registry

[nix-shell:~/projects/bb/dispatch](rider_show_link)$ flyctl deploy -c fly.staging.toml --remote-only
Deploying bike-brigade-staging
==> Validating app configuration
--> Validating app configuration done
Services
TCP 80/443 ⇢ 4000
Remote builder fly-builder-wispy-violet-2739 ready
==> Creating build context
--> Creating build context done
==> Building image with Docker
--> docker host: 20.10.12 linux x86_64
[+] Building 4.4s (0/1)
[+] Building 30.9s (4/4) FINISHED
 => [internal] load remote build context                                                                                                                                     0.0s
 => copy /context /                                                                                                                                                          0.7s
 => ERROR [internal] load metadata for docker.io/hexpm/elixir:1.13.0-erlang-24.1-alpine-3.14.0                                                                              30.0s
 => ERROR [internal] load metadata for docker.io/library/alpine:3.14                                                                                                        30.0s
------
 > [internal] load metadata for docker.io/hexpm/elixir:1.13.0-erlang-24.1-alpine-3.14.0:
------
------
 > [internal] load metadata for docker.io/library/alpine:3.14:
------
Error error building: failed to solve with frontend dockerfile.v0: failed to create LLB definition: failed to do request: Head "https://registry-1.docker.io/v2/hexpm/elixir/manifests/1.13.0-erlang-24.1-alpine-3.14.0": dial tcp 34.237.244.67:443: i/o timeout

I believe this is an issue with fly and not the docker registry because docker pull hexpm/elixir:1.13.0-erlang-24.1-alpine-3.14.0 works correctly for me.

2

Perhaps this is the culprit? https://github.com/superfly/flyctl-actions/commit/372a8fcc09d83f0210be78687813a9ef4cfde339

3

This is saying that it can’t connect to that IP to get the erlang image to build with. You can try running fly apps destroy fly-builder-wispy-violet-2739, then running your build again. This gives you a builder on different hardware/network.

The Docker public registry has been very unreliable lately. Network timeouts like that are usually caused by some routing issue between us Docker. We have some workarounds in mind but we haven’t been able to build them yet.

That github actions commit does default to a remote builder. You can try running with --local-only to use your own Docker. It’s very slow in GitHub because there’s no image caching, but it works ok.

4

Thanks @kurt

Destroying the builder and trying again fixed the issue with remote builds.

I found that the --local-only flag was broken in the action because BASH IS THE WORST - and left a PR fixing it :slight_smile: https://github.com/superfly/flyctl-actions/pull/15