mix local.hex --force network error in remote builder

Hey,
I think I’m running into the same issue as this person:

I’m also on an M1 Mac, but in CDG. Tried last week and got the same error repeatedly until I gave up. Thought it was just a fluke. Tried again today with the same result. Tried to destroy the builder so a new one would be created, but same issue.
Here is the full output in case it helps:

$ fly deploy
==> Verifying app config
--> Verified app config
==> Building image
Remote builder fly-builder-shy-forest-113 ready
==> Creating build context
--> Creating build context done
==> Building image with Docker
--> docker host: 20.10.12 linux x86_64
[+] Building 0.3s (0/1)
[+] Building 1.7s (13/28)
 => CACHED [internal] load remote build context                                                                                   0.0s
 => CACHED copy /context /                                                                                                        0.0s
 => [internal] load metadata for docker.io/library/debian:bullseye-20210902-slim                                                  0.6s
 => [internal] load metadata for docker.io/hexpm/elixir:1.13.4-erlang-24.1.7-debian-bullseye-20210902-slim                        0.7s
 => [builder  1/18] FROM docker.io/hexpm/elixir:1.13.4-erlang-24.1.7-debian-bullseye-20210902-slim@sha256:38a5bc18ada41443c1f1cb  0.0s
 => [stage-1 1/6] FROM docker.io/library/debian:bullseye-20210902-slim@sha256:e3ed4be20c22a1358020358331d177aa2860632f25b21681d7  0.0s
 => CACHED [stage-1 2/6] RUN apt-get update -y && apt-get install -y libstdc++6 openssl libncurses5 locales   && apt-get clean &  0.0s
 => CACHED [stage-1 3/6] RUN sed -i '/en_US.UTF-8/s/^# //g' /etc/locale.gen && locale-gen                                         0.0s
 => CACHED [stage-1 4/6] WORKDIR /app                                                                                             0.0s
 => CACHED [stage-1 5/6] RUN chown nobody /app                                                                                    0.0s
 => CACHED [builder  2/18] RUN apt-get update -y && apt-get install -y build-essential git nodejs npm     && apt-get clean && rm  0.0s
 => CACHED [builder  3/18] WORKDIR /app                                                                                           0.0s
 => ERROR [builder  4/18] RUN mix local.hex --force &&     mix local.rebar --force                                                1.0s
------
 > [builder  4/18] RUN mix local.hex --force &&     mix local.rebar --force:
#13 0.974 ** (Mix) httpc request failed with: {:failed_connect, [{:to_address, {'repo.hex.pm', 443}}, {:inet, [:inet], :closed}]}
#13 0.974
#13 0.974 Could not install Hex because Mix could not download metadata at https://repo.hex.pm/installs/hex-1.x.csv.
#13 0.974
#13 0.974 Alternatively, you can compile and install Hex directly with this command:
#13 0.974
#13 0.974     mix archive.install github hexpm/hex branch latest
#13 0.974
------
Error failed to fetch an image or build from source: error building: executor failed running [/bin/sh -c mix local.hex --force &&     mix local.rebar --force]: exit code: 1

edit:
I tried to do the remote build in another region, but it doesn’t work and I’m not even sure it’s supposed to:

$ fly destroy <builder>
$ fly deploy --remote-only --region <another>
$ fly regions list <new-builder>
Region Pool:
cdg
Backup Region:

I tried in these regions now, all of them fail with the same error: cdg, mad, sea. Hex works on my local machine, so it appears fly.io network is borked or being blocked maybe?

I get a slightly different error message sometimes:

? Would you like to deploy now? Yes
==> Building image
Remote builder fly-builder-dawn-river-3367 ready
==> Creating build context
--> Creating build context done
==> Building image with Docker
--> docker host: 20.10.12 linux x86_64
[+] Building 0.1s (0/1)
[+] Building 2.0s (15/28)
 => [internal] load remote build context                                                                                          0.0s
 => copy /context /                                                                                                               0.1s
 => [internal] load metadata for docker.io/library/debian:bullseye-20210902-slim                                                  0.6s
 => [internal] load metadata for docker.io/hexpm/elixir:1.13.4-erlang-24.1.7-debian-bullseye-20210902-slim                        0.7s
 => [builder  1/18] FROM docker.io/hexpm/elixir:1.13.4-erlang-24.1.7-debian-bullseye-20210902-slim@sha256:38a5bc18ada41443c1f1cb  0.0s
 => [stage-1 1/6] FROM docker.io/library/debian:bullseye-20210902-slim@sha256:e3ed4be20c22a1358020358331d177aa2860632f25b21681d7  0.0s
 => CACHED [stage-1 2/6] RUN apt-get update -y && apt-get install -y libstdc++6 openssl libncurses5 locales   && apt-get clean &  0.0s
 => CACHED [stage-1 3/6] RUN sed -i '/en_US.UTF-8/s/^# //g' /etc/locale.gen && locale-gen                                         0.0s
 => CACHED [stage-1 4/6] WORKDIR /app                                                                                             0.0s
 => CACHED [stage-1 5/6] RUN chown nobody /app                                                                                    0.0s
 => CACHED [builder  2/18] RUN apt-get update -y && apt-get install -y build-essential git nodejs npm     && apt-get clean && rm  0.0s
 => CACHED [builder  3/18] WORKDIR /app                                                                                           0.0s
 => CACHED [builder  4/18] RUN mix local.hex --force &&     mix local.rebar --force                                               0.0s
 => CACHED [builder  5/18] COPY mix.exs mix.lock ./                                                                               0.0s
 => ERROR [builder  6/18] RUN mix deps.get --only prod                                                                            1.2s
------
 > [builder  6/18] RUN mix deps.get --only prod:
#15 1.073 Failed to check for new Hex version
#15 1.073 Failed to fetch record for mime from registry (using cache instead)
#15 1.138 {:failed_connect, [{:to_address, {'repo.hex.pm', 443}}, {:inet, [:inet], :closed}]}
#15 1.138 {:failed_connect, [{:to_address, {'repo.hex.pm', 443}}, {:inet, [:inet], :closed}]}
#15 1.139 Failed to fetch record for ecto_sql from registry (using cache instead)
#15 1.139 Failed to fetch record for phoenix_html from registry (using cache instead)
#15 1.139 Failed to fetch record for parse_trans from registry (using cache instead)
#15 1.139 Failed to fetch record for plug_cowboy from registry (using cache instead)
#15 1.139 {:failed_connect, [{:to_address, {'repo.hex.pm', 443}}, {:inet, [:inet], :closed}]}
#15 1.139 {:failed_connect, [{:to_address, {'repo.hex.pm', 443}}, {:inet, [:inet], :closed}]}
#15 1.139 {:failed_connect, [{:to_address, {'repo.hex.pm', 443}}, {:inet, [:inet], :closed}]}
#15 1.139 {:failed_connect, [{:to_address, {'repo.hex.pm', 443}}, {:inet, [:inet], :closed}]}
#15 1.139 Failed to fetch record for db_connection from registry (using cache instead)
#15 1.139 {:failed_connect, [{:to_address, {'repo.hex.pm', 443}}, {:inet, [:inet], :closed}]}
#15 1.139 Failed to fetch record for telemetry_poller from registry (using cache instead)
#15 1.139 {:failed_connect, [{:to_address, {'repo.hex.pm', 443}}, {:inet, [:inet], :closed}]}
#15 1.139 Failed to fetch record for cowboy_telemetry from registry (using cache instead)
#15 1.139 {:failed_connect, [{:to_address, {'repo.hex.pm', 443}}, {:inet, [:inet], :closed}]}
#15 1.139 Failed to fetch record for idna from registry (using cache instead)
#15 1.139 {:failed_connect, [{:to_address, {'repo.hex.pm', 443}}, {:inet, [:inet], :closed}]}
#15 1.139 Failed to fetch record for mimerl from registry (using cache instead)
#15 1.140 {:failed_connect, [{:to_address, {'repo.hex.pm', 443}}, {:inet, [:inet], :closed}]}
#15 1.140 Failed to fetch record for cors_plug from registry (using cache instead)
#15 1.140 {:failed_connect, [{:to_address, {'repo.hex.pm', 443}}, {:inet, [:inet], :closed}]}
#15 1.140 Failed to fetch record for plug_crypto from registry (using cache instead)
#15 1.140 Failed to fetch record for certifi from registry (using cache instead)
#15 1.140 {:failed_connect, [{:to_address, {'repo.hex.pm', 443}}, {:inet, [:inet], :closed}]}
#15 1.140 {:failed_connect, [{:to_address, {'repo.hex.pm', 443}}, {:inet, [:inet], :closed}]}
#15 1.140 Failed to fetch record for expo from registry (using cache instead)
#15 1.140 {:failed_connect, [{:to_address, {'repo.hex.pm', 443}}, {:inet, [:inet], :closed}]}
#15 1.140 {:failed_connect, [{:to_address, {'repo.hex.pm', 443}}, {:inet, [:inet], :closed}]}
#15 1.140 {:failed_connect, [{:to_address, {'repo.hex.pm', 443}}, {:inet, [:inet], :closed}]}
#15 1.140 Failed to fetch record for aws from registry (using cache instead)
#15 1.140 Failed to fetch record for phoenix_live_view from registry (using cache instead)
#15 1.140 Failed to fetch record for decimal from registry (using cache instead)
#15 1.140 {:failed_connect, [{:to_address, {'repo.hex.pm', 443}}, {:inet, [:inet], :closed}]}
#15 1.149 Failed to fetch record for ssl_verify_fun from registry (using cache instead)
#15 1.149 {:failed_connect, [{:to_address, {'repo.hex.pm', 443}}, {:inet, [:inet], :closed}]}
#15 1.149 Failed to fetch record for phoenix_live_reload from registry (using cache instead)
#15 1.150 {:failed_connect, [{:to_address, {'repo.hex.pm', 443}}, {:inet, [:inet], :closed}]}
#15 1.150 Failed to fetch record for telemetry_metrics from registry (using cache instead)
#15 1.151 {:failed_connect, [{:to_address, {'repo.hex.pm', 443}}, {:inet, [:inet], :closed}]}
#15 1.179 ** (Mix) Unknown package mime in lockfile
------
Error failed to fetch an image or build from source: error building: executor failed running [/bin/sh -c mix deps.get --only $MIX_ENV]: exit code: 1

I’ve run into the same issue with RUN mix local.hex --force using a remote builder in regions LHR. I’m on an M1 Mac so can’t build locally. Here’s my console output:

 => ERROR [builder  4/14] RUN mix local.hex --force &&     mix local.rebar --force                                                                                            1.1s 
------                                                                                                                                                                             
 > [builder  4/14] RUN mix local.hex --force &&     mix local.rebar --force:                                                                                                       
#13 1.116 ** (Mix) httpc request failed with: {:failed_connect, [{:to_address, {'repo.hex.pm', 443}}, {:inet, [:inet], :closed}]}                                                  
#13 1.116                                                                                                                                                                          
#13 1.116 Could not install Hex because Mix could not download metadata at https://repo.hex.pm/installs/hex-1.x.csv.signed.                                                        
#13 1.116                                                                                                                                                                          
#13 1.116 Alternatively, you can compile and install Hex directly with this command:                                                                                               
#13 1.116 
#13 1.116     mix archive.install github hexpm/hex branch latest
#13 1.116 
------
Error failed to fetch an image or build from source: error building: executor failed running [/bin/sh -c mix local.hex --force &&     mix local.rebar --force]: exit code: 1

I just posted the same issue :joy:

Seems to be trouble with fly.io being blocked by external sources?

My first fail today was #22 0.895 09:40:27.635 [debug] Downloading tailwind from https://github.com/tailwindlabs/tailwindcss/releases/download/v3.2.4/tailwindcss-linux-x64 which at some point stopped succeeding.

I just retried with a new remote builder (I deleted the old one) and it built and deployed successfully.

Works now it seems.

Now, I have errors connections when deploying all my apps, … a nightmare today:

(DBConnection.ConnectionError) connection not available and request was dropped from queue

I just dispensed with the remote builder and built locally with fly deploy --local-only - all works