Error when running `fly deploy` for generated phoenix app after `fly launch`

after running mix and fly launch successfully fly deploy fails. I’ve made no changes to the auto generated phoenix app.

Dockerfile builds locally, mix release runs fine, runtime has no issues since it’s a brand new app.

 > [builder  6/17] RUN mix deps.get --only prod:
#16 2.834
#16 2.834 16:47:40.292 [notice] Application hex exited: exited in: Hex.Application.start(:normal, [])
#16 2.834     ** (EXIT) an exception was raised:
#16 2.834         ** (MatchError) no match of right hand side value: {:map_reduce_ok, 3}
#16 2.834             (elixir 1.13.0) src/elixir_def.erl:102: :elixir_def.store_definition/5
#16 2.834             (mix 1.13.0) lib/mix/scm.ex:130: Mix.SCM.available/0
#16 2.834             (hex 1.0.1) lib/hex/application.ex:9: Hex.Application.start/2
#16 2.834             (kernel 8.1.3) application_master.erl:293: :application_master.start_it_old/4
#16 2.840
#16 2.840 16:47:40.357 [notice] Application inets exited: :stopped
#16 2.844
#16 2.844 16:47:40.361 [notice] Application ssl exited: :stopped
#16 2.844
#16 2.844 16:47:40.361 [notice] Application public_key exited: :stopped
#16 2.844
#16 2.844 16:47:40.361 [notice] Application asn1 exited: :stopped
#16 2.844
#16 2.844 16:47:40.361 [notice] Application crypto exited: :stopped
#16 2.850 Could not start Hex. Try fetching a new version with "mix local.hex" or uninstalling it with "mix archive.uninstall hex.ez"
#16 2.854 ** (MatchError) no match of right hand side value: {:error, {:hex, {:bad_return, {{Hex.Application, :start, [:normal, []]}, {:EXIT, {{:badmatch, {:map_reduce_ok, 3}}, [{:elixir_def, :store_definition, 5, [file: 'src/elixir_def.erl', line: 102]}, {Mix.SCM, :available, 0, [file: 'lib/mix/scm.ex', line: 130]}, {Hex.Application, :start, 2, [file: 'lib/hex/application.ex', line: 9]}, {:application_master, :start_it_old, 4, [file: 'application_master.erl', line: 293]}]}}}}}}
#16 2.854     (hex 1.0.1) lib/hex.ex:5: Hex.start/0
#16 2.854     (mix 1.13.0) lib/mix/hex.ex:59: Mix.Hex.start/0
#16 2.854     (mix 1.13.0) lib/mix/dep/loader.ex:194: Mix.Dep.Loader.with_scm_and_app/4
#16 2.854     (mix 1.13.0) lib/mix/dep/loader.ex:147: Mix.Dep.Loader.to_dep/3
#16 2.854     (elixir 1.13.0) lib/enum.ex:1593: Enum."-map/2-lists^map/1-0-"/2
#16 2.854     (mix 1.13.0) lib/mix/dep/loader.ex:364: Mix.Dep.Loader.mix_children/2
#16 2.854     (mix 1.13.0) lib/mix/dep/loader.ex:18: Mix.Dep.Loader.children/0
#16 2.854     (mix 1.13.0) lib/mix/dep/converger.ex:57: Mix.Dep.Converger.all/4
Error failed to fetch an image or build from source: error building: executor failed running [/bin/sh -c mix deps.get --only $MIX_ENV]: exit code: 1

Hello @user87!

I’m not sure what’s going on here. What versions of Elixir/Erlang are you using?

This looks to me like the hex library has a problem. Since that’s not likely due to your project, I’m looking at versions. This shows Elixir 1.13.0 in stack trace. Do the expected versions match up with the Dockerfile version being used?

Sorry… looking for more info.

The Dockerfile is the one that’s generated by fly launch and it does match from what I can tell.

# Find eligible builder and runner images on Docker Hub. We use Ubuntu/Debian instead of
# Alpine to avoid DNS resolution issues in production.
# This file is based on these images:
#   - - for the build image
#   - - for the release image
#   - - resource for finding needed packages
#   - Ex: hexpm/elixir:1.13.0-erlang-24.1.7-debian-bullseye-20210902-slim
ARG BUILDER_IMAGE="hexpm/elixir:1.13.0-erlang-24.1.7-debian-bullseye-20210902-slim"
ARG RUNNER_IMAGE="debian:bullseye-20210902-slim"

FROM ${BUILDER_IMAGE} as builder

# install build dependencies
RUN apt-get update -y && apt-get install -y build-essential git \
    && apt-get clean && rm -f /var/lib/apt/lists/*_*

# prepare build dir

# install hex + rebar
RUN mix local.hex --force && \
    mix local.rebar --force

# set build ENV
ENV MIX_ENV="prod"

# install mix dependencies
COPY mix.exs mix.lock ./
RUN mix deps.get --only $MIX_ENV
RUN mkdir config

# copy compile-time config files before we compile dependencies
# to ensure any relevant config change will trigger the dependencies
# to be re-compiled.
COPY config/config.exs config/${MIX_ENV}.exs config/
RUN mix deps.compile

COPY priv priv

# note: if your project uses a tool like,
# which customizes asset compilation based on what it finds in
# your Elixir templates, you will need to move the asset compilation
# step down so that `lib` is available.
COPY assets assets

# compile assets
RUN mix assets.deploy

# Compile the release
COPY lib lib

RUN mix compile

# Changes to config/runtime.exs don't require recompiling the code
COPY config/runtime.exs config/

COPY rel rel
RUN mix release

# start a new build stage so that the final image will only contain
# the compiled release and other runtime necessities

RUN apt-get update -y && apt-get install -y libstdc++6 openssl libncurses5 locales \
  && apt-get clean && rm -f /var/lib/apt/lists/*_*

# Set the locale
RUN sed -i '/en_US.UTF-8/s/^# //g' /etc/locale.gen && locale-gen


WORKDIR "/app"
RUN chown nobody /app

# Only copy the final release from the build stage
COPY --from=builder --chown=nobody:root /app/_build/prod/rel/test ./

USER nobody

CMD ["/app/bin/server"]

# Appended by flyctl
ENV ERL_AFLAGS "-proto_dist inet6_tcp"

What’s interesting is that the initial build when running fly launch works. As far as I can tell there’s no difference in the output when building the docker image between the two commands besides fly deploy failing

A post was split to a new topic: fly launch fails when creating postgres

Does it work when you run this command?

fly deploy --remote-only

The first fly launch command does a remote build. The second time you run fly deploy, it runs the docker build locally.

So, does it work when you explicitly use the remote builders?

Yeah, that worked.

Building the docker file using docker cli locally works which is also interesting.

1 Like

@brainlid Is there any information regarding this issue?

Can you try with DOCKER_BUILDKIT=1 fly deploy?

That also worked.

Might be worth mentioning I’m using and m1 mac. Not sure if that makes a difference.

We’ve seen issues with M1 and non-Buildkit builds. Buildkit is the newer, superior build system - the default in new Docker installs. Our remote builders also default to it.

We’re not sure why this particular issue happens on classic builds, but it’s safe to just use buildkit! You can set that environment variable in your shell and enjoy the benefits of buildkit everywhere.

1 Like

Will do thanks for your help