I have a new phoenix app I want to deploy. I’m getting an error where (it looks like…) docker build is hanging on a build step and eventually a timeout occurs. The Dockerfile is correctly building locally using docker build --no-cache ., so I would expect it to work when launching / deploying, but it is not.
Update: seems like maybe an issue with the debian version: bullseye-20220801-slim.
I created a brand new phoenix app and didn’t change anything other than the DEBIAN_VERSION to bullseye-20220801-slim in the Dockerfile. Deploying the generated phoenix app is now hanging with same output as above.
Hey @Mark, thank you for your reply I appreciate it!
Yes it builds locally without errors when running docker build (either with or without the --no-cache flag). However, after trying the method you suggested, it fails with a segfault:
=> ERROR [builder 4/17] RUN mix local.hex --force && mix local.rebar --force 0.6s
------
> [builder 4/17] RUN mix local.hex --force && mix local.rebar --force:
#13 0.606 qemu: uncaught target signal 11 (Segmentation fault) - core dumped
#13 0.616 Segmentation fault
------
Error failed to fetch an image or build from source: error building: executor failed running [/bin/sh -c mix local.hex --force && mix local.rebar --force]: exit code: 139
The Fly Elixir guide mentions this as an M1 issue and recommends using the --remote-only option which brings me back to the original issue in this post. I’m not sure what the fly CLI is doing differently from my docker build command. I briefly took a look at the flyctl source but nothing immediately stood out in the time I had to debug.
In either case, it seems I’m currently unable to deploy this using local or remote builds on a fresh phoenix app (v1.6.11). If I have more time I can try downgrading the base image versions since the 1.14.0 images are new.
I went back to your gist with the Dockerfile. I took that as a starting point and, before using it, tweaked it according to other known good Dockerfiles of mine. I was able to deploy using the remote builders. Here’s my working Dockerfile. Keep in mind, my app doesn’t use anything of the Rust tooling, so that might be an issue on your end.
# Find eligible builder and runner images on Docker Hub. We use Ubuntu/Debian instead of
# Alpine to avoid DNS resolution issues in production.
#
# https://hub.docker.com/r/hexpm/elixir/tags?page=1&name=ubuntu
# https://hub.docker.com/_/ubuntu?tab=tags
#
#
# This file is based on these images:
#
# - https://hub.docker.com/r/hexpm/elixir/tags - for the build image
# - https://hub.docker.com/_/debian?tab=tags&page=1&name=bullseye-20210902-slim - for the release image
# - https://pkgs.org/ - resource for finding needed packages
# - Ex: hexpm/elixir:1.14.0-erlang-25.0.4-debian-bullseye-20210902-slim
#
ARG ELIXIR_VERSION=1.14.0
ARG OTP_VERSION=25.0.4
ARG DEBIAN_VERSION=bullseye-20220801-slim
ARG BUILDER_IMAGE="hexpm/elixir:${ELIXIR_VERSION}-erlang-${OTP_VERSION}-debian-${DEBIAN_VERSION}"
ARG RUNNER_IMAGE="debian:${DEBIAN_VERSION}"
FROM ${BUILDER_IMAGE} as builder
# install build dependencies
# RUN apt-get update -y && apt-get install -y build-essential git \
# && apt-get clean && rm -f /var/lib/apt/lists/*_*
# install build dependencies (including curl and node)
# https://github.com/nodesource/distributions/blob/master/README.md#debmanual
RUN apt-get update -y && apt-get install -y build-essential git curl gcc g++ make \
&& curl -fsSL https://deb.nodesource.com/setup_18.x | bash - \
&& apt-get install -y nodejs \
&& rm -f /var/lib/apt/lists/*_* \
&& apt-get clean
# # Install curl which is used to install rust and node
# RUN apt-get update -y && apt-get install -y curl
# Install Rust which is needed to compile NIFs written in rust (like the siwe elixir lib)
RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- --default-toolchain stable --profile minimal --target x86_64-unknown-linux-gnu -y
# Add .cargo/bin (Rust) to PATH
ENV PATH="/root/.cargo/bin:${PATH}"
# # Install Node/npm which is needed for installing node dependencies
# RUN curl -fsSL https://deb.nodesource.com/setup_18.x | bash - \
# && apt-get install -y nodejs
# prepare build dir
WORKDIR /app
# install hex + rebar
RUN mix local.hex --force && \
mix local.rebar --force
# set build ENV
ENV MIX_ENV="prod"
# install mix dependencies
COPY mix.exs mix.lock ./
RUN mix deps.get --only $MIX_ENV
RUN mkdir config
# copy compile-time config files before we compile dependencies
# to ensure any relevant config change will trigger the dependencies
# to be re-compiled.
COPY config/config.exs config/${MIX_ENV}.exs config/
RUN mix deps.compile
COPY priv priv
COPY lib lib
COPY assets assets
# install node packagesbuild assets
RUN npm --prefix ./assets ci --progress=false --no-audit --loglevel=error
# # Install JS dependencies
# WORKDIR /app/assets
# RUN npm ci
# WORKDIR /app
# compile assets
RUN mix assets.deploy
# Compile the release
RUN mix compile
# Changes to config/runtime.exs don't require recompiling the code
COPY config/runtime.exs config/
COPY rel rel
RUN mix release
# start a new build stage so that the final image will only contain
# the compiled release and other runtime necessities
FROM ${RUNNER_IMAGE}
RUN apt-get update -y && apt-get install -y libstdc++6 openssl libncurses5 locales \
&& apt-get clean && rm -f /var/lib/apt/lists/*_*
# Set the locale
RUN sed -i '/en_US.UTF-8/s/^# //g' /etc/locale.gen && locale-gen
ENV LANG en_US.UTF-8
ENV LANGUAGE en_US:en
ENV LC_ALL en_US.UTF-8
WORKDIR "/app"
RUN chown nobody /app
# set runner ENV
ENV MIX_ENV="prod"
# Only copy the final release from the build stage
COPY --from=builder --chown=nobody:root /app/_build/${MIX_ENV}/rel/siwe_example ./
USER nobody
CMD ["/app/bin/server"]
# Appended by flyctl
ENV ECTO_IPV6 true
ENV ERL_AFLAGS "-proto_dist inet6_tcp"
# # Appended by flyctl
# ENV ECTO_IPV6 true
# ENV ERL_AFLAGS "-proto_dist inet6_tcp"
# # Appended by flyctl
# ENV ECTO_IPV6 true
# ENV ERL_AFLAGS "-proto_dist inet6_tcp"
# # Appended by flyctl
# ENV ECTO_IPV6 true
# ENV ERL_AFLAGS "-proto_dist inet6_tcp"
# # Appended by flyctl
# ENV ECTO_IPV6 true
# ENV ERL_AFLAGS "-proto_dist inet6_tcp"
I left your original parts but commented them out so you can see.
Currently having the same issue deploying with github actions.
deploy_to_dev:
name: Deploy to dev
needs: [dependencies, static_code_analysis, unit_tests]
runs-on: ubuntu-latest
timeout-minutes: 25
environment: dev
env:
TARGET: dev
REPO_NAME: ${{ github.event.repository.name }}
steps:
- uses: actions/checkout@v3
- uses: superfly/flyctl-actions/setup-flyctl@master
- run: flyctl deploy --dockerfile ./Dockerfile --remote-only
Maybe something with the signatures? This guy keeps popping up and never gets downloaded:
#10 [builder 1/19] FROM docker.io/hexpm/elixir:1.14.0-erlang-25.0.4-debian-bullseye-20220801-slim@sha256:cf8adcc00f80ae2d7ac417a62fe8138a3f8bf1b4e55d2a60f94b80663440c440
What was missing was updating the github action that setups up the BEAM, particularly the uses commit hash for the correct version that supports OTP 25.
- uses: actions/checkout@v3
- name: Set up Elixir
uses: erlef/setup-beam@69fb25ba7e2c546dfc085871ecf2fc5382c9843d <----- THIS
with:
elixir-version: 1.14.0
otp-version: 25.0.4
fetch-depth: 0
Hey @Mark, thanks again for your help here, much appreciated.
To be clear, I also cannot get a fresh phoenix app to build, that is, without rust and NPM and any special elixir dependencies. If I run mix phx.new and then try to fly launch (after changing to the correct base images since the generated Dockerfile contains broken links), it does not work. This is true when using the local-only and remote-only options. docker build works.
I did try the Dockerfile you sent me as well in my existing project, and that too still hangs with the original issue in the top of this post.
One thing I’m wondering about is if you are using fly launch to deploy? That’s what is sounds like. Given that the initial fly launch failed, but it completed the primary tasks of registering the fly app in the system, generating the Elixir release commands and the Dockerfile, then it’s purpose is done.
At that point, you should just use fly deploy. The fly launch command is also appending ENV settings to the end of the Dockerfile. I saw several copies of those settings in your original Dockerfile file and I deleted them.
Also, try this command: fly apps list. Then, delete any builder apps you see. They are named like fly-builder-.... At this point I’m just guessing, but perhaps the builder has a problem.
You delete using the fly apps destroy fly-builder-app-name-123.
From the builder builder logs, it sounds like it failed trying to fetch the docker image having tried for 0s.
You can also try the fly doctor command to see if any issues can be spotted.
@Mark Thanks for reply. So I have been deleting all the apps and trying to start again from scratch, but still no luck. This is reproducible on my machine by destroying all fly apps, running mix phx.new and then fly launch. I’ve tried in ORD and San Jose regions.
The one thing that is consistent is it’s hanging on this exact same docker layer each time on the remote builds:
This is true whether or not it’s on the first go using fly launch or on subsequent attempts with fly deploy. I did destroy the remote build app in between launch/deploy attempts. When I use the local-only option, it doesn’t hang on that layer, but it does segfault at the end of the build because of the M1 issue.
The only next step I see is to try to understand exactly what that layer is referring to / why it’s hanging. I could also do this on another machine but I was evaluating fly as a potential new hosting for my team who all have M1 macs, so hoping to get past any issues that may be related to that.
That’s a strange issue. M1 user here, and my only problem was when deploying from GHA (and that was because the action that sets erlang and elixir up was not the latest version (which prevented it from setting up erlang). What is the layer it’s hanging on? Mine was
#10 [builder 1/19] FROM docker.io/hexpm/elixir:1.14.0-erlang-25.0.4-debian-bullseye-20220801-slim@sha256:cf8adcc00f80ae2d7ac417a62fe8138a3f8bf1b4e55d2a60f94b80663440c440
