How to fix permission denied with Phoenix shell scripts?

Questions / Help
1

Hello,

Ran my first deployment yesterday and all good.

I’m trying to run a second deployment and this is resulting in a permission denied error for the Phoenix shell scripts that are generated by mix phx.gen.release (which worked fine on the first deployment).

No significant changes have been made to the repository in the failed deployment. The release works fine locally.

This is the error when Fly.io attempts to run the release command:

Starting instance
         Configuring virtual machine
         Pulling container image
         Unpacking image
         Preparing kernel init
         Configuring firecracker
         Starting virtual machine
         Starting init (commit: 0c50bff)...
         Preparing to run: `/app/bin/migrate` as nobody
         Error: UnhandledIoError(Os { code: 13, kind: PermissionDenied, message: "Permission denied" })
         [    0.116931] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000100
         [    0.118145] CPU: 0 PID: 1 Comm: init Not tainted 5.12.2 #1
         [    0.119001] Call Trace:
         [    0.119411]  dump_stack+0x63/0x7e
         [    0.119979]  panic+0xf3/0x2a1
         [    0.120427]  do_exit+0x967/0xb50
         [    0.120902]  ? __vm_munmap+0x96/0xd0
         [    0.121419]  do_group_exit+0x3e/0xa0
         [    0.121974]  __x64_sys_exit_group+0x13/0x20
         [    0.122580]  do_syscall_64+0x37/0x50
         [    0.123093]  entry_SYSCALL_64_after_hwframe+0x44/0xae
         [    0.123820] RIP: 0033:0x6f0675
         [    0.124262] Code: eb ef 48 8b 76 28 e9 76 05 00 00 64 48 8b 04 25 00 00 00 00 48 8b b0 b0 00 00 00 e9 af ff ff ff 48 63 ff b8 e7 00 00 00 0f 05 <ba> 3c 00 00 00 48 89 d0 0f 05 eb f9 66 2e 0f 1f 84 00 00 00 00 00
         [    0.126849] RSP: 002b:00007ffe45262ea8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e7
         [    0.127912] RAX: ffffffffffffffda RBX: 00000000005c38b0 RCX: 00000000006f0675
         [    0.129047] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
         [    0.130442] RBP: 0000000000000001 R08: 0000000000000000 R09: 0000000000000000
         [    0.131814] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffe45262f08
         [    0.133221] R13: 00007ffe45262f18 R14: 0000000000000000 R15: 0000000000000000
         [    0.134705] Kernel Offset: disabled
         [    0.135434] Rebooting in 1 seconds..

Commenting out the /app/bin/migrate/ release command from the fly.toml script moves the error to /app/bin/server (another Phoenix shell script), also with permission denied:

[info] /bin/sh: 1: /app/bin/server: Permission denied
[info] Main child exited normally with code: 126

Ultimately resulting in a failed due to unhealthy allocations error.

This is the Dockerfile (the default one from running fly launch):

ARG BUILDER_IMAGE="hexpm/elixir:1.13.1-erlang-24.2-debian-bullseye-20210902-slim"
ARG RUNNER_IMAGE="debian:bullseye-20210902-slim"

FROM ${BUILDER_IMAGE} as builder

# install build dependencies
RUN apt-get update -y && apt-get install -y build-essential git \
    && apt-get clean && rm -f /var/lib/apt/lists/*_*

# prepare build dir
WORKDIR /app

# install hex + rebar
RUN mix local.hex --force && \
    mix local.rebar --force

# set build ENV
ENV MIX_ENV="prod"

# install mix dependencies
COPY mix.exs mix.lock ./
RUN mix deps.get --only $MIX_ENV
RUN mkdir config

# copy compile-time config files before we compile dependencies
# to ensure any relevant config change will trigger the dependencies
# to be re-compiled.
COPY config/config.exs config/${MIX_ENV}.exs config/
RUN mix deps.compile

COPY priv priv

# note: if your project uses a tool like https://purgecss.com/,
# which customizes asset compilation based on what it finds in
# your Elixir templates, you will need to move the asset compilation
# step down so that `lib` is available.
COPY assets assets

# compile assets
RUN mix assets.deploy

# Compile the release
COPY lib lib

RUN mix compile

# Changes to config/runtime.exs don't require recompiling the code
COPY config/runtime.exs config/

COPY rel rel
RUN mix release

# start a new build stage so that the final image will only contain
# the compiled release and other runtime necessities
FROM ${RUNNER_IMAGE}

RUN apt-get update -y && apt-get install -y libstdc++6 openssl libncurses5 locales \
  && apt-get clean && rm -f /var/lib/apt/lists/*_*

# Set the locale
RUN sed -i '/en_US.UTF-8/s/^# //g' /etc/locale.gen && locale-gen

ENV LANG en_US.UTF-8
ENV LANGUAGE en_US:en
ENV LC_ALL en_US.UTF-8

WORKDIR "/app"
RUN chown nobody /app

# Only copy the final release from the build stage
COPY --from=builder --chown=nobody:root /app/_build/prod/rel/name_of_app ./

USER nobody

CMD /app/bin/server
# Appended by flyctl
ENV ECTO_IPV6 true
ENV ERL_AFLAGS "-proto_dist inet6_tcp"

SSH’ing into Fly confirms that the files under /app/* are indeed owned by nobody and set to 0755. Although these could be the files from the first deploy, either way, Dockerfile when it copies sets chown to nobody.

So I’m puzzled that these shell scripts cannot be run on the second attempt at deploying, especially if the permissions are set correctly, what could I be missing? Any help would be appreciated, thanks!

2

This is odd. Did you happen to attach a volume between first and second deploys?

3

Hi Chris, I have not attached any volumes. There is only the app and app-db instances (created by following Deploy an Elixir Phoenix Application, i.e. running fly launch on an existing Phoenix app).

I just checked fly volumes list too just in case and it’s empty.

Happy to provide any additional info that could help troubleshoot this, I know I could rewrite the Dockerfile to not to use the Phoenix scripts but this is a pretty vanilla deployment and I’m curious to figure out what went wrong on the second deploy to avoid this issue when deploying other apps.

4

Try adding this to your Dockerfile, right before the CMD line:

chmod +x /app/bin/*

That error can either mean the file is missing entirely or that the file isn’t marked as executable.

4 Likes
5

That worked, thanks!

Would it be safe to assume therefore that by default uploaded shell scripts are not marked as executable by Fly, except for the first deploy?

6

Well that’s the interesting question.

Phoenix generates executable scripts. And when they’re added to the Docker image, they keep their executable flag. If you’re using the same set of files each time the deploy will work great!

There are a lot of things that can make files lose their executable bit. If you copied them between hosts or pushed to git and cloned on a different OS, or if you’re using Windows? I can’t remember all the things I’ve seen here. That Dockerfile could just include that line, though, it won’t hurt anything!

7

Aha, that makes a lot of sense! Yes, I had indeed re-cloned in a transition from WSL1 to WSL2.

1 Like
8

I just encountered the problem when using the github action auto deployment.

Same error message

using {:phoenix, "~> 1.7.0-rc.2", override: true}

and the github action job at .github/workflows/fly.yml

jobs:
  deploy:
    name: Deploy app
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: superfly/flyctl-actions/setup-flyctl@master
        with:
          version: 0.0.450
      - run: flyctl deploy --remote-only --build-arg DATABASE_URL=$DATABASE_URL

and at the lowest part of the Dockerfile CMD ["chmod +x /app/bin/server"]

it still failed. wondering what am missing here. Even DATABASE_URL is provided for the DB migration.

I have tried with and without the chmod +x and still failed.

Note: