How to deploy a Phoenix Umbrella project?

Hello, I’m playing to see how deploy a Phoenix Umbrella on Fly platform. While deploy a non umbrella Elixir/Phoenix app is definitely trivial (just create a new project, fly launch it and you’re done), it seems that umbrella projects aren’t yet supported out of the box.

I’ve tried to deploy a basic umbrella using the same steps we have for non umbrella:

$ mix phx.new hello_elixir --umbrella  --no-ecto --install
$ cd hello_elixir_umbrella
$ export SECRET_KEY_BASE=$(mix phx.gen.secret)

but the builder complains it needs an explicit elixir_version in a elixir_buildpack.config file:

$ fly launch
Downloading from https://cnb-shim.herokuapp.com/v1/hashnuke/elixir
1.89 MB/-1 B
===> DETECTING
hashnuke/elixir 0.1
===> ANALYZING
Previous image with name "registry.fly.io/hello-elixir-umbrella:cache" not found
===> RESTORING
===> BUILDING
-----> Will export the following config vars:
       * MIX_ENV=prod
-----> Checking Erlang and Elixir versions
       WARNING: elixir_buildpack.config wasn't found in the app
       Using default config from Elixir buildpack

       IMPORTANT: The default elixir_version will be removed on 2021-06-01. Please explicitly set an elixir_version in your elixir_buildpack.config before then or your deploys will fail.

       Will use the following versions:
       * Stack heroku-20
       * Erlang 20.1
       * Elixir v1.5.3
-----> Stack changed, will rebuild
-----> Fetching Erlang 20.1 from https://repo.hex.pm/builds/otp/ubuntu-20.04/OTP-20.1.tar.gz
-----> Installing Erlang 20.1 (changed)

-----> Fetching Elixir v1.5.3 for OTP 20 from https://repo.hex.pm/builds/elixir/v1.5.3-otp-20.zip
-----> Installing Elixir v1.5.3 (changed)
-----> Installing Hex
** (Mix.Config.LoadError) could not load config config/config.exs
    ** (CompileError) config/config.exs:10: module Config is not loaded and could not be found
    (elixir) lib/code.ex:176: Code.eval_string/3
    (mix) lib/mix/config.ex:180: Mix.Config.read!/2
[...]

ok, so lets add the required elixir_buildpack.config file:

$ cat <<EOF>./elixir_buildpack.config
elixir_version=1.13.3
erlang_version=24.0
EOF

and try to deploy again. Now the builder goes through more steps than before until it stops with v0 failed - Failed due to unhealthy allocations - no stable job version to auto revert to and deploying as v1 error:
(log stripped for readability)

$ fly deploy
==> Verifying app config
==> Building image
==> Building image with Buildpacks
===> DETECTING
===> ANALYZING
===> RESTORING
===> BUILDING
==> gettext
===> Failed to restore /workspace/deps/ranch/_build/prod/lib/.rebar3/rebar_compiler_erl/source.dag file. Discarding it.
===> Analyzing applications...
===> Compiling ranch
===> Failed to restore /workspace/deps/telemetry/_build/prod/lib/.rebar3/rebar_compiler_erl/source.dag file. Discarding it.
===> Analyzing applications...
===> Compiling telemetry
==> telemetry_metrics
Compiling 7 files (.ex)
Generated telemetry_metrics app
===> Failed to restore /workspace/deps/telemetry_poller/_build/prod/lib/.rebar3/rebar_compiler_erl/source.dag file. Discarding it.
===> Analyzing applications...
===> Compiling telemetry_poller
==> jason
==> castore
==> esbuild
==> phoenix_pubsub
===> Failed to restore /workspace/deps/cowlib/_build/prod/lib/.rebar3/rebar_compiler_erl/source.dag file. Discarding it.
===> Analyzing applications...
===> Compiling cowlib
===> Failed to restore /workspace/deps/cowboy/_build/prod/lib/.rebar3/rebar_compiler_erl/source.dag file. Discarding it.
===> Analyzing applications...
===> Compiling cowboy
===> Failed to restore /workspace/deps/cowboy_telemetry/_build/prod/lib/.rebar3/rebar_compiler_erl/source.dag file. Discarding it.
===> Analyzing applications...
===> Compiling cowboy_telemetry
==> mime
==> plug_crypto
==> plug
==> phoenix_html
==> phoenix_view
==> plug_cowboy
==> phoenix
==> phoenix_live_view
==> phoenix_live_dashboard
==> swoosh
==> hello_elixir
==> hello_elixir_web
===> EXPORTING
==> Pushing image to fly
==> Monitoring deployment
 1 desired, 1 placed, 0 healthy, 1 unhealthy [restarts: 2] [health checks: 1 total, 1 critical]
Failed Instances

Failure #1

Instance
ID          PROCESS VERSION REGION  DESIRED STATUS  HEALTH CHECKS       RESTARTS    CREATED
64002145            0       fra     run     running 1 total, 1 critical 2           32s ago

Recent Events
TIMESTAMP               TYPE        MESSAGE
2022-03-26T09:35:51Z    Received    Task received by client
2022-03-26T09:35:51Z    Task Setup  Building Task Directory
2022-03-26T09:36:02Z    Started     Task started by client
2022-03-26T09:36:06Z    Terminated  Exit Code: 1
2022-03-26T09:36:06Z    Restarting  Task restarting in 1.127253406s
2022-03-26T09:36:12Z    Started     Task started by client
2022-03-26T09:36:16Z    Terminated  Exit Code: 1
2022-03-26T09:36:16Z    Restarting  Task restarting in 1.243652422s
2022-03-26T09:36:24Z    Started     Task started by client

2022-03-26T09:36:14Z   [info]    (stdlib 3.15) erl_eval.erl:123: :erl_eval.exprs/5
2022-03-26T09:36:14Z   [info]    (elixir 1.13.3) lib/code.ex:404: Code.validated_eval_string/3
2022-03-26T09:36:14Z   [info]    (elixir 1.13.3) lib/config.ex:260: Config.__eval__!/3
2022-03-26T09:36:14Z   [info]    (elixir 1.13.3) lib/config/reader.ex:92: Config.Reader.read!/2
2022-03-26T09:36:15Z   [info]Stin child exited normally with code: 1
2022-03-26T09:36:15Z   [info]Starting clean up.
2022-03-26T09:36:22Z   [info]Starting instance
2022-03-26T09:36:22Z   [info]Configuring virtual machine
2022-03-26T09:36:22Z   [info]Pulling container image
2022-03-26T09:36:22Z   [info]Unpacking image
2022-03-26T09:36:22Z   [info]Preparing kernel init
2022-03-26T09:36:23Z   [info]Configuring firecracker
2022-03-26T09:36:24Z   [info]Starting virtual machine
2022-03-26T09:36:24Z   [info]Starting init (commit: 6f9865f)...
2022-03-26T09:36:24Z   [info]Preparing to run: `/cnb/process/web` as heroku
2022-03-26T09:36:24Z   [info]2022/03/26 09:36:24 listening on [fdaa:0:547a:a7b:66:6400:2145:2]:22 (DNS: [fdaa::3]:53)
2022-03-26T09:36:25Z   [info]==> hello_elixir
2022-03-26T09:36:25Z   [info]Generated hello_elixir app
2022-03-26T09:36:25Z   [info]==> hello_elixir_web
2022-03-26T09:36:25Z   [info]Generated hello_elixir_web app
2022-03-26T09:36:26Z   [info]** (RuntimeError) environment variable SECRET_KEY_BASE is missing.
2022-03-26T09:36:26Z   [info]You can generate one by calling: mix phx.gen.secret
2022-03-26T09:36:26Z   [info]    (stdlib 3.15) erl_eval.erl:685: :erl_eval.do_apply/6
2022-03-26T09:36:26Z   [info]    (stdlib 3.15) erl_eval.erl:446: :erl_eval.expr/5
2022-03-26T09:36:26Z   [info]    (stdlib 3.15) erl_eval.erl:123: :erl_eval.exprs/5
2022-03-26T09:36:26Z   [info]    (elixir 1.13.3) lib/code.ex:404: Code.validated_eval_string/3
2022-03-26T09:36:26Z   [info]    (elixir 1.13.3) lib/config.ex:260: Config.__eval__!/3
2022-03-26T09:36:26Z   [info]    (elixir 1.13.3) lib/config/reader.ex:92: Config.Reader.read!/2
2022-03-26T09:36:27Z   [info]Main child exited normally with code: 1
2022-03-26T09:36:27Z   [info]Starting clean up.
--> v0 failed - Failed due to unhealthy allocations - no stable job version to auto revert to and deploying as v1

--> Troubleshooting guide at https://fly.io/docs/getting-started/troubleshooting/
Error abort

I’ve also noticed that fly deploy miss to add a default Dockerfile to the project, so I’ve tried to add one for myself, but getting the very same errors as above.

I understand that probably the builder isn’t yet ready to automatically manage umbrella projects, but in the meantime what is the canonical way to manually deal with umbrella’s on Fly platform? I’ve tried to search on this forum without luck and strangely neither googling around I’ve seen any information on this specific subject. Maybe I’m missing something obvious?

What version of Phoenix is this? For an umbrella app on Phoenix before 1.6.3, you may need to follow this guide to get it going: Deploy an Elixir Phoenix Application (pre v1.6.3)

The fly.toml includes a [builder] section that is overriding your Dockerfile in favor of a buildpack. If you remove that section, it’ll use your Dockerfile.

Hello Kurt,

For an umbrella app on Phoenix before 1.6.3, you may need to follow

yes I’m aware of that, but I’m using the last version of Phoenix:

$ mix phx.new --version
Phoenix installer v1.6.6

$ elixir --version
Erlang/OTP 24 [erts-12.3] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1] [jit] [dtrace]
Elixir 1.13.3 (compiled with Erlang/OTP 24)

The fly.toml includes a [builder] section that is overriding your Dockerfile in favor of a buildpack. If you remove that section, it’ll use your Dockerfile.

actually I don’t need to use mine, I’ve added one just to try to overcome the problem

After some attempts I’ve succeeded in deploying the umbrella. Unlike a plain Phoenix app, the deployment of an Umbrella application with the current builder needs some more steps. Hope this explanation could be useful to someone else Fly newbie struggling with umbrellas:

Plain App

A plain Phoenix app need just these four steps to be deployed successfully:

$ mix phx.new hello  --no-ecto --install
$ cd hello
$ export SECRET_KEY_BASE=$(mix phx.gen.secret)
$ fly launch

Umbrella App

Instead, the following are all the exact steps needed today to deploy a minimal umbrella application

  1. Setup the project:
$ mix phx.new hello  --umbrella --no-ecto --install
$ cd hello_umbrella
  1. enable server in config/runtime.exs:
config :hello_web, HelloWeb.Endpoint, server: true
  1. adds elixir_buildpack.config with the preferred Ex/Ph versions:
$ cat <<EOF>./elixir_buildpack.config
elixir_version=1.13.3
erlang_version=24.0
EOF
  1. deploy the assets from within the web app directory:
$ cd apps/hello_web
$ MIX_ENV=prod mix assets.deploy
$ cd ../../
  1. Setup the Fly’s stuff:
$ fly launch
# answer "no" to deploy request and leave fly launch, then:
  1. set up the secret:
$ export SECRET_KEY_BASE=$(mix phx.gen.secret)
$ fly secrets set SECRET_KEY_BASE=$SECRET_KEY_BASE
  1. deploy
$ fly deploy
  1. enjoy your app!
$ fly open

and the umbrella application is finally happily up and running!

Besides the steps above, here are the summary of the current differences in how Fly handles deployment between plain and umbrella.

to deploy an Umbrella via Fly:

  • you have to explicitly set server: true in config/runtime.exs
  • you have to add the required elixir_buildpack.config file to the project
  • you have to build the application assets
  • you have to set the SECRET_KEY_BASE executing fly secrets after fly launch and before fly deploy (you can’t just export it to an environment variable)
  • the Fly builder will not generate a Docker file as is the case with plain apps (but if you supply one, the builder will use it)