Brand new Phoenix app failed due to unhealthy allocations

Updated flyctl, running again. Will report here.

@kurt, thanks for the reply.

After updating, I deleted everything, and created a brand new Phoenix project.

I still got this message on the cli after running the command fly launch

WARN The running flyctl agent (v0.0.335) is older than the current flyctl (v0.0.336).
WARN The out-of-date agent will be shut down along with existing wireguard connections. The new agent will start automatically as needed.

(...)

We recommend upgrading to Phoenix 1.6.3 which includes a release configuration for Docker-based deployment.

If you do upgrade, you can run 'fly launch' again to get the required deployment setup.

If you don't want to uprade, you'll need to add a few files and configuration options manually.
We've placed a Dockerfile compatible with other Phoenix 1.6 apps in this directory. See
https://hexdocs.pm/phoenix/fly.html for details, including instructions for setting up
a Postgresql database.

It hanged for minutes and failed.

Same error from before (check reply #8).

Here’s the full output from the terminal:

❯ fly launch
An existing fly.toml file was found for app dark-night-4938
App is not running, deploy...
Deploying dark-night-4938
==> Validating app configuration
--> Validating app configuration done
Services
TCP 80/443 ⇢ 8080
Remote builder fly-builder-broken-rain-7190 ready
==> Creating build context
--> Creating build context done
==> Building image with Docker
--> docker host: 20.10.12 linux x86_64
Sending build context to Docker daemon  40.39kB
[+] Building 92.1s (28/28) FINISHED
 => [internal] load remote build context                                                                                                                        0.0s
 => copy /context /                                                                                                                                             0.1s
 => [internal] load metadata for docker.io/library/debian:bullseye-20210902-slim                                                                                2.2s
 => [internal] load metadata for docker.io/hexpm/elixir:1.13.4-erlang-25.0.2-debian-bullseye-20210902-slim                                                      2.2s
 => [builder  1/17] FROM docker.io/hexpm/elixir:1.13.4-erlang-25.0.2-debian-bullseye-20210902-slim@sha256:9d9b1c244804e80da042719578670ad08018efc614bb266641f9  2.7s
 => => resolve docker.io/hexpm/elixir:1.13.4-erlang-25.0.2-debian-bullseye-20210902-slim@sha256:9d9b1c244804e80da042719578670ad08018efc614bb266641f96a288a898b  0.0s
 => => sha256:247ec2825166b6f151e5e8477bd79616e47200edccdebbefd1d8b5075a450200 5.06MB / 5.06MB                                                                  0.2s
 => => sha256:9d9b1c244804e80da042719578670ad08018efc614bb266641f96a288a898b1e 772B / 772B                                                                      0.0s
 => => sha256:3d33e8efd534a27054b5f21d1b6a12b1b0155f82d026cfa1396f5b2eb5643812 1.16kB / 1.16kB                                                                  0.0s
 => => sha256:0d8f1c16af737465560144aeb21cf0678f4ae382c52eaddbf9519039feac0efd 2.33kB / 2.33kB                                                                  0.0s
 => => sha256:f8416d8bac72cefc0ce17bd2dc0c03aa43e123d309db92ee23be9382192cf2ed 31.37MB / 31.37MB                                                                0.3s
 => => sha256:11193a2496952011f9e3047cbccd0566ee78fe5ab8ff446c6ea8107a5026e48e 16.23MB / 16.23MB                                                                0.3s
 => => sha256:0245b0f09800ea694d5cfba629bd93700e51b0663e8b4f36daada1c9ee9fa7a7 44.38MB / 44.38MB                                                                0.6s
 => => extracting sha256:f8416d8bac72cefc0ce17bd2dc0c03aa43e123d309db92ee23be9382192cf2ed                                                                      89.4s
 => => extracting sha256:11193a2496952011f9e3047cbccd0566ee78fe5ab8ff446c6ea8107a5026e48e                                                                       0.2s
 => => extracting sha256:0245b0f09800ea694d5cfba629bd93700e51b0663e8b4f36daada1c9ee9fa7a7                                                                       0.8s
 => => extracting sha256:247ec2825166b6f151e5e8477bd79616e47200edccdebbefd1d8b5075a450200                                                                       0.1s
 => [stage-1 1/6] FROM docker.io/library/debian:bullseye-20210902-slim@sha256:e3ed4be20c22a1358020358331d177aa2860632f25b21681d79204ace20455a6                  1.4s
 => => resolve docker.io/library/debian:bullseye-20210902-slim@sha256:e3ed4be20c22a1358020358331d177aa2860632f25b21681d79204ace20455a6                          0.0s
 => => sha256:7beb0248fd8163ea6effdf58af3e836c47f2f97d7e44e5109c95333c92190a48 1.46kB / 1.46kB                                                                  0.0s
 => => sha256:f8416d8bac72cefc0ce17bd2dc0c03aa43e123d309db92ee23be9382192cf2ed 31.37MB / 31.37MB                                                                0.3s
 => => sha256:e3ed4be20c22a1358020358331d177aa2860632f25b21681d79204ace20455a6 1.85kB / 1.85kB                                                                  0.0s
 => => sha256:8aa2e47f9a6cf001ecf3ad0f8439e1005743a3024b98e7bbf023ace55afea903 529B / 529B                                                                      0.0s
 => => extracting sha256:f8416d8bac72cefc0ce17bd2dc0c03aa43e123d309db92ee23be9382192cf2ed                                                                       1.0s
 => [stage-1 2/6] RUN apt-get update -y && apt-get install -y libstdc++6 openssl libncurses5 locales   && apt-get clean && rm -f /var/lib/apt/lists/*_*        12.8s
 => [builder  2/17] RUN apt-get update -y && apt-get install -y build-essential git     && apt-get clean && rm -f /var/lib/apt/lists/*_*                       25.0s
 => [stage-1 3/6] RUN sed -i '/en_US.UTF-8/s/^# //g' /etc/locale.gen && locale-gen                                                                              1.6s
 => [stage-1 4/6] WORKDIR /app                                                                                                                                  0.0s
 => [stage-1 5/6] RUN chown nobody /app                                                                                                                         0.3s
 => [builder  3/17] WORKDIR /app                                                                                                                                0.0s
 => [builder  4/17] RUN mix local.hex --force &&     mix local.rebar --force                                                                                    8.4s
 => [builder  5/17] COPY mix.exs mix.lock ./                                                                                                                    0.0s
 => [builder  6/17] RUN mix deps.get --only prod                                                                                                                3.3s
 => [builder  7/17] RUN mkdir config                                                                                                                            0.3s
 => [builder  8/17] COPY config/config.exs config/prod.exs config/                                                                                              0.0s
 => [builder  9/17] RUN mix deps.compile                                                                                                                       44.2s
 => [builder 10/17] COPY priv priv                                                                                                                              0.0s
 => [builder 11/17] COPY lib lib                                                                                                                                0.0s
 => [builder 12/17] COPY assets assets                                                                                                                          0.0s
 => [builder 13/17] RUN mix assets.deploy                                                                                                                       2.6s
 => [builder 14/17] RUN mix compile                                                                                                                             0.9s
 => [builder 15/17] COPY config/runtime.exs config/                                                                                                             0.0s
 => [builder 16/17] COPY rel rel                                                                                                                                0.0s
 => [builder 17/17] RUN mix release                                                                                                                             1.9s
 => [stage-1 6/6] COPY --from=builder --chown=nobody:root /app/_build/prod/rel/marmita ./                                                                       0.2s
 => exporting to image                                                                                                                                          0.2s
 => => exporting layers                                                                                                                                         0.2s
 => => writing image sha256:8a39a7584f495058a34fd98ae49e369392f8883e80a1dafd8d14dfbbf2e67f12                                                                    0.0s
 => => naming to registry.fly.io/dark-night-4938:deployment-1656194350                                                                                          0.0s
--> Building image done
==> Pushing image to fly
The push refers to repository [registry.fly.io/dark-night-4938]
dec147c11a3c: Pushed
e67a3a6b5673: Pushed
12d7ac3da410: Pushed
dbea167f9f4b: Pushed
178a2602f6bd: Pushed
ba5a5fe43301: Pushed
deployment-1656194350: digest: sha256:65ae649e874b2fdf52ffd362d7ff8b66e74d8fd7aa5fa14cc7b6e6ad889474af size: 1576
--> Pushing image done
Image: registry.fly.io/dark-night-4938:deployment-1656194350
Image size: 118 MB
==> Creating release
Release v2 created
Release command detected: this new release will not be available until the command succeeds.

You can detach the terminal anytime without stopping the deployment
==> Release command
Command: /app/bin/migrate
	 Starting instance
	 Configuring virtual machine
	 Pulling container image
	 Unpacking image
	 Preparing kernel init
	 Starting virtual machine
	 Preparing to run: `/app/bin/migrate` as nobody
	 2022/06/25 22:01:24 listening on [fdaa:0:71f1:a7b:1f60:6476:5de9:2]:22 (DNS: [fdaa::3]:53)
	 Main child exited normally with code: 0
	 Reaped child process with pid: 569 and signal: SIGUSR1, core dumped? false
	 Starting clean up.
Monitoring Deployment

1 desired, 1 placed, 0 healthy, 1 unhealthy [health checks: 1 total, 1 critical]
v0 failed - Failed due to unhealthy allocations - no stable job version to auto revert to
Failed Instances

==> Failure #1

Instance
  ID            = ab70b3a9
  Process       =
  Version       = 0
  Region        = gru
  Desired       = run
  Status        = running
  Health Checks = 1 total, 1 critical
  Restarts      = 0
  Created       = 4m57s ago

Recent Events
TIMESTAMP            TYPE       MESSAGE
2022-06-25T22:01:30Z Received   Task received by client
2022-06-25T22:01:30Z Task Setup Building Task Directory
2022-06-25T22:01:34Z Started    Task started by client

Recent Logs
2022-06-25T22:01:32.000 [info] Unpacking image
2022-06-25T22:01:32.000 [info] Preparing kernel init
2022-06-25T22:01:33.000 [info] Configuring firecracker
2022-06-25T22:01:34.000 [info] Starting virtual machine
2022-06-25T22:01:34.000 [info] Starting init (commit: e21acb3)...
2022-06-25T22:01:34.000 [info] Preparing to run: `/app/bin/server` as nobody
2022-06-25T22:01:34.000 [info] 2022/06/25 22:01:34 listening on [fdaa:0:71f1:a7b:1f60:ab70:b3a9:2]:22 (DNS: [fdaa::3]:53)
2022-06-25T22:01:35.000 [info] Reaped child process with pid: 551, exit code: 0
2022-06-25T22:01:36.000 [info] 22:01:36.421 [info] Running MarmitaWeb.Endpoint with cowboy 2.9.0 at :::4000 (http)
2022-06-25T22:01:36.000 [info] 22:01:36.423 [info] Access MarmitaWeb.Endpoint at https://dark-night-4938.fly.dev
2022-06-25T22:01:37.000 [info] Reaped child process with pid: 572 and signal: SIGUSR1, core dumped? false
***v0 failed - Failed due to unhealthy allocations - no stable job version to auto revert to and deploying as v1

Troubleshooting guide at https://fly.io/docs/getting-started/troubleshooting/
Error abort

I will kill the machine to stop using my credits.

I’m also having what I assume are similar problems deploying boilerplate:

% fly version
fly v0.0.336 darwin/amd64 Commit: d58c532 BuildDate: 2022-06-25T18:23:16Z
% mix phx.new -v
Phoenix installer v1.6.10

I generated a new application with mix phx.new example and downgraded it to Phoenix 1.6.9 in an effort to temporarily fix the afformentioned. That ephemeral application has the automatically generated name empty-fog-8029.

The only thing I changed from the generator was the Phoenix version in mix.exs, from ~> 1.6.10 to "== 1.6.9". It still hung.

Will you post the fly.toml we generated here?

Oh, interesting! It gave it a port of 8080; shouldn’t that be 4000 if an environment variable for PORT isn’t being set? Manually changing it to 4000 myself let the app launch, as here: https://empty-fog-8029.fly.dev/

# fly.toml file generated for empty-fog-8029 on 2022-06-25T19:09:20-04:00

app = "empty-fog-8029"
kill_signal = "SIGTERM"
kill_timeout = 5
processes = []

[deploy]
  release_command = "/app/bin/migrate"

[env]
  PHX_HOST = "empty-fog-8029.fly.dev"

[experimental]
  allowed_public_ports = []
  auto_rollback = true

[[services]]
  http_checks = []
  internal_port = 8080
  processes = ["app"]
  protocol = "tcp"
  script_checks = []
  [services.concurrency]
    hard_limit = 25
    soft_limit = 20
    type = "connections"

  [[services.ports]]
    force_https = true
    handlers = ["http"]
    port = 80

  [[services.ports]]
    handlers = ["tls", "http"]
    port = 443

  [[services.tcp_checks]]
    grace_period = "1s"
    interval = "15s"
    restart_limit = 0
    timeout = "2s"
1 Like

Ok, something else is up here. That “1.6.1” message is informational only. What’s breaking is the environment variable for PORT, which gets set here: flyctl/scan.go at master · superfly/flyctl · GitHub

I don’t understand why this isn’t included in the generated fly.toml. We’re doing some debugging to find out what’s up. For now you can either at the PORT env var, or you can change the internal_port to 4000.

FWIW, you don’t need to set it to "== 1.6.9" and lock the app to that version. The CLI is checking the version string in your deps, so ~> 1.6.9 will also pass its checks, and through optimistic versioning you will still get a 1.6.10 app if you run mix deps.update --all

fly.toml

# fly.toml file generated for your-app-name on 2022-06-23T21:05:23+07:00

app = "your-app-name"

kill_signal = "SIGTERM"
kill_timeout = 5
processes = []

[deploy]
  release_command = "/app/bin/migrate"

#[mounts]
#  source="my_data"
# destination="/app/data"

[env]
  PHX_HOST = "you-app-name"
  # PORT = "8080"

[experimental]
  allowed_public_ports = []
  auto_rollback = true

[[services]]
  http_checks = []
  # internal_port = 8080
  internal_port = 4000
  processes = ["app"]
  protocol = "tcp"
  script_checks = []

  [services.concurrency]
    hard_limit = 25
    soft_limit = 20
    type = "connections"

  [[services.ports]]
    force_https = true
    handlers = ["http"]
    port = 80

  [[services.ports]]
    handlers = ["tls", "http"]
    port = 443

  [[services.tcp_checks]]
    # grace_period = "1s"
    grace_period = "30s"
    interval = "15s"
    restart_limit = 0
    timeout = "2s"

comment out PORT in [env]
change internal_port to your default port ( 4000) in this case
increase grace_period to 25s or 30s

1 Like

ooo, thank you. Well done. This toml guidance fixed it for me.

1 Like

I just want to be clear for anyone reading this in the future.

The only change you need to make is to change the internal_port value in the fly.toml file from 8080 to 4000.

The diff looks like this:

[[services]]
   ...
-  internal_port = 8080
+  internal_port = 4000

This change alone makes all that’s necessary configuration changes you need to get your app deployed after running fly launch using the fly CLI version 0.0.338

1 Like

We fixed in the latest fly CLI, so it now properly generates PORT 8080 env var. The internal port of 4000 will work because phoenix’s default config falls back, but I think exporting an PORT env is the most expected solution. Folks can fly version update and all should be good now for fly launch. Thanks!

2 Likes

I feel bad my original topic was closed, since the person (me) who started the issue did not mark as solved. And… I feel bad the original thread will be deleted since there is real feedback here to the onboarding and the previous issue.

I couldn’t post this reply early on this official support channel because I was limited to post again only in 14 hours. Also feels really bad since I had a real problem and this is the only channel for support.

On the bright side:

  • The feedback was fast and thanks @chrismccord for replying me after I pinged on Slack.
  • The CLI was patched over the weekend and Sunday night a second release was available.

Yet all the rest makes me feel sour. Some people even said “good luck having support on AWS”, but that is not the point: AWS is a cheaper and commodity. I feel like my onboarding was terrible and using the forum for support (and merging threads) was not an ideal way for an already paying customer.

I’ll refrain from recommending fly to companies until I feel an improve in these points. I still feel I was not well received and serviced. I hope this gets better.

If this is a community-led product… I really hope this feedback at least get some attention. It comes from someone that wants to see this service getting better.