Release command never completes - deployment times out

Questions / Help Build debugging
,
1

I’m trying to deploy a Phoenix app on Fly.io with a PostgreSQL database, but I keep running into a problem during the release phase — the release command just never finishes and ends up timing out.

Here’s what I’ve got in my fly.toml:

[deploy]
  release_command = '/app/bin/migrate'

This is the error log:

INFO Preparing to run: `/app/bin/migrate` as nobody
INFO SSH listening listen_address=[...]
❌ release command failed - aborting deployment. timeout reached

and this is my runtime.ex

if config_env() == :prod do
  config :server, Server.Repo,
    url: System.get_env("DATABASE_URL"),
    ssl: true,
    pool_size: 10
end

I even tried increasing the timeout to 30 minutes using --release-command-timeout, but the command still hangs with no output and eventually fails.

Is there a way to debug what’s going wrong? Maybe I’m missing something obvious — this is my first time setting up deployment like this.

2

I’d start off with removing that line from your config, redeploying your site so it is operational, shell in, then run that command manually. Does it work then?

3

I tried that too, but when I run the command manually in the console, it just freezes and nothing happens.

4

Great, that helps narrow things down; we can assume this is why it won’t run automatically. Does this command have a --help switch, or maybe a --verbose mode?

5

I gave those flags a try, but the command still hangs just like before. There’s no output, and I have to force close the session.

6

Right, but you can’t give up there; you’re just getting started in a nice debugging session. Consider making some observations here, by way of experimentation, to help us help you.

Is /app/bin/migrate a binary, or does it have a hashbang? If the latter, what is its runtime?

I assume Phoenix is a web framework and migrate is part of a built-in ORM. Point us to the docs; that should indicate what switches would be helpful.

Is this library open source and on GitHub? If so, is there a relevant bug report?

Can you spin up this machine locally in Docker? If so, does the migrate command behave differently there?

7

Here’s what I’ve found so far:

  • The file /app/bin/migrate is a shell script generated by mix phx.gen.release.
    Its contents are:
#!/bin/sh
set -eu

cd -P -- "$(dirname -- "$0")"
exec ./server eval Server.Release.migrate
  • The function Server.Release.migrate is defined as:
def migrate do
  load_app()

  for repo <- repos() do
    {:ok, _, _} = Ecto.Migrator.with_repo(repo, &Ecto.Migrator.run(&1, :up, all: true))
  end
end
  • I tried running the same Docker image locally and executing the same command manually:
 fly deploy --local-only

Unfortunately, it also freezes and doesn’t produce any logs or errors

8

Splendid, great debugging. Keep going! Can you try the binary for Ecto Migrate?

9

Hey, I tested that!

I tried running /app/bin/migrate manually from the Fly.io SSH console. The binary is definitely there (I can see it listed in /app/bin), but as soon as I run it, the terminal just exits and throws me back into my local shell

root@7811196b062468:/app# /app/bin/migrate
aleiw@aleo62-system C:\....\server 

(ignore question mark box, I am using oh-my-posh)

So it doesn’t freeze anymore, but it also doesn’t do anything visible, no logs, no error, just exits immediately. Not sure if it crashed silently or if something is misconfigured.

10

Urgh, this looks like you’re trying to run a Windows binary in Linux. That definitely ain’t gonna work. Or, at any rate, maybe a Windows path (“C:\…”) is configured somewhere.

OK. Say more? It helps if you expand on what you’ve tried, in substantial detail, otherwise it requires me to expend another question loop asking you what you tried.

It looks like Ecto is a Python system. So, as a baseline, you need to get some Ecto code running in Python to print out the state of your migrations.

Final question for this evening; what base Docker image are you using? If it is Debian you should be fine, if it is something lightweight, like Alpine, there are subtle differences in some libraries that make things a bit harder to get going.

11

have you looked at your app’s logs? a crash kicking you out of your ssh session makes me think the app OOMed.

1 Like