confused about Ecto migrations

New user here.

This article on How to migrate Mix Release projects · Fly explains that you cannot use mix and instead have to do e.g.,

bin/my_app eval "MyApp.Release.migration_status()"

What is the context of bin/my_app …?

Does this mean that you use ssh into your vm fto issue these commands?

Also, the above link notes, "Does the deployed code assume the migrations were already run? If so, you need to run your migrations first and start the application after " –

But – to get the migration code onto the vm you need to do flyctl deploy, but when you flyctl deploy it starts the app – so what am I missing?

Is it, instead, possible to tunnel the db connection to the db instance directly and just run migrations from my dev machine?

Thanks

New user here as well and I’m having the same problem, my application was crashing due to no migrations being run, but I couldn’t ssh into the VM because it had shut down, so I added some exception handling to keep the application running despite DB errors, now I can ssh but I don’t see /app/bin or /app/releases to start the Iex shell.
I did see an /app/rel directory and in there is overlays/bin which looked promising because it has a migrate shell script, but it doesn’t work because it’s trying to run <my_app> binary which doesn’t exist.
The fly dashboard and fly status show the application running but I don’t see anything with <my_app> in ps -aux so I’m not sure where the actual start script is in the system or how to connect to it.

I haven’t had time to explore it yet, but I’m thinking that this may be what they mean:

  • do a deploy of the old app code + new migration code, and run migrations (from a module function, as described above)
  • do a second deploy of new app code (with same migrations, but they’ve been run already)

Or, build a system where you set some app state flag to call new code vs. old code; then, set this flag after you run the migrations.

Either way, I think the intent of fly is to not have to resort to ssh.

Hello @sensiblearts!

The way migrations work on Fly with Phoenix. If you create a new Phoenix app today with a 1.6.4+ then run fly launch, it detects the Phoenix app, runs some mix commands, creates a Dockerfile and creates your fly.toml file.

In that fly.toml file you’ll find this…

[deploy]
  release_command = "/app/bin/migrate"

This uses a migrate script created in the Dockerfile that runs your project’s migrations. That works like this:

  1. deploying your new code (but not starting your app) and running the migrations using that code.
  2. After the migrations complete, a new instance of your application is deployed and started.
  3. Once it reports “healthy”, the old instance is shutdown.

So there is no need for SSH to run the migrations. But your project benefits from being designed to understand this deployment process.

The context for bin/my_app eval "MyApp.Release.migration_status()" is a command that can be run via SSH on the VM manually. It’s checking the current migration status as a manual process. A sort of “sanity check”.

I hope that helps!

1 Like

The files /app/bin/your_app are specific to the Dockerfile building the project. Are you using the Dockerfile created by Phoenix? Or is it a custom Dockerfile?

Are you able to build the Docker container locally and inspect if it has what you expect inside?

Is there a plan to implement something similar for rollbacks? Or for failed deployments?

If the migration fails, it doesn’t deploy. Manual intervention is required for rollbacks. There’s no safe way to perform those. :slight_smile:

So here is the scenario I’m envisioning:

  1. New code is deployed, but not started. The migration is run via the deploy_command configuration.
  2. The database is now (potentially) inconsistent with the old version of the app.
  3. The new version is started, but health checks fail. The old instance stays online.

The situation is obviously avoidable if you design your database and migrations correctly, but it feels like a source of problems to me. I also don’t see a clear solution, since any kind of automatic undo_release_command would potentially be unsafe like you pointed out.

Yes, I think your scenario makes sense and that could happen.

I have rolled back migrations manually when needed. Assuming you’ve created the MyApp.Release module, then getting an IEx terminal into the app and running MyApp.Release.rollback(MyApp.Repo, 20210416112207) rolls back to the specified migration.

Part of the Safe Ecto Migrations approach is to create non-breaking additions and roll out otherwise breaking changes as multi-stage deploys. This helps prevent the situation where you’ve broken the currently running app.

Another danger with trying to auto-rollback is that it’s up to the developer to write the down migration part. Sometimes people just burn that bridge and say, “this can’t be undone” and raise an exception.

So you are right… there are ways to break a system. This is not unique to Phoenix or Ecto though. Many of the learnings and strategies in Safe Ecto Migrations came from Rails experience.