Using fly volumes with Mnesia, Phoenix and Pow

[oleksify]

Hey. I’m having an issue with Phoenix app that uses Pow (GitHub - pow-auth/pow: Robust, modular, and extendable user authentication system) for auth. I’m using Mnesia to save session. Locally everything works fine, and without adding Volumes everything also works fine. But when I’m adding a volume to app with:

[mounts]
source = "handshake_mnesia"
destination = "/mnesia"

and setting

config :mnesia, dir: '/mnesia'

in config/releases.exs

I get this error when starting an app:

[error] Couldn't initialize mnesia cluster because: {:change_table_copy_type, {:aborted, {:badarg, :schema, :unknown}}}

[info] Application handshake exited: Handshake.Application.start(:normal, []) returned an error: shutdown: failed to start child: Pow.Store.Backend.MnesiaCache
2021-06-22T11:53:01Z [info]     ** (EXIT) {:change_table_copy_type, {:aborted, {:badarg, :schema, :unknown}}}

Am I missing something? Thanks!

[kurt]

That error isn’t super helpful but there’s a good chance you’re hitting a permissions problem. The Elixir process might not have permissions to write to /mnesia.

[oleksify]

Thanks! That actually fixed it!

[OldhamMade]

Hey, glad to hear you resolved the issue! Can you share what you did to fix the problem? It may help other developers in the future. Thanks in advance!

[sam]

I’ve just run into this as well, it’d be super helpful if you could post what you did

[oleksify]

Sorry, kind of missed notifications on this thread. I thought I managed to fix it at some point, but then errors appear again after redeploy. I eventually gave it up, and we switched from Pow to a custom auth solution that does not require Mnesia. So, unfortunately - no solution

[joshsmith]

@kurt what do we need to do to grant write permissions to /mnesia?

I can’t do this in my Dockerfile prior to the [mounts] config in my fly.toml since the directory doesn’t yet exist.

Thanks in advance!

[jstiebs]

You may need to ssh in and chown the volume to give yourself permissions.

@peregrine

• chown nobody ./menisa

[joshsmith]

Unfortunately, the default owner already appears to be nobody.

# ls -la
...
drwxr-xr-x   3 nobody nogroup 4096 Feb 28 15:01 mnesia
...

[joshsmith]

I just went ahead and cd-ed into mnesia and ran chown nobody ./* and that may have fixed it. This is definitely weird since AFAICT the files in mnesia can’t be created until the Pow store is initialized, and that process fails unless the files are given the proper permissions. Don’t have much confidence that this build process won’t fail again in the future.

[jstiebs]

I’m glad you were able to get it working. I will ask around to see if there is a way for us to make this change more permanent.

[joshsmith]

It no longer seems to be working, which was as I expected.

As I alluded to before, when the app first initializes mnesia it appears to create a few files that don’t inherit the permissions of the parent directory. Of course, I then won’t have any active instances of my server running to be able to manually SSH in and fix the permissions myself unless I redeploy with a change that removes Pow.Store.Backend.MnesiaCache from my application’s start.

Here are the files created:

drwxr-xr-x  3 nobody nogroup  4096 Mar  1 16:20 .
drwxr-xr-x 24 root   root     4096 Mar  1 16:20 ..
-rw-r--r--  1 nobody nogroup   153 Mar  1 16:20 DECISION_TAB.LOG
-rw-r--r--  1 nobody nogroup     8 Nov 18 16:24 Elixir.Pow.Store.Backend.MnesiaCache.DCD
-rw-r--r--  1 nobody nogroup    92 Mar  1 16:20 LATEST.LOG
drwx------  2 nobody root    16384 Nov 18 16:24 lost+found
-rw-r--r--  1 nobody nogroup  9412 Nov 18 16:24 schema.DAT

[joshsmith]

I can’t even replicate the fix that I had suggested above. It’s totally broken with the same error as the original poster and no change I make seems to affect it at all.

[joshsmith]

@jstiebs More interesting discoveries. I blew away the volume and recreated it and everything seemed to work just fine. However, I then restarted my app and the error has returned. I’m not sure why this would be the case, but may help us narrow down the cause of the problem.

[dangra]

Hi @joshsmith. I’m trying to reproduce this issue with the bare simplest app and so far no luck.

I’ve created this app GitHub - dangra/nobodyapp: A test Fly app to test, but couldn’t reproduce, It always create permissions as nobody and works after a restart.

I’m not proficient in Erlang/Elixir/Mnesia, do you mind sharing the smallest app you could think of that reproduce this error?

[dangra]

Hey, I think the problem is on permissions set for lost+found folder which is root:root no matter what.

Mnesia could be scanning the parent directory and failing to stat lost+found, thus the failure.

I simply workaround is to mount the volume under /data and initialize mnesia under /data/mnesia instead of /data

This is a hunch though, we could test it once I’m able to reproduce.

[jstiebs]

I put together a minimal test case that would exercise the mnesia code erroring here GitHub - jeregrine/perm_error and I also cannot reproduce the error seen here.

[hodoli]

Any updates on the issue?
I’m experiencing the same problem.

[hodoli]

FYI,
fly.io now automatically generates env.sh.eex file containing env variables.

#!/bin/sh

# configure node for distributed erlang with IPV6 support
export ERL_AFLAGS="-proto_dist inet6_tcp"
export ECTO_IPV6="true"
export DNS_CLUSTER_QUERY="${FLY_APP_NAME}.internal"
export RELEASE_DISTRIBUTION="name"
export RELEASE_NODE="${FLY_APP_NAME}-${FLY_IMAGE_REF##*-}@${FLY_PRIVATE_IP}"

you should change RELEASE_NODE to be persistent each deploy for mnesia clustering
eg)
export RELEASE_NODE="${FLY_APP_NAME}@${FLY_PRIVATE_IP}"