Out of memory on Elixir/Phoenix app when connecting `iex` session

Phoenix
1

I’m trying out Fly for the first time to deploy an Elixir Phoenix app with Postgres.

I’ve created two apps with 256MB of memory 1) for the app, and 2) for Postgres.

I got the app deployed and seemingly working, so tried connecting to iex using fly ssh console followed by /app/bin/<appname> remote and it connected fine. Then when I try to create a record in the database it terminates due to being out of memory.

*** ERROR: Shell process terminated! (^G to start new job) ***

And the logs from fly logs show the app restarting.

2022-01-28T11:56:58.234 app[3b89ab36] lhr [info][48902.313199] Out of memory: Killed process 510 (beam.smp) total-vm:1857876kB, anon-rss:190028kB, file-rss:0kB, shmem-rss:0kB, UID:65534 pgtables:580kB oom_score_adj:0
2022-01-28T11:56:58.351 app[3b89ab36] lhr [info]Main child exited with signal (with signal 'SIGKILL', core dumped? false)
2022-01-28T11:56:58.353 app[3b89ab36] lhr [info]Reaped child process with pid: 551, exit code: 0
2022-01-28T11:56:58.354 app[3b89ab36] lhr [info]Reaped child process with pid: 568, exit code: 0
2022-01-28T11:56:58.354 app[3b89ab36] lhr [info]Starting clean up.
2022-01-28T11:56:58.360 app[3b89ab36] lhr [info]Process appears to have been OOM killed!
2022-01-28T11:57:05.074 runner[3b89ab36] lhr [info]Starting instance
2022-01-28T11:57:05.117 runner[3b89ab36] lhr [info]Configuring virtual machine
2022-01-28T11:57:05.118 runner[3b89ab36] lhr [info]Pulling container image
2022-01-28T11:57:05.638 runner[3b89ab36] lhr [info]Unpacking image
2022-01-28T11:57:05.643 runner[3b89ab36] lhr [info]Preparing kernel init
2022-01-28T11:57:06.118 runner[3b89ab36] lhr [info]Configuring firecracker
2022-01-28T11:57:06.338 runner[3b89ab36] lhr [info]Starting virtual machine
2022-01-28T11:57:06.488 app[3b89ab36] lhr [info]Starting init (commit: 0c50bff)...
2022-01-28T11:57:06.513 app[3b89ab36] lhr [info]Preparing to run: `/app/bin/server` as nobody
2022-01-28T11:57:06.540 app[3b89ab36] lhr [info]2022/01/28 11:57:06 listening on [fdaa:0:36fc:a7b:2809:3b89:ab36:2]:22 (DNS: [fdaa::3]:53)

But the metric graph for the app doesn’t seem to indicate being out of memory?

CleanShot 2022-01-28 at 11.53.12@2x
CleanShot 2022-01-28 at 11.53.12@2x1222×526 37.5 KB

Scaling the memory to 512MB let me create a record in the database using iex.

I was expecting that 256MB would be enough for what is, at the moment, basically a skeleton Phoenix app, but perhaps my expectations are incorrect?

Thanks for any insight!

2

I’ve remote iex’d into a 256mb instance countless times without OOM, so it’s hard to say what is going on here without seeing your app. Is “basically skeleton” really only some crud Repo.insert's or are you doing other more specialized things? My current non-trivial app runs about 180mb under general operation so your baseline looks reasonable. Note that the erlang VM running a stock phoenix app out of the box runs like 16mb, but BEAM will use whatever memory it needs under operation.

1 Like
3

Thanks for the response.

It’s pretty uninteresting at the moment :slight_smile: I generated the app under Phoenix 1.5.8 and recently upgraded to 1.6.6. It has phx.gen.auth authentication added, and it has a single JSON API endpoint. Nothing much else going on at the moment. You can find it here.

The actual code I was running in iex was the user registration function that phx.gen.auth generates for you:

def register_user(attrs) do
  %User{}
  |> User.registration_changeset(attrs)
  |> Repo.insert()
end

I noticed that these lines are logged as soon as I run /app/bin/<name> remote. Seems like something is happening with a child process straight away?

2022-01-28T14:53:31.166 app[3b89ab36] lhr [info]Reaped child process with pid: 611, exit code: 0
2022-01-28T14:53:31.167 app[3b89ab36] lhr [info]Reaped child process with pid: 613, exit code: 0
2022-01-28T14:53:31.169 app[3b89ab36] lhr [info]Reaped child process with pid: 632 and signal: SIGUSR1, core dumped? false

I also played around some more in iex - created some other records etc and it worked fine. But if I call the register_user function it kills iex every time.

Interesting. Yes, seems like this app should be fine.

:thinking:

4

My original report above said that this issue happened when registering a user. I since noticed that it also happened when logging in via the web. I had inkling that maybe Argon2 had something to do it because password hashing is involved in both of those actions.

I then found this on the Elixir Forum which describes the issue and possible fixes.

I changed config/prod.exs to have the following as described here.

# Set cost of password hashing complexity to use less memory
config :argon2_elixir, t_cost: 2, m_cost: 15 # ~60ms

And now everything seems good. Will report back if anything changes.

Thanks!

1 Like
5

Ah good catch! We actually made a change to phx.gen.auth in phoenix 1.6.1 for exactly this reason. (using bcrypt instead):

## 1.6.1 (2021-10-08)

### Enhancements
  ...
  * [phx.gen.auth] No longer set `argon2` as the default hash algorithm
   for `phx.gen.auth` in favor of bcrypt for performance reasons on smaller
   hardware
6

Thanks for letting me know. I may consider going back to bcrypt. Glad it’s fixed now :slight_smile: