my_app ran out of memory and crashed

[mathiasn]

Hello,

I get these kind of email every 2 days or so. Sometimes on our Elixir app (up2date Phoenix with LiveView), sometimes on the Postgres App…

Out of memory: Killed process 520 (beam.smp) total-vm:1677288kB, anon-rss:393412kB, file-rss:0kB, shmem-rss:114180kB, UID:65534 pgtables:1300kB oom_score_adj:0

The raw total vm amount is pretty high, but I am wondering why I cannot see it on the metrics page directly in Fly or in Grafana:

(it crashed 17:13)

Any hints how to debug that? Thanks a lot!

[tj1]

1. You can’t see it on metrics due to sampling. Probably a single request causing this.

2. It could be argon2 settings.

https://elixirforum.com/t/staging-environment-how-to-debug-out-of-memory-errors-in-production-on-fly-io/42763/2

3. You could add some swap space by having a custom entrypoint script.

if [ $UID -eq 0 ]; then
  fallocate -l 1024M /swapfile
  chmod 0600 /swapfile
  mkswap /swapfile
  echo 10 > /proc/sys/vm/swappiness
  swapon /swapfile
  echo 1 > /proc/sys/vm/overcommit_memory
fi
exec  /app/bin/server

[mathiasn]

Thanks for your reply!

We’re not using argon2 as we don’t have user/passwords.

Is it common best practice to use swap on machines like that?

[tj1]

Best practice? Not sure.

I do it to avoid random OOM’s on my staging machine which has very little memory, but zero load. Wouldn’t want to be swapping in production though.

[mathiasn]

Just found that it’s even mentioned in the docs. That was also helpful.

[ben-io]

Of course, you can also just increase the VM memory size with fly scale memory 512.