I am recently getting email saying my app has run out of memory when it hasn’t. The log I get in the email looks something like:
Out of memory: Killed process 525 (graphql-engine) total-vm:1074342616kB, anon-rss:641932kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:1388kB oom_score_adj:0
When I look at the deployment, it is healthy, the app is running, and the app is only using ~400MB (out of the available 1GB).
How can I debug this?
Are you running Hasura GraphQL Engine? That’s my guess based on the process name.
I found this issue: Hasura consuming massive amounts of memory · Issue #4077 · hasura/graphql-engine · GitHub
It sounds like you’ll need more memory to run it. You can try fly scale memory 2048 and see if that’s enough.
Memory usage likely varies during the lifetime of the your process.
You can likely see it happening in your metrics in the Fly dashboard.
@Jerome ah I see. It is strange as that is not reflected in my usage. You can see the two blips here from the past two days that were deployments (one of which causes the email to trigger saying run out of memory).
I would assume that the spikes would be much bigger?
Memory spikes happen very fast, it’s reasonably normal to not see them in metrics. Metrics get scraped every 15-30 seconds. They’re good for trends, but not specific events.
If you use the Open In Grafana link you can navigate to a per instance dashboard showing a lot more memory usage info.
The log line shows the metric at kill time: total-vm: 1074342616kB.
