python app OoM-ing every week

I have a python webapp defined here: mini_healer_tools/fly.toml at main · raylu/mini_healer_tools · GitHub

it runs fairly well but every week, it runs out of memory and I get an email saying “[Fly.io] mini-healer-tools ran out of memory and crashed”

grafana shows a sawtooth pattern in the memory usage


which strongly suggests a memory leak. however, having tried mprof, memory_profiler, fil-profile, and memray, I can’t identify the cause of the leak. it’s also such a simple webapp that I would expect to be able to find it pretty quickly if there was one

my remaining ideas are that

  1. something is causing the python GC to not run (perhaps it thinks it has more memory than it actually has to work with?)
  2. the memory usage is caused by the fly.io thing that serves static files

does anyone have any experience like this or any leads?

1 Like