I’ve taken memory snapshots of my running app locally (simulating the production version the best I can) and I haven’t been able to get the memory above 50 MB and it doesn’t increase over time either. It stays pretty steady between 30 and 50.
I’m at a loss of what to do at this point. Suggestions are welcome
Looking at the charts, it does look like this instance (7ae2dc40) is operating with ~8MB of free memory at all times. So it would be pretty sensitive to any increase in memory usage. You can see memory charts in our dashboard if you navigate to your app’s metrics page.
I’d suggest flyctl ssh console-ing into your app and looking at top (or htop) and maybe watching dmesg to see what’s happening. It might kick you out as soon as it OOMs though so you might not be able to catch it.
In any case, this would show you what’s using the most memory inside the instance and might help you troubleshoot.
I wonder if
cross-env NODE_ENV=production remix-serve build
keeps too much memory around? I don’t really know how remix works, but it sounds like it might be building your app at runtime? If you can pre-build it in your docker image, that would be ideal. Were you testing locally with this exact command?
This is running on a fairly small instance too (1x shared CPU and 256MB of memory), you might need more memory to run this specific app.
Thanks for the info! I have no idea why it’s consuming so much memory on Fly. remix-serve is a simple express server. The build part is just the directory that the express server should serve. We’re not building at runtime.
Thanks for the tips! I’m going to try and identify the memory issue. It really doesn’t make sense that it would take up that much memory in the first place. But at least it makes sense that I’m running out
Just a quick note, I’ve just verified that my app has been this way on Fly for as long as I’ve been deploying on fly. I’m concerned about why the memory usage is so high on Fly but not locally, but at least I know that I didn’t break something recently
That client includes numbers for active handles and GC and event loop lag, all of which are helpful when Node apps stumble. If you get that exporting we can help you with a Grafana dashboard.
I hadn’t considered that. How much memory overhead is there in fly running generally? Would that explain the ~180MBs of extra memory used (the ~40MBs I measured locally vs the ~220MBs on Fly)?
There’s very little from us, easily <5MB. Probably closer to 1MB at any given time. But since you’re running in a VM, you get all the OS related memory allocations to think about too. Here’s how one of your VMs was using memory recently:
That yellow section (RAM used) is most likely what node is doing. The Cach + Buffer (blue) is the operating system’s cache, which gets evicted when your process needs it.
It seems like your node process grabbed 240mb RAM at one point, and then settled, then bounced back up a little.
We need to make it easier to setup Grafana so you can see all this. It’s a pain in the ass the first time you start trying to use it, after that it’s incredibly helpful.
Did any of you ever figure out a solution to this?
Below is a screenshot of the memory usage my app has on Heroku, which never even begins to approach the 512MB quota. Here on Fly, deploying with 256MB, 512MB and 1GB has all failed due to OOM.
In addition to what @ignoramous said, post your OOM message here? It should be in an email.
Our platform is different enough from Heroku that it’s hard to compare graphs. Your processes run in isolated VMs here. This means there are more memory metrics that might trigger an OOM.
With 2 GB it starts, but it stays near max memory usage over several minutes. If you look at the Heroku screenshot, there’s a small dip right before 12 PM, which corresponds to the daily reboot, and memory usage is below 100 MB from start, and stays that way for hours. This last part matches the experience when I run the application locally, with ~100 MB memory being used.
If you can get it running for a decent amount of time, try this:
fly ssh console
Then run top and hit shift +m. This should sort processes by memory usage.
You can also look here to get a breakdown of what memory is actually in use. It’s possible some specific metric is varying over time, but the page cache is keeping it “full”: Sign In · Fly
My best guess on what’s happening here is that Heroku sets some environment variable that controls Node apps memory usage, and we’re not bringing it over. But that’s a wild guess with no evidence.
I put a $10 credit on your account so you have time to debug this.
