Error on deploy: Out of memory: Killed process 257 (node)

1

Hello,

I wanted to upgrade the dependencies of my NodeJS app today. Once pushed to GitHub, the deploy job kept failing with error "Out of memory: Killed process 257 (node) total-vm:740560kB, anon-rss:137112kB, file-rss:4kB, shmem-rss:0kB, UID:1000 pgtables:1488kB oom_score_adj:0
"

What I don’t understand is that I didn’t change anything to the Toml config or VM config online. My last deployment a week ago worked perfectly fine yet today it keeps failing?
Btw on my machine the memory usage seems completely normal and checking last week’s log the NodeJS and NPM versions are the same.

Has anything changed this week on Fly’s end or does anyone has a clue to why it is suddenly failing? Thanks

Here are the deploy command logs:

==> Verifying app config
--> Verified app config
Validating /home/runner/work/api/api/fly.toml
Platform: machines
✓ Configuration is valid
==> Building image
Waiting for remote builder fly-builder-shy-bird-7117...
Remote builder fly-builder-shy-bird-7117 ready
==> Building image with Buildpacks
--> docker host: 20.10.12 linux x86_64
20: Pulling from heroku/buildpacks
Digest: sha256:455ac544557f82e72a606744d4ad912ad833c0731c439bfb417f8df8ecc20fe0
Status: Image is up to date for heroku/buildpacks:20
20-cnb: Pulling from heroku/heroku
Digest: sha256:f74f9ebed587fb048ef1d771917088b94cc264fe89b8363fb7f8dcee358b32ad
Status: Image is up to date for heroku/heroku:20-cnb
Warning: Platform requested deprecated API '0.6'
===> DETECTING
Warning: Buildpack 'heroku/ruby@0.0.0' requests deprecated API '0.4'
Warning: Buildpack 'heroku/python@0.0.0' requests deprecated API '0.4'
Warning: Buildpack 'heroku/scala@0.0.0' requests deprecated API '0.4'
Warning: Buildpack 'heroku/php@0.0.0' requests deprecated API '0.4'
Warning: Buildpack 'heroku/go@0.0.0' requests deprecated API '0.4'
Warning: Buildpack 'heroku/gradle@0.0.0' requests deprecated API '0.4'
2 of 3 buildpacks participating
heroku/nodejs-engine 1.1.1
heroku/nodejs-npm    1.1.1
===> ANALYZING
Restoring metadata for "heroku/nodejs-engine:dist" from app image
Restoring metadata for "heroku/nodejs-engine:web_env" from app image
Restoring metadata for "heroku/nodejs-npm:node_modules" from cache
Restoring metadata for "heroku/nodejs-npm:toolbox" from cache
===> RESTORING
Restoring data for "heroku/nodejs-engine:dist" from cache
Restoring data for "heroku/nodejs-npm:node_modules" from cache
Restoring data for "heroku/nodejs-npm:toolbox" from cache
===> BUILDING

[Heroku Node.js Engine Buildpack]

[Checking Node.js version]
Detected Node.js version range: >=16.0.0 <19.0.0
Resolved Node.js version: 18.16.1

  npm notice Run `npm install -g npm@9.8.0` to update!
  npm notice
   INFO Main child exited normally with code: 137
   INFO Starting clean up.
   WARN hallpass exited, pid: 238, status: signal: 15 (SIGTERM)
  2023/07/19 13:42:14 listening on [fdaa:0:8daa:a7b:169:ec36:aad4:2]:22 (DNS: [fdaa::3]:53)
  [    4.257399] reboot: Restarting system
  machine did not have a restart policy, defaulting to restart
   INFO Starting init (commit: 1d1821d)...
   INFO Preparing to run: `/cnb/process/web` as heroku
   INFO [fly api proxy] listening at /.fly/api
  2023/07/19 13:42:16 listening on [fdaa:0:8daa:a7b:169:ec36:aad4:2]:22 (DNS: [fdaa::3]:53)
  > colloscope-api@3.3.7 start
  > node --max-old-space-size=200 dist/index.js
  [    2.177217] Out of memory: Killed process 257 (node) total-vm:740560kB, anon-rss:137112kB, file-rss:4kB, shmem-rss:0kB, UID:1000 pgtables:1488kB oom_score_adj:0
  Killed
  npm notice
  npm notice New minor version of npm available! 9.5.1 -> 9.8.0
  npm notice Changelog: <https://github.com/npm/cli/releases/tag/v9.8.0>
  npm notice Run `npm install -g npm@9.8.0` to update!
  npm notice
   INFO Main child exited normally with code: 137
   INFO Starting clean up.
   WARN hallpass exited, pid: 238, status: signal: 15 (SIGTERM)
  2023/07/19 13:42:18 listening on [fdaa:0:8daa:a7b:169:ec36:aad4:2]:22 (DNS: [fdaa::3]:53)
  [    3.266954] reboot: Restarting system
Error: smoke checks for 683d471f779428 failed: the app appears to be crashing
1 Like
2

It seems you have a very low memory threshhold on your node command:

node --max-old-space-size=200 dist/index.js

Try to set --max-old-space-size=1048? Also, check that the RAM associated with yout application (fly scale show) is high enough.

Doc to upgrade RAM here: Scale Machine CPU and RAM · Fly Docs

3

I have to add that I have also memory-related crashed from my application starting three days ago. Everything was working fine before, but it suddenly started to crash regularly because “out of memory”. It seems the application started to need a lot more RAM all of a sudden.

Maybe it’s more related to Node than to Fly? I’m using a custom docker file with node:lts-alpine, is it the same for the Heroku builder? Maybe node:lts-alpine just got an update.

→ Just checked that and indeed, node:lts-alpine got an update 7 days ago. I think the issue comes from here, but I couldn’t find any more information.

4

Thank you for your reply! I added that memory threshold to try to mitigate the “out of memory” issue. It would also fail without it.

I’ll take a look at the Heroku builder image and see if an update might have caused this.

5

An update of the Heroku NodeJS buildpack was released yesterday but it just adds support for NodeJS 18.17.0. Not really sure if that could cause an issue.

The fact my app crashed a few days ago too (at runtime) due to insufficient memory makes me believe the migration to v2 to be the cause. They must have changed the way memory is handled on these machines and now we’re kind of forced to pay to upgrade. A bummer for running such a simple NodeJS app.

6

Alright well deploying from my PC via the command line works fine. Seems like it could be a GitHub Action issue?

7

Never mind, did a second deployment for testing and now crashes on both GitHub Actions and from my local PC. Might need to move on to a better provider.

8

@younesaassila we didn’t change how we handle memory directly but we did change what deploys images (we have a new thing called flyd).

Your --max-old-space-size=1048 could help but I can see your machine still is set to 256MB and that’s ok.

Sam Ruby did an amazing job at making NodeJS apps in Fly work with Swap memory too and one suggestion that could help with your case is to switch from buildpacks to Dockerfile. Here’s what you’d need to do in case you want to go that route.

First generate a Dockefile for you. We do that when you fly launch but you already have an app so you can do this directly using: npx --yes @flydotio/dockerfile@latest --swap=1024 --port=CHANGE_TO_THE_PORT_YOUR_APP_LISTES_TO

You can learn more about the options in the link below. You’d want the flag --dev if you use devDependencies in prod for example

Do a quick sanity check on the generated files. After that, make sure you remove references to buildpacks on your fly.toml, these should be under [builder] somewhere. We will use your Dockerfile on your next deployment.

Let me know how this goes.

3 Likes
9

Thank you for your help! Switching to a Dockerfile has indeed fixed my RAM issues! I did have to bump the swap value from 1024 to 512M since the entry point complained that swap needed to be at least 40 KiB and 512M seems to be the default Fly value.

3 Likes
10

Glad to know it’s working and thanks for sharing the additional steps you had to make.

11

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.