FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory

When fly.io introduced swap_size_mb, I moved from creating the swap manually with a script to letting fly handle swap creation

And since then I’m getting the following error during fly deploy

#16 110.8 apps/***.com build: <--- Last few GCs --->
#16 110.8 apps/***.com build: [226:0x69db440]    79655 ms: Mark-Compact (reduce) 2039.4 (2081.6) -> 2039.3 (2082.1) MB, 1327.93 / 0.01 ms  (+ 34.2 ms in 6 steps since start of marking, biggest step 17.1 ms, walltime since start of marking 1385 ms) (average mu = 0.304, current mu = 0.2[226:0x69db440]    81531 ms: Mark-Compact (reduce) 2039.7 (2082.3) -> 2039.7 (2082.6) MB, 1359.31 / 0.00 ms  (average mu = 0.290, current mu = 0.275) allocation failure; GC in old space requested
#16 110.8 apps/***.com build: <--- JS stacktrace --->
#16 110.8 apps/***.com build: FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
#16 110.8 apps/***.com build: ----- Native stack trace -----
#16 110.8 apps/***.com build:  1: 0xb84bd6 node::OOMErrorHandler(char const*, v8::OOMDetails const&) [node]
#16 110.8 apps/***.com build:  2: 0xefec30 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [node]
#16 110.8 apps/***.com build:  3: 0xefef17 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [node]
#16 110.8 apps/***.com build:  4: 0x1110925  [node]
#16 110.8 apps/***.com build:  5: 0x11287a8 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
#16 110.8 apps/***.com build:  6: 0x10fe8c1 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
#16 110.9 apps/***.com build:  7: 0x10ffa55 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
#16 110.9 apps/***.com build:  8: 0x10dd0a6 v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [node]
#16 110.9 apps/***.com build:  9: 0x1537fe1 v8::internal::Runtime_AllocateInOldGeneration(int, unsigned long*, v8::internal::Isolate*) [node]
#16 110.9 apps/***.com build: 10: 0x7fc57fe99ef6 
#16 111.2 apps/***.com build: Aborted (core dumped)
#16 111.2 apps/***.com build:  ELIFECYCLE  Command failed with exit code 134.
#16 111.2 apps/***.com build: ERROR: "build:remix" exited with 1.

so basically, before I was using the following script to create the swap manually at container boot time

#!/usr/bin/env node

import { writeFile } from 'node:fs/promises'
import { $ } from 'execa'

console.log('setting up swapfile...')
await $`fallocate -l 512M /swapfile`
await $`chmod 0600 /swapfile`
await $`mkswap /swapfile`
await writeFile('/proc/sys/vm/swappiness', '10')
await $`swapon /swapfile`
await writeFile('/proc/sys/vm/overcommit_memory', '1')
console.log('swapfile setup complete')

and now I ditched the script to instead use swap_size_mb inside my fly.toml file
the changes I applied match this commit dffa1e6

My project is not that large, 2 libraries and 1 app at the moment, in a monorepo managed with pnpm.
This error is happening while pnpm build is running inside the docker
Increasing the swap to 1024 doesn’t change a thing

Is someone else experiencing the same issue? How can I resolve this issue?

See Scaling · Fly Docs

Something to try, add the following to your fly.toml:

[env]
NODE_OPTIONS='--max-old-space-size=4096'

thank you for the suggestion rubys
I played with max-old-space-size, also upgraded pnpm to v9. the error stays

#16 117.9 apps/***.com build: <--- Last few GCs --->
#16 117.9 apps/***.com build: [225:0x65b44d0]    87522 ms: Scavenge (reduce) 2037.3 (2080.4) -> 2037.0 (2080.6) MB, 8.47 / 0.00 ms  (average mu = 0.339, current mu = 0.301) allocation failure; 
#16 117.9 apps/***.com build: [225:0x65b44d0]    89356 ms: Mark-Compact (reduce) 2038.6 (2081.9) -> 2038.5 (2082.6) MB, 1289.74 / 0.01 ms  (+ 57.4 ms in 6 steps since start of marking, biggest step 20.2 ms, walltime since start of marking 1373 ms) (average mu = 0.313, current mu = 0.2
#16 117.9 apps/***.com build: <--- JS stacktrace --->
#16 117.9 apps/***.com build: FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
#16 117.9 apps/***.com build: ----- Native stack trace -----
#16 117.9 apps/***.com build:  1: 0xb84bd6 node::OOMErrorHandler(char const*, v8::OOMDetails const&) [node]
#16 117.9 apps/***.com build:  2: 0xefec30 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [node]
#16 117.9 apps/***.com build:  3: 0xefef17 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [node]
#16 117.9 apps/***.com build:  4: 0x1110925  [node]
#16 117.9 apps/***.com build:  5: 0x11287a8 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
#16 117.9 apps/***.com build:  6: 0x10fe8c1 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
#16 117.9 apps/***.com build:  7: 0x10ffa55 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
#16 117.9 apps/***.com build:  8: 0x10dd0a6 v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [node]
#16 117.9 apps/***.com build:  9: 0x1537fe1 v8::internal::Runtime_AllocateInOldGeneration(int, unsigned long*, v8::internal::Isolate*) [node]
#16 117.9 apps/***.com build: 10: 0x7f7b83e99ef6 
#16 118.3 apps/***.com build: Aborted (core dumped)
#16 118.3 apps/***.com build:  ELIFECYCLE  Command failed with exit code 134.
#16 118.3 apps/***.com build: ERROR: "build:remix" exited with 134.
#16 118.3 apps/***.com build: Failed
#16 118.3 /usr/src/app/apps/***.com:
#16 118.3  ERR_PNPM_RECURSIVE_RUN_FIRST_FAIL  ***.com@ build: `run-s build:*`
#16 118.3 Exit status 1
#16 ERROR: process "/bin/sh -c SENTRY_AUTH_TOKEN=$(cat /run/secrets/SENTRY_AUTH_TOKEN)   NODE_OPTIONS=--max-old-space-size=2048   pnpm run --filter=\"***.com...\" build" did not complete successfully: exit code: 1
------
 > [build 7/8] RUN --mount=type=secret,id=SENTRY_AUTH_TOKEN   SENTRY_AUTH_TOKEN=$(cat /run/secrets/SENTRY_AUTH_TOKEN)   NODE_OPTIONS=--max-old-space-size=2048   pnpm run --filter="***.com..." build:
117.9 apps/***.com build:  8: 0x10dd0a6 v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [node]
117.9 apps/***.com build:  9: 0x1537fe1 v8::internal::Runtime_AllocateInOldGeneration(int, unsigned long*, v8::internal::Isolate*) [node]
117.9 apps/***.com build: 10: 0x7f7b83e99ef6 
118.3 apps/***.com build: Aborted (core dumped)
118.3 apps/***.com build:  ELIFECYCLE  Command failed with exit code 134.
118.3 apps/***.com build: ERROR: "build:remix" exited with 134.
118.3 apps/***.com build: Failed
118.3 /usr/src/app/apps/***.com:
118.3  ERR_PNPM_RECURSIVE_RUN_FIRST_FAIL  ***.com@ build: `run-s build:*`
118.3 Exit status 1
------

I noticed that my swap_size_mb is still set to 512. I’m gonna try to delete the machine and create it again

I didn’t realize that this was during the build step. Swap is not relevant. And delete the max-old-space-size option as build machines start at 4Gb.

If you run fly apps list, one of those will be your builder (it will have a name starting with fly-builder). Run fly apps destroy on that builder.

Then try deploying again with a bigger builder:

flyctl deploy --remote-only --vm-size shared-cpu-4x:8196

Scale build machine - #2 by manoelcampos

1 Like

thank you for the suggestion. I am now sorted. The solution I went with was to reduce the memory usage of my vite build. I might try to get a bigger builder later when I face the issue again.

to reduce memory usage I did the following:

  • I updated most of my dependencies, especially vite and typescript (and pnpm)
  • I applied the performance advices from Performance | Vite especially I reduced the number of file extensions vite looks for by updating resolve.extensions
  • I tried to get the same version for most of the dependencies that were used in different projects

For anyone arriving on this page, the command to increase the VM would be flyctl deploy --remote-only --vm-size shared-cpu-4x --vm-memory 8196. I don’t know why but for me the creation of the VM happens after the build. I do not see any build machine when I run fly app list. So maybe the build was running inside the github runner even tho I did not specify --local-only and --remote-only is said to be the default

1 Like

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.