PCI: Fatal: No config space access function found

Questions / Help Build debugging
1

Hi.

I am having an issue to deploy a Remix app I was able to deploy in the past. I am using Prisma and following the recommendation using a swap file.

Here is the errog log showed during the deployment:

-------
 ✖ release_command failed
-------
Error release_command failed running on machine 90801936b20908 with exit code 134.
Check its logs: here's the last 100 lines below, or run 'fly logs -i 90801936b20908':
  Pulling container image registry.fly.io/****:deployment-01HWTSVM53JY8W68W8WGNEETEE
  Successfully prepared image registry.fly.io/****:deployment-01HWTSVM53JY8W68W8WGNEETEE (7.056954017s)
  Configuring firecracker
  [    0.037002] PCI: Fatal: No config space access function found
   INFO Starting init (commit: c1e2693b)...
   INFO Preparing to run: `docker-entrypoint.sh bash ./scripts/migrate.sh` as root
   INFO [fly api proxy] listening at /.fly/api
  2024/05/01 19:28:16 INFO SSH listening listen_address=[fdaa:0:648d:a7b:40:f30f:e9c7:2]:22 dns_server=[fdaa::3]:53
  Machine created and started in 9.024s
  fallocate: fallocate failed: Text file busy
  mkswap: error: /swapfile is mounted; will not make swapspace
  swapon: /swapfile: swapon failed: Device or resource busy
  npm WARN exec The following package was not found and will be installed: prisma@5.13.0
  <--- Last few GCs --->
  [315:0x6682850]    14315 ms: Scavenge (reduce) 253.5 (257.7) -> 253.4 (258.5) MB, 21.1 / 0.0 ms  (average mu = 0.668, current mu = 0.728) allocation failure;
  [315:0x6682850]    14858 ms: Mark-sweep (reduce) 254.4 (258.5) -> 254.1 (259.5) MB, 379.1 / 0.0 ms  (+ 190.7 ms in 30 steps since start of marking, biggest step 16.9 ms, walltime since start of marking 799 ms) (average mu = 0.585, current mu = 0.501) allo
  <--- JS stacktrace --->
  FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
   1: 0xb9a330 node::Abort() [npm exec prisma migrate deploy]
   2: 0xaa07ee  [npm exec prisma migrate deploy]
   3: 0xd71ed0 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [npm exec prisma migrate deploy]
   4: 0xd72277 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [npm exec prisma migrate deploy]
   5: 0xf4f635  [npm exec prisma migrate deploy]
   6: 0xf61b0d v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [npm exec prisma migrate deploy]
   7: 0xf3c1fe v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [npm exec prisma migrate deploy]
   8: 0xf3d5c7 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [npm exec prisma migrate deploy]
   9: 0xf1dff2 v8::internal::Factory::AllocateRawWithAllocationSite(v8::internal::Handle<v8::internal::Map>, v8::internal::AllocationType, v8::internal::Handle<v8::internal::AllocationSite>) [npm exec prisma migrate deploy]
  10: 0xf2884c v8::internal::Factory::NewJSObjectFromMap(v8::internal::Handle<v8::internal::Map>, v8::internal::AllocationType, v8::internal::Handle<v8::internal::AllocationSite>) [npm exec prisma migrate deploy]
  11: 0x1057e05 v8::internal::JsonParser<unsigned short>::BuildJsonObject(v8::internal::JsonParser<unsigned short>::JsonContinuation const&, v8::base::SmallVector<v8::internal::JsonProperty, 16ul, std::allocator<v8::internal::JsonProperty> > const&, v8::internal::Handle<v8::internal::Map>) [npm exec prisma migrate deploy]
  12: 0x1059a88 v8::internal::JsonParser<unsigned short>::ParseJsonValue() [npm exec prisma migrate deploy]
  13: 0x105a19f v8::internal::JsonParser<unsigned short>::ParseJson() [npm exec prisma migrate deploy]
  14: 0xdf7983 v8::internal::Builtin_JsonParse(int, unsigned long*, v8::internal::Isolate*) [npm exec prisma migrate deploy]
  15: 0x1710839  [npm exec prisma migrate deploy]
  ./scripts/migrate.sh: line 9:   315 Aborted                 npx prisma migrate deploy
   INFO Main child exited normally with code: 134
   INFO Starting clean up.
  [   16.374888] reboot: Restarting system
  machine restart policy set to 'no', not restarting
-------
Checking DNS configuration for *****.fly.dev
Error: release command failed - aborting deployment. error release_command machine 90801936b20908 exited with non-zero status of 134

Here are the key errors:
PCI: Fatal: No config space access function found

  fallocate: fallocate failed: Text file busy
  mkswap: error: /swapfile is mounted; will not make swapspace
  swapon: /swapfile: swapon failed: Device or resource busy

Any help would be greatly appreciated :pray:

2

Can you share the contents of this file? The fix is likely to remove a bunch of lines in that file that set up swap, and add one line to your fly.toml: Fly Launch configuration (fly.toml) · Fly Docs

3

Hi @rubys

Sure, here is my fly.toml:

app = "f***-e**m"
kill_signal = "SIGINT"
kill_timeout = 5
processes = []
swap_size_mb = 1024
...

and the migrate script:

#!/bin/bash

fallocate -l 1024M /swapfile
chmod 0600 /swapfile
mkswap /swapfile
echo 10 >/proc/sys/vm/swappiness
swapon /swapfile
echo 1 >/proc/sys/vm/overcommit_memory
npx prisma migrate deploy

Based on your remark

The fix is likely to remove a bunch of lines in that file that set up swap

I thought I could remove the first lines, to the point to only keep the last line. But it did not succeed.

4

If your resulting migrate script looked like the following, at a minimum you would have gotten a different error:

#!/bin/bash
npx prisma migrate deploy

What error are you now seeing?

5

I get more or less the same message without the swap thingy:

Running f***m release_command: bash ./scripts/migrate.sh

-------
 ✖ release_command failed
-------
Error release_command failed running on machine 4d890e31c69d38 with exit code 134.
Check its logs: here's the last 100 lines below, or run 'fly logs -i 4d890e31c69d38':
  Pulling container image registry.fly.io/f***m:deployment-01HWWC82Q531M7MPMP5MGYNTH4
  Successfully prepared image registry.fly.io/f***m:deployment-01HWWC82Q531M7MPMP5MGYNTH4 (6.748393531s)
  Configuring firecracker
  [    0.038294] PCI: Fatal: No config space access function found
   INFO Starting init (commit: c1e2693b)...
   INFO Preparing to run: `docker-entrypoint.sh bash ./scripts/migrate.sh` as root
   INFO [fly api proxy] listening at /.fly/api
  Machine created and started in 8.409s
  2024/05/02 10:05:40 INFO SSH listening listen_address=[fdaa:0:648d:a7b:39:d04f:596f:2]:22 dns_server=[fdaa::3]:53
  npm WARN exec The following package was not found and will be installed: prisma@5.13.0
  <--- Last few GCs --->
  [311:0x6c6b850]    16314 ms: Mark-sweep (reduce) 253.5 (258.0) -> 253.3 (258.5) MB, 569.6 / 0.0 ms  (+ 73.9 ms in 15 steps since start of marking, biggest step 17.8 ms, walltime since start of marking 765 ms) (average mu = 0.572, current mu = 0.392) alloc[311:0x6c6b850]    16962 ms: Mark-sweep (reduce) 254.4 (258.5) -> 254.2 (259.5) MB, 632.9 / 0.0 ms  (average mu = 0.377, current mu = 0.026) allocation failure; scavenge might not succeed
  <--- JS stacktrace --->
  FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
   1: 0xb9a330 node::Abort() [npm exec prisma migrate deploy]
   2: 0xaa07ee  [npm exec prisma migrate deploy]
   3: 0xd71ed0 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [npm exec prisma migrate deploy]
   4: 0xd72277 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [npm exec prisma migrate deploy]
   5: 0xf4f635  [npm exec prisma migrate deploy]
   6: 0xf61b0d v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [npm exec prisma migrate deploy]
   7: 0xf3c1fe v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [npm exec prisma migrate deploy]
   8: 0xf3d5c7 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [npm exec prisma migrate deploy]
   9: 0xf1db40 v8::internal::Factory::AllocateRaw(int, v8::internal::AllocationType, v8::internal::AllocationAlignment) [npm exec prisma migrate deploy]
  10: 0xf155b4 v8::internal::FactoryBase<v8::internal::Factory>::AllocateRawWithImmortalMap(int, v8::internal::AllocationType, v8::internal::Map, v8::internal::AllocationAlignment) [npm exec prisma migrate deploy]
  11: 0xf17868 v8::internal::FactoryBase<v8::internal::Factory>::NewRawOneByteString(int, v8::internal::AllocationType) [npm exec prisma migrate deploy]
  12: 0x1057999 v8::internal::JsonParser<unsigned short>::MakeString(v8::internal::JsonString const&, v8::internal::Handle<v8::internal::String>) [npm exec prisma migrate deploy]
  13: 0x10596b6 v8::internal::JsonParser<unsigned short>::ParseJsonValue() [npm exec prisma migrate deploy]
  14: 0x105a19f v8::internal::JsonParser<unsigned short>::ParseJson() [npm exec prisma migrate deploy]
  15: 0xdf7983 v8::internal::Builtin_JsonParse(int, unsigned long*, v8::internal::Isolate*) [npm exec prisma migrate deploy]
  16: 0x1710839  [npm exec prisma migrate deploy]
  ./scripts/migrate.sh: line 9:   311 Aborted                 npx prisma migrate deploy
   INFO Main child exited normally with code: 134
   INFO Starting clean up.
  [   18.393589] reboot: Restarting system
  machine restart policy set to 'no', not restarting
-------
Checking DNS configuration for f***m.fly.dev
Error: release command failed - aborting deployment. error release_command machine 4d890e31c69d38 exited with non-zero status of 134
6

Weird.

That can be ignored. It sounds scary, but it is normal.

Seems odd, but things are proceeding, so probably is harmless.

That likely is the the real problem. 1Gb of RAM (+ whatever you have as real memory, probably 256M?) should be enough?

Try the following migrate script:

#!/bin/bash
swapon -s
npx prisma migrate deploy

When run you should see output like the following:

  Filename				Type		Size		Used	Priority
  /dev/vdc                                partition	1048572		0	-2

That will confirm that you do, indeed, have swap space defined.

7

Yes, swapon -s did return the expected result:

  Filename				Type		Size	Used	Priority
  /swapfile                              	file    	1048572	0	-2

Although it says file and not partition.

8

Perhaps try this suggestion:

9

I just tried with this line in my Dockerfile

ENV NODE_OPTIONS="–max-old-space-size=256"

Still the same error. I’m starting to feel hopeless.

10

I think you need to make that number bigger, and perhaps only want to set it for release. My read is that without that option, node is not taking advantage of swap, and you need to tell it that it can use more.

Try:

#!/bin/bash
export NODE_OPTIONS=–max-old-space-size=1024
npx prisma migrate deploy

It also may be worth experimenting with increasing real memory - which is faster, but a paid feature. But the charges are only while in use, so trying it briefly shouldn’t be expensive:

11

We’ve had success asking people to use this in their fly.toml or set that node options variable elsewhere if suitable:

[env]
NODE_OPTIONS='--max-old-space-size=4096'

The above was tested on a 512MB system.

As Sam mentioned, this doesn’t mean it’ll require 4GB of physical RAM, it just tells node it can use up to 4GB. Note that if it does try to use more it’ll rightfully OOM, but that would be an OS-level OOM and not a node-level “the configured heap is too small even if there’s enough physical memory for more”.

  • Daniel
3 Likes
12

Thanks! I’ve updated the docs: Scaling · Fly Docs

Apparently if you can avoid setting it too high, JS’s garbage collection will kick in before the machine OOMs.

1 Like
13

It worked!
Thank you @roadmr and @rubys, your persistence succeeded :muscle:

1 Like