Deploy fail for Node app, exit code 51

aaronpeters · February 11, 2025, 3:03pm

Last deployed 10 months ago (success), tried a new deploy today, and failed.
I’ve gone through the threads here, read the docs for fly deploy and App configuration (fly.toml). I’m stuck
App runs fine on local (macbook pro).

Output from fly deploy --remote-only --debug :

==> Verifying app config
Validating /Users/aaronpeters/Work/cdntimings/fly.toml
✓ Configuration is valid
--> Verified app config
==> Building image
Remote builder fly-builder-late-water-5784 ready
==> Building image with Buildpacks
--> docker host: 24.0.7 linux x86_64
base: Pulling from paketobuildpacks/builder
Digest: sha256:17ea21162ba8c7717d3ead3ee3836a368aced7f02f2e59658e52029bd6d149e7
Status: Image is up to date for paketobuildpacks/builder:base
base-cnb: Pulling from paketobuildpacks/run
Digest: sha256:1af9935d8987fd52b2266d288200c9482d1dd5529860bbf5bc2d248de1cb1a38
Status: Image is up to date for paketobuildpacks/run:base-cnb
latest: Pulling from paketo-buildpacks/nodejs
Digest: sha256:49560cd644341c718e8b78f2ad28ae04a2317d3f17a1d4fd9762ab66b8e048ff
Status: Image is up to date for gcr.io/paketo-buildpacks/nodejs:latest
latest: Pulling from paketo-buildpacks/nodejs
Digest: sha256:49560cd644341c718e8b78f2ad28ae04a2317d3f17a1d4fd9762ab66b8e048ff
Status: Image is up to date for gcr.io/paketo-buildpacks/nodejs:latest
Builder is trusted but additional modules were added; using the untrusted (5 phases) build flow
0.16.5: Pulling from buildpacksio/lifecycle
Digest: sha256:6328c7b662c452111e39f1f5d5d097292d07c218ba4a873f0ad0c2cb67710869
Status: Image is up to date for buildpacksio/lifecycle:0.16.5
===> ANALYZING
[analyzer] Image with name "registry.fly.io/cdntimings:cache" not found
===> DETECTING
[detector] ======== Output: paketo-buildpacks/node-run-script@2.0.6 ========
[detector] could not find script(s) [build] in package.json
[detector] err:  paketo-buildpacks/node-run-script@2.0.6 (1)
[detector] ======== Output: paketo-buildpacks/node-run-script@2.0.6 ========
[detector] could not find script(s) [build] in package.json
[detector] err:  paketo-buildpacks/node-run-script@2.0.6 (1)
[detector] 5 of 10 buildpacks participating
[detector] paketo-buildpacks/ca-certificates 3.9.0
[detector] paketo-buildpacks/node-engine     5.2.2
[detector] paketo-buildpacks/npm-install     1.8.1
[detector] paketo-buildpacks/node-start      2.1.18
[detector] paketo-buildpacks/npm-start       2.1.6
===> RESTORING
===> BUILDING
[builder] 
[builder] Paketo Buildpack for CA Certificates 3.9.0
[builder]   https://github.com/paketo-buildpacks/ca-certificates
[builder]   Build Configuration:
[builder]     $BP_EMBED_CERTS                    false  Embed certificates into the image
[builder]     $BP_ENABLE_RUNTIME_CERT_BINDING    true   Deprecated: Enable/disable certificate helper layer to add certs at runtime
[builder]     $BP_RUNTIME_CERT_BINDING_DISABLED  false  Disable certificate helper layer to add certs at runtime
[builder]   Launch Helper: Contributing to layer
[builder]     Creating /layers/paketo-buildpacks_ca-certificates/helper/exec.d/ca-certificates-helper
[builder] Paketo Buildpack for Node Engine 5.2.2
[builder]   Resolving Node Engine version
[builder]     Candidate version sources (in priority order):
[builder]       package.json -> "18.20.4"
[builder]       <unknown>    -> ""
[builder] 
[builder]     Selected Node Engine version (using package.json): 18.20.4
[builder] 
[builder]   Executing build process
[builder]     Installing Node Engine 18.20.4
[builder]       Completed in 5.987s
[builder] 
[builder]   Generating SBOM for /layers/paketo-buildpacks_node-engine/node
[builder]       Completed in 0s
[builder] 
[builder]   Configuring build environment
[builder]     NODE_ENV     -> "production"
[builder]     NODE_HOME    -> "/layers/paketo-buildpacks_node-engine/node"
[builder]     NODE_OPTIONS -> "--use-openssl-ca"
[builder]     NODE_VERBOSE -> "false"
[builder] 
[builder]   Configuring launch environment
[builder]     NODE_ENV     -> "production"
[builder]     NODE_HOME    -> "/layers/paketo-buildpacks_node-engine/node"
[builder]     NODE_OPTIONS -> "--use-openssl-ca"
[builder]     NODE_VERBOSE -> "false"
[builder] 
[builder]     Writing exec.d/0-optimize-memory
[builder]       Calculates available memory based on container limits at launch time.
[builder]       Made available in the MEMORY_AVAILABLE environment variable.
[builder]     Writing exec.d/1-inspector
[builder] 
[builder] Paketo Buildpack for NPM Install 1.8.1
[builder]   Resolving installation process
[builder]     Process inputs:
[builder]       node_modules      -> "Found"
[builder]       npm-cache         -> "Not found"
[builder]       package-lock.json -> "Found"
[builder] 
[builder]     Selected NPM build process: 'npm rebuild'
[builder] 
[builder]     error: node: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.28' not found (required by node)
[builder]     
[builder] failed to execute npm get user-agent: exit status 1
[builder] ERROR: failed to build: exit status 1
==> Building image
✓ compatible remote builder found
INFO Override builder host with: https://fly-builder-late-water-5784.fly.dev (was tcp://[fdaa:0:1f78:a7b:c207:6f6c:f8bd:2]:2375)

Remote builder fly-builder-late-water-5784 ready
==> Building image with Buildpacks
--> docker host: 24.0.7 linux x86_64
base: Pulling from paketobuildpacks/builder
Digest: sha256:17ea21162ba8c7717d3ead3ee3836a368aced7f02f2e59658e52029bd6d149e7
Status: Image is up to date for paketobuildpacks/builder:base
base-cnb: Pulling from paketobuildpacks/run
Digest: sha256:1af9935d8987fd52b2266d288200c9482d1dd5529860bbf5bc2d248de1cb1a38
Status: Image is up to date for paketobuildpacks/run:base-cnb
latest: Pulling from paketo-buildpacks/nodejs
Digest: sha256:49560cd644341c718e8b78f2ad28ae04a2317d3f17a1d4fd9762ab66b8e048ff
Status: Image is up to date for gcr.io/paketo-buildpacks/nodejs:latest
latest: Pulling from paketo-buildpacks/nodejs
Digest: sha256:49560cd644341c718e8b78f2ad28ae04a2317d3f17a1d4fd9762ab66b8e048ff
Status: Image is up to date for gcr.io/paketo-buildpacks/nodejs:latest
Builder is trusted but additional modules were added; using the untrusted (5 phases) build flow
0.16.5: Pulling from buildpacksio/lifecycle
Digest: sha256:6328c7b662c452111e39f1f5d5d097292d07c218ba4a873f0ad0c2cb67710869
Status: Image is up to date for buildpacksio/lifecycle:0.16.5
===> ANALYZING
[analyzer] Image with name "registry.fly.io/cdntimings:cache" not found
===> DETECTING
[detector] ======== Output: paketo-buildpacks/node-run-script@2.0.6 ========
[detector] could not find script(s) [build] in package.json
[detector] err:  paketo-buildpacks/node-run-script@2.0.6 (1)
[detector] ======== Output: paketo-buildpacks/node-run-script@2.0.6 ========
[detector] could not find script(s) [build] in package.json
[detector] err:  paketo-buildpacks/node-run-script@2.0.6 (1)
[detector] 5 of 10 buildpacks participating
[detector] paketo-buildpacks/ca-certificates 3.9.0
[detector] paketo-buildpacks/node-engine     5.2.2
[detector] paketo-buildpacks/npm-install     1.8.1
[detector] paketo-buildpacks/node-start      2.1.18
[detector] paketo-buildpacks/npm-start       2.1.6
===> RESTORING
===> BUILDING
[builder] 
[builder] Paketo Buildpack for CA Certificates 3.9.0
[builder]   https://github.com/paketo-buildpacks/ca-certificates
[builder]   Build Configuration:
[builder]     $BP_EMBED_CERTS                    false  Embed certificates into the image
[builder]     $BP_ENABLE_RUNTIME_CERT_BINDING    true   Deprecated: Enable/disable certificate helper layer to add certs at runtime
[builder]     $BP_RUNTIME_CERT_BINDING_DISABLED  false  Disable certificate helper layer to add certs at runtime
[builder]   Launch Helper: Contributing to layer
[builder]     Creating /layers/paketo-buildpacks_ca-certificates/helper/exec.d/ca-certificates-helper
[builder] Paketo Buildpack for Node Engine 5.2.2
[builder]   Resolving Node Engine version
[builder]     Candidate version sources (in priority order):
[builder]       package.json -> "18.20.4"
[builder]       <unknown>    -> ""
[builder] 
[builder]     Selected Node Engine version (using package.json): 18.20.4
[builder] 
[builder]   Executing build process
[builder]     Installing Node Engine 18.20.4
[builder]       Completed in 5.935s
[builder] 
[builder]   Generating SBOM for /layers/paketo-buildpacks_node-engine/node
[builder]       Completed in 0s
[builder] 
[builder]   Configuring build environment
[builder]     NODE_ENV     -> "production"
[builder]     NODE_HOME    -> "/layers/paketo-buildpacks_node-engine/node"
[builder]     NODE_OPTIONS -> "--use-openssl-ca"
[builder]     NODE_VERBOSE -> "false"
[builder] 
[builder]   Configuring launch environment
[builder]     NODE_ENV     -> "production"
[builder]     NODE_HOME    -> "/layers/paketo-buildpacks_node-engine/node"
[builder]     NODE_OPTIONS -> "--use-openssl-ca"
[builder]     NODE_VERBOSE -> "false"
[builder] 
[builder]     Writing exec.d/0-optimize-memory
[builder]       Calculates available memory based on container limits at launch time.
[builder]       Made available in the MEMORY_AVAILABLE environment variable.
[builder]     Writing exec.d/1-inspector
[builder] 
[builder] Paketo Buildpack for NPM Install 1.8.1
[builder]   Resolving installation process
[builder]     Process inputs:
[builder]       node_modules      -> "Found"
[builder]       npm-cache         -> "Not found"
[builder]       package-lock.json -> "Found"
[builder] 
[builder]     Selected NPM build process: 'npm rebuild'
[builder] 
[builder] failed to execute npm get user-agent: exit status 1
[builder]     error: node: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.28' not found (required by node)
[builder]     
[builder] ERROR: failed to build: exit status 1
Error: failed to fetch an image or build from source: executing lifecycle: failed with status code: 51
Stacktrace:
goroutine 1 [running]:
runtime/debug.Stack()
	runtime/debug/stack.go:26 +0x5e
github.com/superfly/flyctl/internal/cli.printError(0xc000032000, 0xc000d07c1e, 0xc0005c0308, {0xed9b420, 0xc0003a5b20})
	github.com/superfly/flyctl/internal/cli/cli.go:184 +0x59e
github.com/superfly/flyctl/internal/cli.Run({0xedc2450?, 0xc0002e5640?}, 0xc000032000, {0xc0001c2090, 0x3, 0x3})
	github.com/superfly/flyctl/internal/cli/cli.go:117 +0x9c5
main.run()
	github.com/superfly/flyctl/main.go:47 +0x165
main.main()
	github.com/superfly/flyctl/main.go:26 +0x18

Fly.toml file

app = "cdntimings"

primary_region = "iad" 

kill_signal = "SIGINT"
kill_timeout = 5

[build]
  builder = "paketobuildpacks/builder:base"
  buildpacks = ["gcr.io/paketo-buildpacks/nodejs"]

[[services]]
  # http_checks = []
  internal_port = 8080
  # processes = ["app"]
  protocol = "tcp"
  # script_checks = []

  [services.concurrency]
    hard_limit = 100
    soft_limit = 60
    # type = "connections"

  [[services.ports]]
    # force_https = true
    handlers = ["http"]
    port = 80

  [[services.ports]]
    handlers = ["tls", "http"]
    port = 443

  [[services.tcp_checks]]
    grace_period = "5s"
    interval = "10s"
    port = "8080"
    restart_limit = 0
    timeout = "2s"

khuezy · February 11, 2025, 3:46pm

10 months is a long time. The problem is probably due to a version desync. Try locking down the version.

aaronpeters · February 11, 2025, 4:15pm

Version of what?
Thanks !

khuezy · February 11, 2025, 4:22pm

Either nodejs version or the builder base to something more recent? I don’t use paketo to know what env to set.

aaronpeters · February 11, 2025, 4:35pm

[build] config comes straight from App configuration (fly.toml) · Fly Docs

and I don’t understand why downgrading node version would be helpful

khuezy · February 11, 2025, 4:45pm

I don’t know the exact problem, but eg: if the new deployment picked up a newer version of node that depends on GLIBC 2.28 but the base image only has version 2.27, then downgrading node would help.
I’d bump everything up instead of down though.

aaronpeters · February 11, 2025, 6:56pm

Downgrade Node to 16.x would be fine for my app, but I don’t know how to specify a specific Paketo buildpack in the fly.toml
The default in the fly.toml is gcr.io/paketo-buildpacks/nodejs and that requires node 18 and up.
Ok, I got it !
I looked up a buildpack that should work for node 16 and made a change in fly.toml to gcr.io/paketo-buildpacks/nodejs@sha256:26c7da06bc1fcb353f3357250d2735dad28396a39a2269b6cd99a0ee676844fa.
Ran the fly deploy --remote-only again and it picked up this buildpack, and the deploy went through

Thanks a lot for pointing me in the right direction

khuezy · February 11, 2025, 7:10pm

I would use the later version of node and the latest paketo. The later version should be more secure and performant.

system · February 18, 2025, 7:10pm

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.