app no longer accessible

1

I want to run 2 processes - cron and app (Node.js). Until yesterday only app ran and cron failed. Apparently there was an issue in my fly.toml with a section called

[build]
  builder = "gcr.io/buildpacks/builder"
  args = { target = "x86_64-unknown-linux-musl" }

I have removed that and the cron process started. All good (see this topic Can't define cron job for periodic container start - #6 by camodue)

Then I tested the app and it doesn’t start anymore. Looking at the fly.io status there was an issue. Not sure this affected me. Tried again this morning.

2024-04-16T23:40:41.063 app[91852306f50258] syd [info] [ 0.048153] PCI: Fatal: No config space access function found
2024-04-16T23:40:41.282 app[91852306f50258] syd [info] INFO Starting init (commit: 5b8fb02)...
2024-04-16T23:40:41.292 app[91852306f50258] syd [info] INFO Mounting /dev/vdb at /workspace/storage w/ uid: 0, gid: 0 and chmod 0755
2024-04-16T23:40:41.294 app[91852306f50258] syd [info] INFO Resized /workspace/storage to 21458059264 bytes
2024-04-16T23:40:41.295 app[91852306f50258] syd [info] INFO Preparing to run: `docker-entrypoint.sh npm run start` as root
2024-04-16T23:40:41.300 app[91852306f50258] syd [info] INFO [fly api proxy] listening at /.fly/api
2024-04-16T23:40:41.303 app[91852306f50258] syd [info] 2024/04/16 23:40:41 listening on [fdaa:2:f5a2:a7b:243:7854:ebaa:2]:22 (DNS: [fdaa::3]:53)
2024-04-16T23:40:41.312 runner[91852306f50258] syd [info] Machine started in 442ms
2024-04-16T23:40:41.811 app[91852306f50258] syd [info] > ready2invoice@1.0.0 start
2024-04-16T23:40:41.811 app[91852306f50258] syd [info] > node app.js
2024-04-16T23:40:42.008 app[91852306f50258] syd [info] /app/node_modules/bindings/bindings.js:121
2024-04-16T23:40:42.008 app[91852306f50258] syd [info] throw e;
2024-04-16T23:40:42.008 app[91852306f50258] syd [info] ^
2024-04-16T23:40:42.008 app[91852306f50258] syd [info] Error: Error loading shared library /app/node_modules/sqlite3/build/Release/node_sqlite3.node: Exec format error
2024-04-16T23:40:42.008 app[91852306f50258] syd [info] at Module._extensions..node (node:internal/modules/cjs/loader:1464:18)
2024-04-16T23:40:42.008 app[91852306f50258] syd [info] at Module.load (node:internal/modules/cjs/loader:1205:32)
2024-04-16T23:40:42.008 app[91852306f50258] syd [info] at Module._load (node:internal/modules/cjs/loader:1021:12)
2024-04-16T23:40:42.008 app[91852306f50258] syd [info] at Module.require (node:internal/modules/cjs/loader:1230:19)
2024-04-16T23:40:42.008 app[91852306f50258] syd [info] at require (node:internal/modules/helpers:179:18)
2024-04-16T23:40:42.008 app[91852306f50258] syd [info] at bindings (/app/node_modules/bindings/bindings.js:112:48)
2024-04-16T23:40:42.008 app[91852306f50258] syd [info] at Object.<anonymous> (/app/node_modules/sqlite3/lib/sqlite3-binding.js:1:37)
2024-04-16T23:40:42.008 app[91852306f50258] syd [info] at Module._compile (node:internal/modules/cjs/loader:1368:14)
2024-04-16T23:40:42.008 app[91852306f50258] syd [info] at Module._extensions..js (node:internal/modules/cjs/loader:1426:10)
2024-04-16T23:40:42.008 app[91852306f50258] syd [info] at Module.load (node:internal/modules/cjs/loader:1205:32) {
2024-04-16T23:40:42.008 app[91852306f50258] syd [info] code: 'ERR_DLOPEN_FAILED'
2024-04-16T23:40:42.008 app[91852306f50258] syd [info] }
2024-04-16T23:40:42.008 app[91852306f50258] syd [info] Node.js v21.7.3
2024-04-16T23:40:42.300 app[91852306f50258] syd [info] INFO Main child exited normally with code: 1
2024-04-16T23:40:42.313 app[91852306f50258] syd [info] INFO Starting clean up.
2024-04-16T23:40:42.316 proxy[91852306f50258] syd [info] machine started in 1.455080316s
2024-04-16T23:40:42.322 app[91852306f50258] syd [info] INFO Umounting /dev/vdb from /workspace/storage
2024-04-16T23:40:42.323 app[91852306f50258] syd [info] [ 1.306151] reboot: Restarting system
2024-04-16T23:40:42.435 runner[91852306f50258] syd [info] machine has reached its max restart count (10)
2024-04-16T23:40:48.461 proxy[91852306f50258] syd [info] waiting for machine to be reachable on 0.0.0.0:8080 (waited 6.145489809s so far)

Both processes show deployed but strangely the app process runs in the Singapore region and has no mounts despite fly.toml clearly specifying both. Also when I open the console the path is at /app and not /workspace as before

What I also saw is there are now 3 machines

image
image937×245 26.6 KB

fly.toml

# fly.toml app configuration file generated for ready2invoice on 2024-03-14T11:54:26+11:00
#
# See https://fly.io/docs/reference/configuration/ for information about how to use this file.
#

app = 'ready2invoice'
primary_region = 'syd'

[experimental]

#[build]
#  builder = "gcr.io/buildpacks/builder"
#  args = { target = "x86_64-unknown-linux-musl" }

[http_service]
  internal_port = 8080
  force_https = true
  auto_stop_machines = true
  auto_start_machines = true
  min_machines_running = 0
  processes = ['app']

[[vm]]
  memory = '1gb'
  cpu_kind = 'shared'
  cpus = 1

### Added to include crontab
[processes]
  app = "npm run start" # command to start the Node.js app
  cron = "/usr/local/bin/supercronic /app/crontab"

[mounts]
  source="storage_dir"
  destination="/workspace/storage"

Based on the status page yesterday’s issue seems resolved. What else could be the issue?

2

This is still not working. Any idea where this is going wrong?

3

Hi,

This is a total guess as I haven’t used the builder approach or that particular library, but …

First, I would expect two machines to be for your app (since when you define processes in the fly.toml, it’s not obvious but what that actually does is make a machine per-process. Generally. If you want multiple processes on just one machine/vm to reduce cost, you have to manage doing that yourself - for example by using something like supervisor).

I would guess that third machine you are seeing listed is the remote builder machine that Fly creates for you, when you do a remote build. That is the default when you run fly deploy. That would make sense as it’s not in the region of syd that you specified for the app.

Next, the error you are seeing …

… would guess that is why you originally had the x86_64 target specified? Since it looks to be related to ARM:

That would make sense. So I’d think you’d either need to target x86 again, or make sure node_modules is ignored. At least based on that reply above.

Again, I tend to use a Dockerfile and a sibling .dockerignore file to get complete control over what files are added and so this may not be as simple when using their alternative builder/buildpack approach, but it’s basically this kind of thing from this “hello world” node app:

Don’t read too much into that app as it’s not been updated for V2 (aka Machines) however the basic principle still applies. You can see that my .dockerignore excludes node_modules (by not including it). I find it easier to inverse the usual approach which is to specify the files/folders to ignore. As the comments make clear :slight_smile:

Since if that is included, the target machine may differ from the machine it’s built on. So … it wouldn’t run. From that error message, alone it sounds to me like that is why it’s not running :thinking:

Fly then tries X times to get it running and have any healthchecks pass. If it fails over and over, it gives up machine has reached its max restart count (10). So the app won’t respond until it does.

4

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.