Summary
After every deploy, one of my machines (the worker-pdf process below) stops without a failure, the logs just seem to halt. Running a quick fly machine start <id> gets it up and running, but it is annoying that it pauses on it’s own with no signal of a failure.
fly.dev.toml
app = "client-portal-development"
primary_region = "iad"
kill_signal = "SIGINT"
kill_timeout = "5s"
[deploy]
release_command = "npx prisma migrate deploy && yarn run-data-migrations"
[experimental]
auto_rollback = true
[env]
PORT = "8080"
PRIMARY_REGION = "iad"
[processes]
web = "yarn start"
worker-email= "yarn worker:email"
worker-pdf= "yarn worker:pdf"
[[services]]
protocol = "tcp"
internal_port = 8080
processes = ["web"]
[[services.ports]]
port = 80
handlers = ["http"]
force_https = true
[[services.ports]]
port = 443
handlers = ["tls", "http"]
[services.concurrency]
type = "connections"
hard_limit = 25
soft_limit = 20
[[services.tcp_checks]]
interval = "15s"
timeout = "2s"
grace_period = "1s"
restart_limit = 0
Repro steps:
- Check machines
3 machines have been retrieved from app client-portal-development.
View them in the UI here (...)
client-portal-development
ID NAME STATE REGION IMAGE IP ADDRESS VOLUME CREATED LAST UPDATED APP PLATFORM PROCESS GROUP SIZE
1781155ad41189 crimson-wind-9803 started iad client-portal-development:deployment-01H46JA9HHQEH9NAES1E93VK9H fdaa:1:b784:a7b:ab8:6033:4b56:2 2023-06-29T20:29:19Z 2023-06-30T17:00:47Z v2 worker-pdf shared-cpu-1x:2048MB
3d8d501fe51e98 falling-dream-2883 started iad client-portal-development:deployment-01H46JA9HHQEH9NAES1E93VK9H fdaa:1:b784:a7b:93:a57d:109f:2 2023-06-29T20:29:19Z 2023-06-30T16:36:31Z v2 worker-email shared-cpu-1x:2048MB
3287436db63758 hidden-bush-7456 started iad client-portal-development:deployment-01H46JA9HHQEH9NAES1E93VK9H fdaa:1:b784:a7b:143:f481:de07:2 2023-05-08T17:08:30Z 2023-06-30T16:37:10Z v2 web shared-cpu-1x:256MB
- Deploy
$ fly deploy -c fly.dev.toml
...
Running client-portal-development release_command: npx prisma migrate deploy && yarn run-data-migrations
release_command 90800d5b16de18 completed successfully
Updating existing machines in 'client-portal-development' with rolling strategy
[1/3] Machine 1781155ad41189 [worker-pdf] update finished: success
[2/3] Machine 3d8d501fe51e98 [worker-email] update finished: success
[3/3] Machine 3287436db63758 [web] update finished: success
Finished deploying
Visit your newly deployed app at https://client-portal-development.fly.dev/
- Output of the suspended machine (
worker-pdf)
2023-06-30T17:07:44.466 runner[1781155ad41189] iad [info] Pulling container image registry.fly.io/client-portal-development:deployment-01H46M49VBYM6HY78Y0XWJ73HQ
2023-06-30T17:08:03.917 runner[1781155ad41189] iad [info] Successfully prepared image registry.fly.io/client-portal-development:deployment-01H46M49VBYM6HY78Y0XWJ73HQ (19.45091761s)
2023-06-30T17:08:04.195 runner[1781155ad41189] iad [info] Configuring firecracker
2023-06-30T17:08:04.200 app[1781155ad41189] iad [info] INFO Sending signal SIGINT to main child process w/ PID 232
2023-06-30T17:08:04.346 app[1781155ad41189] iad [info] INFO Main child exited with signal (with signal 'SIGINT', core dumped? false)
2023-06-30T17:08:04.346 app[1781155ad41189] iad [info] INFO Starting clean up.
2023-06-30T17:08:04.347 app[1781155ad41189] iad [info] WARN hallpass exited, pid: 233, status: signal: 15 (SIGTERM)
2023-06-30T17:08:04.348 app[1781155ad41189] iad [info] 2023/06/30 17:08:04 listening on [fdaa:1:b784:a7b:ab8:6033:4b56:2]:22 (DNS: [fdaa::3]:53)
2023-06-30T17:08:04.399 app[1781155ad41189] iad [info] npm notice
2023-06-30T17:08:04.399 app[1781155ad41189] iad [info] npm notice New major version of npm available! 8.19.4 -> 9.7.2
2023-06-30T17:08:04.399 app[1781155ad41189] iad [info] npm notice Changelog: <https://github.com/npm/cli/releases/tag/v9.7.2>
2023-06-30T17:08:04.400 app[1781155ad41189] iad [info] npm notice Run `npm install -g npm@9.7.2` to update!
2023-06-30T17:08:04.400 app[1781155ad41189] iad [info] npm notice
2023-06-30T17:08:05.340 app[1781155ad41189] iad [info] [ 437.658016] reboot: Restarting system
It has hung for over an hour now showing this.
Oddly, the other worker machine (worker-email, a very similar process in make-up), shows the same output but doesn’t appear to hang up on reboot: Restarting system.
Finally, running a fly machine start 1781155ad41189 reboots the app and it begins correctly.
... (continuing from above)
2023-06-30T17:08:22.020 app[3d8d501fe51e98] iad [info] INFO Starting init (commit: db101a53)...
2023-06-30T17:08:22.200 app[3d8d501fe51e98] iad [info] INFO Preparing to run: `docker-entrypoint.sh yarn worker:email` as nextjs
2023-06-30T17:08:22.379 app[3d8d501fe51e98] iad [info] INFO [fly api proxy] listening at /.fly/api
2023-06-30T17:08:22.382 app[3d8d501fe51e98] iad [info] 2023/06/30 17:08:22 listening on [fdaa:1:b784:a7b:93:a57d:109f:2]:22 (DNS: [fdaa::3]:53)
2023-06-30T17:08:24.088 app[3d8d501fe51e98] iad [info] yarn run v1.22.19
2023-06-30T17:08:24.118 app[3d8d501fe51e98] iad [info] $ npx ts-node --compiler-options '{"module":"CommonJS"}' ./src/workers/email.ts
2023-06-30T17:08:43.385 app[3d8d501fe51e98] iad [info] (node:271) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023.
2023-06-30T17:08:43.385 app[3d8d501fe51e98] iad [info] Please migrate your code to use AWS SDK for JavaScript (v3).
2023-06-30T17:08:43.385 app[3d8d501fe51e98] iad [info] For more information, check the migration guide at https://a.co/7PzMCcy
2023-06-30T17:08:43.385 app[3d8d501fe51e98] iad [info] (Use `node --trace-warnings ...` to show where the warning was created)
Any help is appreciated, thanks.