Machine fails to start after deploy


After every deploy, one of my machines (the worker-pdf process below) stops without a failure, the logs just seem to halt. Running a quick fly machine start <id> gets it up and running, but it is annoying that it pauses on it’s own with no signal of a failure.

app = "client-portal-development"
primary_region = "iad"
kill_signal = "SIGINT"
kill_timeout = "5s"

  release_command = "npx prisma migrate deploy && yarn run-data-migrations"

  auto_rollback = true

  PORT = "8080"

  web = "yarn start"
  worker-email= "yarn worker:email"
  worker-pdf= "yarn worker:pdf"

  protocol = "tcp"
  internal_port = 8080
  processes = ["web"]

    port = 80
    handlers = ["http"]
    force_https = true

    port = 443
    handlers = ["tls", "http"]

    type = "connections"
    hard_limit = 25
    soft_limit = 20

    interval = "15s"
    timeout = "2s"
    grace_period = "1s"
    restart_limit = 0

Repro steps:

  1. Check machines
3 machines have been retrieved from app client-portal-development.
View them in the UI here (​...)

ID              NAME                    STATE   REGION  IMAGE                                                           IP ADDRESS                      VOLUME  CREATED                 LAST UPDATED            APP PLATFORM    PROCESS GROUP   SIZE                 
1781155ad41189  crimson-wind-9803       started iad     client-portal-development:deployment-01H46JA9HHQEH9NAES1E93VK9H fdaa:1:b784:a7b:ab8:6033:4b56:2         2023-06-29T20:29:19Z    2023-06-30T17:00:47Z    v2              worker-pdf      shared-cpu-1x:2048MB
3d8d501fe51e98  falling-dream-2883      started iad     client-portal-development:deployment-01H46JA9HHQEH9NAES1E93VK9H fdaa:1:b784:a7b:93:a57d:109f:2          2023-06-29T20:29:19Z    2023-06-30T16:36:31Z    v2              worker-email    shared-cpu-1x:2048MB
3287436db63758  hidden-bush-7456        started iad     client-portal-development:deployment-01H46JA9HHQEH9NAES1E93VK9H fdaa:1:b784:a7b:143:f481:de07:2         2023-05-08T17:08:30Z    2023-06-30T16:37:10Z    v2              web             shared-cpu-1x:256MB 
  1. Deploy
$ fly deploy -c
Running client-portal-development release_command: npx prisma migrate deploy && yarn run-data-migrations
  release_command 90800d5b16de18 completed successfully
Updating existing machines in 'client-portal-development' with rolling strategy
  [1/3] Machine 1781155ad41189 [worker-pdf] update finished: success
  [2/3] Machine 3d8d501fe51e98 [worker-email] update finished: success
  [3/3] Machine 3287436db63758 [web] update finished: success
  Finished deploying

Visit your newly deployed app at
  1. Output of the suspended machine (worker-pdf)
2023-06-30T17:07:44.466 runner[1781155ad41189] iad [info] Pulling container image
2023-06-30T17:08:03.917 runner[1781155ad41189] iad [info] Successfully prepared image (19.45091761s)
2023-06-30T17:08:04.195 runner[1781155ad41189] iad [info] Configuring firecracker
2023-06-30T17:08:04.200 app[1781155ad41189] iad [info] INFO Sending signal SIGINT to main child process w/ PID 232
2023-06-30T17:08:04.346 app[1781155ad41189] iad [info] INFO Main child exited with signal (with signal 'SIGINT', core dumped? false)
2023-06-30T17:08:04.346 app[1781155ad41189] iad [info] INFO Starting clean up.
2023-06-30T17:08:04.347 app[1781155ad41189] iad [info] WARN hallpass exited, pid: 233, status: signal: 15 (SIGTERM)
2023-06-30T17:08:04.348 app[1781155ad41189] iad [info] 2023/06/30 17:08:04 listening on [fdaa:1:b784:a7b:ab8:6033:4b56:2]:22 (DNS: [fdaa::3]:53)
2023-06-30T17:08:04.399 app[1781155ad41189] iad [info] npm notice
2023-06-30T17:08:04.399 app[1781155ad41189] iad [info] npm notice New major version of npm available! 8.19.4 -> 9.7.2
2023-06-30T17:08:04.399 app[1781155ad41189] iad [info] npm notice Changelog: <>
2023-06-30T17:08:04.400 app[1781155ad41189] iad [info] npm notice Run `npm install -g npm@9.7.2` to update!
2023-06-30T17:08:04.400 app[1781155ad41189] iad [info] npm notice
2023-06-30T17:08:05.340 app[1781155ad41189] iad [info] [ 437.658016] reboot: Restarting system

It has hung for over an hour now showing this.

Oddly, the other worker machine (worker-email, a very similar process in make-up), shows the same output but doesn’t appear to hang up on reboot: Restarting system.

Finally, running a fly machine start 1781155ad41189 reboots the app and it begins correctly.

... (continuing from above)
2023-06-30T17:08:22.020 app[3d8d501fe51e98] iad [info] INFO Starting init (commit: db101a53)...
2023-06-30T17:08:22.200 app[3d8d501fe51e98] iad [info] INFO Preparing to run: ` yarn worker:email` as nextjs
2023-06-30T17:08:22.379 app[3d8d501fe51e98] iad [info] INFO [fly api proxy] listening at /.fly/api
2023-06-30T17:08:22.382 app[3d8d501fe51e98] iad [info] 2023/06/30 17:08:22 listening on [fdaa:1:b784:a7b:93:a57d:109f:2]:22 (DNS: [fdaa::3]:53)
2023-06-30T17:08:24.088 app[3d8d501fe51e98] iad [info] yarn run v1.22.19
2023-06-30T17:08:24.118 app[3d8d501fe51e98] iad [info] $ npx ts-node --compiler-options '{"module":"CommonJS"}' ./src/workers/email.ts
2023-06-30T17:08:43.385 app[3d8d501fe51e98] iad [info] (node:271) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023.
2023-06-30T17:08:43.385 app[3d8d501fe51e98] iad [info] Please migrate your code to use AWS SDK for JavaScript (v3).
2023-06-30T17:08:43.385 app[3d8d501fe51e98] iad [info] For more information, check the migration guide at
2023-06-30T17:08:43.385 app[3d8d501fe51e98] iad [info] (Use `node --trace-warnings ...` to show where the warning was created)

Any help is appreciated, thanks.

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.