Suddenly suspended machine with SIGTERM signal shutting down

Hey there,

My app is down for no obvious reason and can not bring it back up. It’s in “suspended” state.

The logs indicate that a SIGTERM kill signal is sent during the start. Not sure what’s causing this, I haven’t deployed any change in past 2.5 weeks.

Machine Id: 21781955a10e89

I’m not sure how to address this and it’s blocking me from demo’ing my app to customers tomorrow.

Logs:


2023-07-09T09:17:42.673 app[21781955a10e89] dfw [info] [1686859.631179] reboot: Restarting system

2023-07-09T09:17:43.216 app[21781955a10e89] dfw [info] INFO Starting init (commit: 0b28cec)...

2023-07-09T09:17:43.232 app[21781955a10e89] dfw [info] INFO Preparing to run: `/app/bin/server` as nobody

2023-07-09T09:17:43.242 app[21781955a10e89] dfw [info] INFO [fly api proxy] listening at /.fly/api

2023-07-09T09:17:43.260 app[21781955a10e89] dfw [info] 2023/07/09 09:17:43 listening on [fdaa:0:c354:a7b:cf99:b54b:a166:2]:22 (DNS: [fdaa::3]:53)

2023-07-09T09:17:44.249 app[21781955a10e89] dfw [info] WARN Reaped child process with pid: 549, exit code: 0

2023-07-09T09:17:45.533 health[21781955a10e89] dfw [error] Health check on port 8080 has failed. Your app is not responding properly. Services exposed on ports [80, 443] will have intermittent failures until the health check passes.

2023-07-09T09:17:46.252 app[21781955a10e89] dfw [info] WARN Reaped child process with pid: 571 and signal: SIGUSR1, core dumped? false

2023-07-09T09:17:46.432 app[21781955a10e89] dfw [info] 09:17:46.431 [info] Running SwayWeb.Endpoint with cowboy 2.9.0 at :::8080 (http)

2023-07-09T09:17:46.436 app[21781955a10e89] dfw [info] 09:17:46.435 [info] Access SwayWeb.Endpoint at https://sway.so

2023-07-09T09:17:49.681 app[21781955a10e89] dfw [info] 09:17:49.681 [info] tzdata release in place is from a file last modified Fri, 22 Oct 2021 02:20:47 GMT. Release file on server was last modified Tue, 28 Mar 2023 20:25:39 GMT.

2023-07-09T09:17:50.477 app[21781955a10e89] dfw [info] 09:17:50.475 [info] Tzdata has updated the release from 2021e to 2023c

2023-07-09T09:18:00.533 health[21781955a10e89] dfw [info] Health check on port 8080 is now passing.

2023-07-09T09:24:07.509 app[21781955a10e89] dfw [info] INFO Sending signal SIGTERM to main child process w/ PID 513

2023-07-09T09:24:07.510 app[21781955a10e89] dfw [info] 09:24:07.509 [notice] SIGTERM received - shutting down

2023-07-09T09:24:07.704 app[21781955a10e89] dfw [info] WARN Reaped child process with pid: 573 and signal: SIGUSR1, core dumped? false

2023-07-09T09:24:08.706 app[21781955a10e89] dfw [info] INFO Main child exited normally with code: 0

2023-07-09T09:24:08.706 app[21781955a10e89] dfw [info] INFO Starting clean up.

2023-07-09T09:24:08.707 app[21781955a10e89] dfw [info] WARN hallpass exited, pid: 514, status: signal: 15 (SIGTERM)

2023-07-09T09:24:08.711 app[21781955a10e89] dfw [info] 2023/07/09 09:24:08 listening on [fdaa:0:c354:a7b:cf99:b54b:a166:2]:22 (DNS: [fdaa::3]:53)

2023-07-09T09:24:09.706 app[21781955a10e89] dfw [info] [ 386.565714] reboot: Restarting system

2023-07-09T09:24:10.200 app[21781955a10e89] dfw [info] INFO Starting init (commit: 0b28cec)...

2023-07-09T09:24:10.213 app[21781955a10e89] dfw [info] INFO Preparing to run: `/app/bin/server` as nobody

2023-07-09T09:24:10.222 app[21781955a10e89] dfw [info] INFO [fly api proxy] listening at /.fly/api

2023-07-09T09:24:10.241 app[21781955a10e89] dfw [info] 2023/07/09 09:24:10 listening on [fdaa:0:c354:a7b:cf99:b54b:a166:2]:22 (DNS: [fdaa::3]:53)

2023-07-09T09:24:11.234 app[21781955a10e89] dfw [info] WARN Reaped child process with pid: 550, exit code: 0

2023-07-09T09:24:13.238 app[21781955a10e89] dfw [info] WARN Reaped child process with pid: 572 and signal: SIGUSR1, core dumped? false

2023-07-09T09:24:13.369 app[21781955a10e89] dfw [info] 09:24:13.368 [info] Running SwayWeb.Endpoint with cowboy 2.9.0 at :::8080 (http)

2023-07-09T09:24:13.372 app[21781955a10e89] dfw [info] 09:24:13.372 [info] Access SwayWeb.Endpoint at https://sway.so

2023-07-09T09:24:16.418 app[21781955a10e89] dfw [info] 09:24:16.418 [info] tzdata release in place is from a file last modified Fri, 22 Oct 2021 02:20:47 GMT. Release file on server was last modified Tue, 28 Mar 2023 20:25:39 GMT.

2023-07-09T09:24:17.211 app[21781955a10e89] dfw [info] 09:24:17.210 [info] Tzdata has updated the release from 2021e to 2023

Support is much appreciated.

Thanks

Saw this one in the dashboard:

RSA
Issued
Expired at July 6, 2023

ECDSA
Issued
Expired at July 4, 2023

It looks like it was a problem with certificates. After running bunch of certs commands w/ flyctl, the expired certificates were renewed and problem is solved now.

However, The app still looks “suspended” in the flyctl and Fly dashboard. Deployment is still too slow and couldn’t complete one yet.

My understanding is: certificates should have been renewed automatically before expiration but they were not. And only listing / showing certificates trigger the renewal.

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.