App restarting every 5 minutes

1

Hi,
Without any change with my fly.toml, my app is restarting every 5 minutes, even when users are using it.
Here are my logs :

2023-08-09T13:10:20.876 health[d89109dc641d68] cdg [info] Health check on port 8080 is now passing.

2023-08-09T13:15:55.377 proxy [d89109dc641d68] cdg [info] Downscaling app display-prod4 in region cdg from 1 machines to 0 machines. Automatically stopping machine d89109dc641d68

2023-08-09T13:15:55.380 app[d89109dc641d68] cdg [info] 13:15:55.379 [notice] SIGTERM received - shutting down

2023-08-09T13:15:55.380 app[d89109dc641d68] cdg [info] 13:15:55.380 [info] Shutting down 1 sockets in 1 rounds of 2000ms

2023-08-09T13:15:55.388 app[d89109dc641d68] cdg [info] INFO Sending signal SIGTERM to main child process w/ PID 256

2023-08-09T13:15:56.262 app[d89109dc641d68] cdg [info] WARN Reaped child process with pid: 316 and signal: SIGUSR1, core dumped? false

2023-08-09T13:15:57.267 app[d89109dc641d68] cdg [info] INFO Main child exited normally with code: 0

2023-08-09T13:15:57.268 app[d89109dc641d68] cdg [info] INFO Starting clean up.

2023-08-09T13:15:57.268 app[d89109dc641d68] cdg [info] WARN hallpass exited, pid: 257, status: signal: 15 (SIGTERM)

2023-08-09T13:15:57.269 app[d89109dc641d68] cdg [info] 2023/08/09 13:15:57 listening on [fdaa:0:ef46:a7b:126:2311:5701:2]:22 (DNS: [fdaa::3]:53)

2023-08-09T13:15:58.265 app[d89109dc641d68] cdg [info] [ 353.638136] reboot: Restarting system

2023-08-09T13:16:04.820 proxy[d89109dc641d68] cdg [info] Starting machine

2023-08-09T13:16:04.991 app[d89109dc641d68] cdg [info] [ 0.050485] PCI: Fatal: No config space access function found

2023-08-09T13:16:05.172 app[d89109dc641d68] cdg [info] INFO Starting init (commit: b437b5b)...

2023-08-09T13:16:05.182 app[d89109dc641d68] cdg [info] INFO Preparing to run: `/app/bin/server` as nobody

2023-08-09T13:16:05.188 app[d89109dc641d68] cdg [info] INFO [fly api proxy] listening at /.fly/api

2023-08-09T13:16:05.196 app[d89109dc641d68] cdg [info] 2023/08/09 13:16:05 listening on [fdaa:0:ef46:a7b:126:2311:5701:2]:22 (DNS: [fdaa::3]:53)

2023-08-09T13:16:05.224 proxy[d89109dc641d68] cdg [info] machine started in 403.524921ms

2023-08-09T13:16:07.828 app[d89109dc641d68] cdg [info] 13:16:07.827 [info] Running DisplayWeb.Endpoint with cowboy 2.10.0 at :::8080 (http)

2023-08-09T13:16:07.831 app[d89109dc641d68] cdg [info] 13:16:07.830 [info] Access DisplayWeb.Endpoint at https://display-entreprise.com

2023-08-09T13:16:08.198 app[d89109dc641d68] cdg [info] WARN Reaped child process with pid: 314 and signal: SIGUSR1, core dumped? false

2023-08-09T13:16:08.535 proxy[d89109dc641d68] cdg [info] machine became reachable in 3.311157624s

2023-08-09T13:16:08.537 app[d89109dc641d68] cdg [info] 13:16:08.535 request_id=F3m5Yq5B3ZeR71QAAAEx [info] GET /main

2023-08-09T13:16:08.552 app[d89109dc641d68] cdg [info] 13:16:08.552 request_id=F3m5Yq5B3ZeR71QAAAEx [info] Sent 200 in 16ms

2023-08-09T13:16:08.775 app[d89109dc641d68] cdg [info] 13:16:08.775 [info] CONNECTED TO Phoenix.LiveView.Socket in 30µs

2023-08-09T13:16:08.775 app[d89109dc641d68] cdg [info] Transport: :websocket

2023-08-09T13:16:08.775 app[d89109dc641d68] cdg [info] Serializer: Phoenix.Socket.V2.JSONSerializer

2023-08-09T13:16:08.775 app[d89109dc641d68] cdg [info] Parameters: %{"_csrf_token" => "bzhSJBoXABpYIw8cGDY5egwwbAI4MSQK0_5NRmdXnmAXHep54cTFR_pR", "_live_referer" => "undefined", "_mounts" => "0", "_track_static" => %{"0" => "https://display-entreprise.com/assets/app-ce064f090ea9f30f31f4c266f1bc9536.css?vsn=d", "1" => "https://display-entreprise.com/assets/app-a33aaa8752ca2a3963d3b46a063a5158.js?vsn=d"}, "vsn" => "2.0.0"}

2023-08-09T13:16:11.411 app[d89109dc641d68] cdg [info] 13:16:11.410 [info] tzdata release in place is from a file last modified Fri, 22 Oct 2021 02:20:47 GMT. Release file on server was last modified Tue, 28 Mar 2023 20:25:39 GMT.

2023-08-09T13:16:12.446 app[d89109dc641d68] cdg [info] 13:16:12.445 [info] Tzdata has updated the release from 2021e to 2023c

2023-08-09T13:21:55.419 proxy [d89109dc641d68] cdg [info] Downscaling app display-prod4 in region cdg from 1 machines to 0 machines. Automatically stopping machine d89109dc641d68

2023-08-09T13:21:55.424 app[d89109dc641d68] cdg [info] 13:21:55.424 [notice] SIGTERM received - shutting down

2023-08-09T13:21:55.425 app[d89109dc641d68] cdg [info] 13:21:55.424 [info] Shutting down 1 sockets in 1 rounds of 2000ms

2023-08-09T13:21:55.432 app[d89109dc641d68] cdg [info] INFO Sending signal SIGTERM to main child process w/ PID 256

2023-08-09T13:21:55.600 app[d89109dc641d68] cdg [info] WARN Reaped child process with pid: 316 and signal: SIGUSR1, core dumped? false

2023-08-09T13:21:56.604 app[d89109dc641d68] cdg [info] INFO Main child exited normally with code: 0

2023-08-09T13:21:56.605 app[d89109dc641d68] cdg [info] INFO Starting clean up.

2023-08-09T13:21:56.605 app[d89109dc641d68] cdg [info] WARN hallpass exited, pid: 257, status: signal: 15 (SIGTERM)

2023-08-09T13:21:56.606 app[d89109dc641d68] cdg [info] 2023/08/09 13:21:56 listening on [fdaa:0:ef46:a7b:126:2311:5701:2]:22 (DNS: [fdaa::3]:53)

2023-08-09T13:21:57.252 health[d89109dc641d68] cdg [error] Health check on port 8080 has failed. Your app is not responding properly. Services exposed on ports [80, 443, 5432] will have intermittent failures until the health check passes.

2023-08-09T13:21:57.602 app[d89109dc641d68] cdg [info] [ 352.642561] reboot: Restarting system

2023-08-09T13:22:04.083 proxy[d89109dc641d68] cdg [info] Starting machine

2023-08-09T13:22:04.248 app[d89109dc641d68] cdg [info] [ 0.048081] PCI: Fatal: No config space access function found

2023-08-09T13:22:04.401 app[d89109dc641d68] cdg [info] INFO Starting init (commit: b437b5b)...

2023-08-09T13:22:04.411 app[d89109dc641d68] cdg [info] INFO Preparing to run: `/app/bin/server` as nobody

2023-08-09T13:22:04.416 app[d89109dc641d68] cdg [info] INFO [fly api proxy] listening at /.fly/api

2023-08-09T13:22:04.424 app[d89109dc641d68] cdg [info] 2023/08/09 13:22:04 listening on [fdaa:0:ef46:a7b:126:2311:5701:2]:22 (DNS: [fdaa::3]:53)

2023-08-09T13:22:04.459 proxy[d89109dc641d68] cdg [info] machine started in 376.475691ms

2023-08-09T13:22:07.078 app[d89109dc641d68] cdg [info] 13:22:07.078 [info] Running DisplayWeb.Endpoint with cowboy 2.10.0 at :::8080 (http)

2023-08-09T13:22:07.081 app[d89109dc641d68] cdg [info] 13:22:07.080 [info] Access DisplayWeb.Endpoint at https://display-entreprise.com

2023-08-09T13:22:07.428 app[d89109dc641d68] cdg [info] WARN Reaped child process with pid: 314 and signal: SIGUSR1, core dumped? false

2023-08-09T13:22:08.069 proxy[d89109dc641d68] cdg [info] machine became reachable in 3.61013116s
2

This seems to be normal autostop behavior when no new connections are seen in a few minutes. Can you please check your fly.toml and check whether you have:

auto_start_machines = true
auto_stop_machines = true

If so, you could try two things:

  • Specify min_machines_running = 1 or some other number to always keep machines running.
  • Set the two settings I mentioned above to false to always keep machines running.

Cheers,

  • Daniel
1 Like
3

Hi @roadmr ,
Thx for your reply.
Here are my fly.toml config concerning auto_stop/start :

  auto_stop_machines = true
  auto_start_machines = true

I don’t want to set a min_machines_running = 0 as I have very little traffic on my website and I don’t want it to be always started.

What i don’t understand is :

  • why does the app restart as I’m currently using it (browsing pages, etc.) ?
  • why does it restart and not just stop (as if it was the normal autostop behavior) ?

It worked perfectly find until this morning (when i noticed it)

Thx for your time!

Thomas

4

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.