Machine shutting down while still running a function

I have a runner that gets me lots of ga data (function takes like 10 min to complete. After like 2-4 minutes the machine always shuts down

2023-05-03T12:27:04.266 app[e784e900c27598] ams [info] Running benchmark for 2022-03-01 to 2022-04-30

2023-05-03T12:27:04.266 app[e784e900c27598] ams [info] Running main benchmark runnner for view 162466809

2023-05-03T12:27:04.266 app[e784e900c27598] ams [info] Running benchmark for ga:userType

2023-05-03T12:27:04.266 app[e784e900c27598] ams [info] 2023-05-03 12:27:04,267:INFO - Running main benchmark runnner for view 162466809

2023-05-03T12:27:04.266 app[e784e900c27598] ams [info] 2023-05-03 12:27:04,267:INFO - Running benchmark for ga:userType

2023-05-03T12:27:24.953 proxy [e784e903f27008] ams [info] Downscaling app drip-gaia-prod in region ams. Automatically stopping machine e784e903f27008. 2 instances are running, 0 are at soft limit, we only need 1 running

2023-05-03T12:27:24.957 app[e784e903f27008] ams [info] Sending signal SIGINT to main child process w/ PID 526

2023-05-03T12:27:24.991 app[e784e903f27008] ams [info] INFO: Shutting down

2023-05-03T12:27:25.091 app[e784e903f27008] ams [info] INFO: Waiting for application shutdown.

2023-05-03T12:27:25.091 app[e784e903f27008] ams [info] INFO: Application shutdown complete.

2023-05-03T12:27:25.092 app[e784e903f27008] ams [info] INFO: Finished server process [526]

2023-05-03T12:27:26.097 app[e784e903f27008] ams [info] Starting clean up.

2023-05-03T12:27:27.098 app[e784e903f27008] ams [info] [ 421.705943] reboot: Restarting system

2023-05-03T12:28:16.711 proxy [e784e900c27598] ams [info] Downscaling app drip-gaia-prod in region ams. Automatically stopping machine e784e900c27598. 1 instance is running but has no load

2023-05-03T12:28:16.713 app[e784e900c27598] ams [info] Sending signal SIGINT to main child process w/ PID 526

2023-05-03T12:28:16.768 app[e784e900c27598] ams [info] INFO: Shutting down

2023-05-03T12:28:16.868 app[e784e900c27598] ams [info] INFO: Waiting for background tasks to complete. (CTRL+C to force quit)

2023-05-03T12:28:21.760 app[e784e900c27598] ams [info] Sending signal SIGTERM to main child process w/ PID 526

2023-05-03T12:28:26.779 app[e784e900c27598] ams [warn] Virtual machine exited abruptly

my fly.toml looks like this:

primary_region = "ams"
kill_signal = "SIGINT"
kill_timeout = 5



[http_service]
  internal_port = 8080
  force_https = true

[[services]]
  protocol = "tcp"
  internal_port = 8080
  processes = ["app"]

  [[services.ports]]
    port = 80
    handlers = ["http"]
    force_https = true

Any ideas on where to start debugging or potential fixed? I already set

timeout_in_sec = 60 * 60
socket.setdefaulttimeout(timeout_in_sec) 

But still no luck

1 Like

Hi Fabian1, just making sure, but were any of your machines run with the --autostop flyctl flag? If so, then your machines will get stopped automatically when there aren’t any network requests for a few minutes

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.