I’ve created a super simple node server which automatically shuts itself down after a time of inactivity. The machine then restarts on requests. This worked fine for a while, with the app being paused/off most of the time.
Today when I looked it seems to have been running for a couple of days with zero downtime, and the monitor shows that the machine restarts the second it shuts down, being stuck in an infinite restart loop.
Is this a bug, or is there some setting I have missed in the command-line interface?
Closing server due to inactivity
Starting clean up.
Umounting /dev/vdb from /opt/app/data
reboot: Restarting system
Can you let us know how you created this app? (Machines API, via flyctl’s fly launch or whatever you used. Do you have a fly.toml?)
Your grafana graphs should show HTTP responses from your application that the proxy received after forwarding inbound requests (scroll down and expand it to see a count of HTTP responses). That should help correlate requests turning on the application.
Assuming the application is a web app responding to HTTP requests, does it output any logs per request? The logs for the app should show some hints on what’s going on, unless the app is “silent” (not programed to log requests it receives).
I have re-created the app a few times, but the problem still persists. Here’s the output:
Machine ID: R
Instance ID: T
State: started
VM
ID = X
Instance ID = Y
State = started
Image = tileset-explorer:deployment-Z
Name = white-sky-7396
Private IP = W
Region = cdg
Process Group =
CPU Kind = shared
vCPUs = 1
Memory = 256
Created = 2023-03-19T11:11:33Z
Updated = 2023-03-20T17:30:18Z
Command =
Volume = V
Event Logs
STATE EVENT SOURCE TIMESTAMP INFO
started start flyd 2023-03-20T19:30:18.807+02:00
starting start flyd 2023-03-20T19:30:18.44+02:00
stopped exit flyd 2023-03-20T19:30:18.339+02:00 exit_code=0,oom_killed=false,requested_stop=false
started start flyd 2023-03-20T19:25:14.217+02:00
starting start flyd 2023-03-20T19:25:13.779+02:00
I think the restart policy on your machine may be set to always. In that case, flyd will attempt to restart your machine even when your app exits successfully (which is what’s happening, per the exit_code in the status output you posted).
Could you try setting your machine’s restart policy to on-fail or no (depending on the behavior you want)? E.g. flyctl machine update <machine id> --restart no. If it shows a pending change from always, that might be it.
(There have been some changes in flyctl having to do with machine restart policies in the last two or three weeks. In particular, we fixed a bug in which flyctl machine update would always set the restart policy to always, overwriting any previous setting. I wonder if you were bitten by this.)
Out of curiosity, are there any wake-on-request docs (or other machine docs) that you’ve been referencing while working on this? If anything is out of date or misleading, I’d like to look into it.
I have been following the tutorial on how to setup a wake-on-request machine using the tired proxy. Since the only thing that proxy does is shutting down the server when idle, I just implemented something similar in node. Those docs do not mention having to configure the machine to use wake-on-request.