I don’t know if this is related or not, but for exploration purposes I tried suspending and resuming my sm-sandbox test server, and then it got stuck so I bumped the count up to 2 instances. It won’t start. See the bottom of these logs, where the health check on both instances is ‘passing’ before it starts failing. I doubled the grace_period just to be sure that it was having enough time to start. This is the same image that is running just fine (for the time being) on sm-prod, and was OK on sm-beta until recently…
2021-02-04T03:24:59.343Z d88e554c sjc [info] Shutting down virtual machine
2021-02-04T03:24:59.419Z d88e554c sjc [info] Sending signal SIGTERM to main child process w/ PID 504
2021-02-04T03:25:00.421Z d88e554c sjc [info] Reaped child process with pid: 504 and signal: SIGTERM
2021-02-04T03:25:02.349Z d88e554c sjc [info] Starting instance
2021-02-04T03:25:02.369Z d88e554c sjc [info] Configuring virtual machine
2021-02-04T03:25:02.371Z d88e554c sjc [info] Pulling container image
2021-02-04T03:25:02.805Z d88e554c sjc [info] Unpacking image
2021-02-04T03:25:02.810Z d88e554c sjc [info] Preparing kernel init
2021-02-04T03:25:03.201Z d88e554c sjc [info] Configuring firecracker
2021-02-04T03:25:03.279Z d88e554c sjc [info] Starting virtual machine
2021-02-04T03:25:03.406Z d88e554c sjc [info] Starting init (commit: 882f7ea)...
2021-02-04T03:25:03.430Z d88e554c sjc [info] Running: `docker-entrypoint.sh node main.js` as root
2021-02-04T03:25:03.437Z d88e554c sjc [info] can't run hallpass: an unhandled IO error occurred: 0
2021-02-04T03:25:06.115Z d88e554c sjc [info] Monti APM: Successfully connected
2021-02-04T03:25:21.725Z d88e554c sjc [info] Shutting down virtual machine
2021-02-04T03:25:21.802Z d88e554c sjc [info] Sending signal SIGTERM to main child process w/ PID 504
2021-02-04T03:25:22.804Z d88e554c sjc [info] Reaped child process with pid: 504 and signal: SIGTERM
2021-02-04T03:25:25.308Z f66021b1 vin [info] Starting instance
2021-02-04T03:25:25.345Z f66021b1 vin [info] Configuring virtual machine
2021-02-04T03:25:25.346Z f66021b1 vin [info] Pulling container image
2021-02-04T03:25:25.566Z f66021b1 vin [info] Unpacking image
2021-02-04T03:25:25.575Z f66021b1 vin [info] Preparing kernel init
2021-02-04T03:25:25.877Z f66021b1 vin [info] Configuring firecracker
2021-02-04T03:25:26.038Z f66021b1 vin [info] Starting virtual machine
2021-02-04T03:25:26.198Z f66021b1 vin [info] Starting init (commit: 882f7ea)...
2021-02-04T03:25:26.228Z f66021b1 vin [info] Running: `docker-entrypoint.sh node main.js` as root
2021-02-04T03:25:26.239Z f66021b1 vin [info] can't run hallpass: an unhandled IO error occurred: 0
2021-02-04T03:25:26.470Z d88e554c sjc [info] Health check status changed to 'passing'
2021-02-04T03:25:28.445Z 25d4d56e ewr [info] Starting instance
2021-02-04T03:25:28.462Z 25d4d56e ewr [info] Configuring virtual machine
2021-02-04T03:25:28.463Z 25d4d56e ewr [info] Pulling container image
2021-02-04T03:25:28.595Z 25d4d56e ewr [info] Unpacking image
2021-02-04T03:25:28.598Z 25d4d56e ewr [info] Preparing kernel init
2021-02-04T03:25:28.955Z 25d4d56e ewr [info] Configuring firecracker
2021-02-04T03:25:28.969Z 25d4d56e ewr [info] Starting virtual machine
2021-02-04T03:25:29.098Z 25d4d56e ewr [info] Starting init (commit: 882f7ea)...
2021-02-04T03:25:29.099Z f66021b1 vin [info] Monti APM: Successfully connected
2021-02-04T03:25:29.119Z 25d4d56e ewr [info] Running: `docker-entrypoint.sh node main.js` as root
2021-02-04T03:25:29.126Z 25d4d56e ewr [info] can't run hallpass: an unhandled IO error occurred: 0
2021-02-04T03:25:31.256Z 25d4d56e ewr [info] Monti APM: Successfully connected
2021-02-04T03:25:55.266Z f66021b1 vin [info] Starting Meteor server on f66021b1
2021-02-04T03:25:56.106Z f66021b1 vin [info] Monti APM: completed instrumenting the app
2021-02-04T03:25:56.536Z f66021b1 vin [info] Strum Machine v333.0 now serving sandbox.strummachine.com on f66021b1
2021-02-04T03:25:58.296Z 25d4d56e ewr [info] Starting Meteor server on 25d4d56e
2021-02-04T03:25:59.004Z 25d4d56e ewr [info] Monti APM: completed instrumenting the app
2021-02-04T03:25:59.396Z 25d4d56e ewr [info] Strum Machine v333.0 now serving sandbox.strummachine.com on 25d4d56e
2021-02-04T03:26:04.042Z f66021b1 vin [info] Shutting down virtual machine
2021-02-04T03:26:04.064Z f66021b1 vin [info] Sending signal SIGTERM to main child process w/ PID 504
2021-02-04T03:26:05.074Z f66021b1 vin [info] Reaped child process with pid: 504 and signal: SIGTERM
2021-02-04T03:26:05.246Z 25d4d56e ewr [info] Shutting down virtual machine
2021-02-04T03:26:05.264Z 25d4d56e ewr [info] Sending signal SIGTERM to main child process w/ PID 504
2021-02-04T03:26:06.266Z 25d4d56e ewr [info] Reaped child process with pid: 504 and signal: SIGTERM
2021-02-04T03:26:07.419Z f66021b1 vin [info] Starting instance
2021-02-04T03:26:07.455Z f66021b1 vin [info] Configuring virtual machine
2021-02-04T03:26:07.457Z f66021b1 vin [info] Pulling container image
2021-02-04T03:26:07.612Z f66021b1 vin [info] Unpacking image
2021-02-04T03:26:07.626Z f66021b1 vin [info] Preparing kernel init
2021-02-04T03:26:07.867Z f66021b1 vin [info] Configuring firecracker
2021-02-04T03:26:07.892Z f66021b1 vin [info] Starting virtual machine
2021-02-04T03:26:08.043Z f66021b1 vin [info] Starting init (commit: 882f7ea)...
2021-02-04T03:26:08.079Z f66021b1 vin [info] Running: `docker-entrypoint.sh node main.js` as root
2021-02-04T03:26:08.088Z f66021b1 vin [info] can't run hallpass: an unhandled IO error occurred: 0
2021-02-04T03:26:08.345Z 25d4d56e ewr [info] Starting instance
2021-02-04T03:26:08.361Z 25d4d56e ewr [info] Configuring virtual machine
2021-02-04T03:26:08.362Z 25d4d56e ewr [info] Pulling container image
2021-02-04T03:26:08.569Z 25d4d56e ewr [info] Unpacking image
2021-02-04T03:26:08.574Z 25d4d56e ewr [info] Preparing kernel init
2021-02-04T03:26:08.920Z 25d4d56e ewr [info] Configuring firecracker
2021-02-04T03:26:08.939Z 25d4d56e ewr [info] Starting virtual machine
2021-02-04T03:26:09.056Z 25d4d56e ewr [info] Starting init (commit: 882f7ea)...
2021-02-04T03:26:09.076Z 25d4d56e ewr [info] Running: `docker-entrypoint.sh node main.js` as root
2021-02-04T03:26:09.083Z 25d4d56e ewr [info] can't run hallpass: an unhandled IO error occurred: 0
2021-02-04T03:26:11.291Z f66021b1 vin [info] Monti APM: Successfully connected
2021-02-04T03:26:11.394Z 25d4d56e ewr [info] Monti APM: Successfully connected
2021-02-04T03:26:16.603Z f66021b1 vin [info] Health check status changed to 'passing'
2021-02-04T03:26:17.740Z 25d4d56e ewr [info] Health check status changed to 'passing'
2021-02-04T03:26:24.640Z 25d4d56e ewr [info] Health check status changed to 'critical'
2021-02-04T03:26:26.599Z f66021b1 vin [info] Health check status changed to 'warning'
2021-02-04T03:26:39.801Z f66021b1 vin [info] Health check status changed to 'critical'
2021-02-04T03:26:44.549Z f66021b1 vin [info] Shutting down virtual machine
2021-02-04T03:26:44.576Z f66021b1 vin [info] Sending signal SIGTERM to main child process w/ PID 504
2021-02-04T03:26:44.850Z 25d4d56e ewr [info] Shutting down virtual machine
2021-02-04T03:26:44.869Z 25d4d56e ewr [info] Sending signal SIGTERM to main child process w/ PID 504
2021-02-04T03:26:45.580Z f66021b1 vin [info] Reaped child process with pid: 504 and signal: SIGTERM
2021-02-04T03:26:45.871Z 25d4d56e ewr [info] Reaped child process with pid: 504 and signal: SIGTERM