Also, machines in state created cannot be destroyed, meaning one can’t give up and tear down the experiment to move on to something else, you need to monitor it until completion just to tear it down, to avoid leaving idle machines around.
Also really worrying to me is how your logging mechanism (NATS?) still loses log entries. See how only one of those two snippets says “Configuring firecracker”. This makes debugging incredibly frustrating, as you simply can’t trust what you’re seeing.
It looks like both of these machines are scheduled machines configured to run hourly. What we do for scheduled machines is setup the VM but delay starting it until the next execution based on the schedule which explains the delay.
We’ve not had a lot of use for scheduled machines so would be curious to know what you’d expect to happen.
Ohh so if I add schedule=hourly it’ll delay up to an hour before the first launch! That explains things.
I guess I need to set them up as “normal” machines first, and then change them to hourly, to get the intuitive behavior I want – see 1 round of batch processing complete now, then refresh every now and then.