Hi. After plenty of struggles, I can finally report success on using machines scheduled hourly to run batch jobs! Here’s what works, as of flyctl 0.0.487 and whatever is running serverside today 2023-03-15:
flyctl apps create --machines exampleapp # Note, you cannot set secrets per-machine, only per-app. # I use multiple apps, to keep my batch jobs isolated. TODO whatever you need to output secrets | flyctl secrets import -a exampleapp # You can use a `fly.toml` file, or just pass everything needed as flags. # # Unfortunately, `flyctl machine run` does not read `[env]` from the config file, and nothing run here "stores env in the app" in any way. # You **must** pass environment variables here, per machine instance. # # This will build your app using Dockerfile etc in current dir: flyctl machine run . -a exampleapp \ --restart no \ --schedule hourly \ --region sjc \ --memory 2048 \ --env FOO=foo \ --env BAR=bar
Now you have a batch job that runs roughly hourly.
Adjust region and memory to fit your use, I’m including them just as examples.
Overriding the default
--restart always is crucial, or your batch job will busy loop instead of running roughly hourly.
If your batch job is normally reliable, you may want
--restart on-fail. Mine is dependent on third party APIs that are flaky, and I don’t want to start hitting them harder after they’ve already failed, so I use
--restart no. Errors cause a retry after an hour’s rest.
: Machine environment errors, machine creation takes 1 hour, flyctl machine run does not respect [build]dockerfile in config, Internal error setting secret for machines apps, Logs displayed by `flyctl deploy` are missing lines and so on