Success report: hourly machines for batch jobs, with secrets and environment

Show & Tell
1

Hi. After plenty of struggles[1], I can finally report success on using machines scheduled hourly to run batch jobs! Here’s what works, as of flyctl 0.0.487 and whatever is running serverside today 2023-03-15:

flyctl apps create --machines exampleapp

# Note, you cannot set secrets per-machine, only per-app.
# I use multiple apps, to keep my batch jobs isolated.
TODO whatever you need to output secrets | flyctl secrets import -a exampleapp

# You can use a `fly.toml` file, or just pass everything needed as flags.
#
# Unfortunately, `flyctl machine run` does not read `[env]` from the config file, and nothing run here "stores env in the app" in any way.
# You **must** pass environment variables here, per machine instance.
#
# This will build your app using Dockerfile etc in current dir:
flyctl machine run . -a exampleapp \
  --restart no \
  --schedule hourly \
  --region sjc \
  --memory 2048 \
  --env FOO=foo \
  --env BAR=bar

Now you have a batch job that runs roughly hourly.
Adjust region and memory to fit your use, I’m including them just as examples.

Overriding the default --restart always is crucial, or your batch job will busy loop instead of running roughly hourly.

If your batch job is normally reliable, you may want --restart on-fail. Mine is dependent on third party APIs that are flaky, and I don’t want to start hitting them harder after they’ve already failed, so I use --restart no. Errors cause a retry after an hour’s rest.

[1]: Machine environment errors, machine creation takes 1 hour, flyctl machine run does not respect [build]dockerfile in config, Internal error setting secret for machines apps, Logs displayed by `flyctl deploy` are missing lines and so on

2 Likes