New feature: Scheduled machines

We now support scheduling machines!

You can schedule a machine with flyctl or via curl like so:

fly machine run —schedule=hourly

curl -i -X POST \ -H "Authorization: Bearer ${FLY_API_TOKEN}" -H "Content-Type: application/json" \ "http://_api.internal:4280/v1/apps/${APPNAME}/machines" \ -d '{ "name": "schedule-my-machine", "config": { "image": "flyio/fastify-functions", "schedule": "hourly", "services": [ ... ] } }'

We currently only support the following for schedule, "monthly", "weekly", "daily", "hourly". This is calculated from the time of machine creation, so the machine will not start at a specific time i.e. on the hour, every wednesday etc

A scheduled machine can be monitored via the UI and the by running fly machine status. Do note, that a schedule machine when created will start in a stopped state and move to a started state at every scheduled interval.

This feature is available on all hosts, and we’d love to hear your feedback and see what you build!

19 Likes

I’d like to try this out! I’ve got a deployed machine (currently stopped, I can start it manually) whose status I can interrogate using fly machine status [machine id]. However when I run fly machine run —schedule=daily [image from status command] I get an auth error:
Error Authentication required to access image "docker.io/library/[name:deployment...]". What am I doing wrong? I should add that I’m using an image built using a local Dockerfile.

If I understand the problem right: You’d need to push that img to Fly registry because Fly can’t deploy from private docker repositories.

Multiple ways to do so: Fly deploy permission denied connecting to Docker daemon - #2 by ignoramous

# (typing these from memory...)
# for machines, I use:
flyctl deploy --local-only --dockerfile <path/to/file> --config <path/to/machines.toml>

# or, build the img and push it to Fly without deploying
flyctl deploy --local-only --build-only --push --dockerfile <path/to/file> -a <app-name>

# create a machine with the img
flyctl m run <registry.fly.io/app-name:img-tag> --schedule=daily --region <r> --name <uniq-name>

2 Likes

Ahh, yep the second option worked – looks like I’ve got a new machine using the pushed image, and the --schedule switch was accepted, so I’ll find out if it runs sometime tomorrow.

So what’s the best approach to deploying updated images? It seems like you can’t / aren’t supposed to update machines with new images – just stopping it if necessary and removing it and creating a new one is preferred? How does that integrate with e.g. Github actions? If I understand correctly, I’d want to:

  1. push the new image
  2. deploy a new machine using the new ID
  3. destroy the existing machine if the deploy is successful.

However for step 2 you’d need the image ID you just created, and for step 3 you’d need the current machine ID.

(Sorry, just thinking out loud here)

The best practice is to use the GraphQL APIs which are more mature (even if as inconsistent) over flyctl m commands.

I prefer flyctl and I have requested that Fly engs pls:

  1. Fix the existing flyctl m update <existing-machine-id> --dockerfile </path/to/file> --config </path/to/machines.toml> command.
  2. Support the --image switch like so, flyctl m update <existing-machine-id> --image <img> --config </path/to/machines.toml>.

It’d also be neat if flyctl m run <img> --name <existing-machine-name> --config </path/to/machines.toml> could deploy to an existing machine. But it doesn’t.


You could destroy machines and create new ones, if that’s what you prefer, or flyctl deploy --image <img> --now --strategy immediate --config </path/to/machines.toml> to deploy the img to all machine VMs of the app (as defined in machines.toml).

2 Likes

That all seems very sensible. I haven’t seen any reference to machines.toml in the docs – is there anything I can have a look at?

It is just your regular fly.toml but within limits (as Machines don’t yet support all configurations options that a regular App does). While flyctl m commands have limitations of their own, as it’s pretty raw and new.

I may have confused matters yesterday (or maybe some changes happened at the backend)? In any case: when a new app image is deployed using flyctl deploy --remote-only, that new image is assigned to any running machines, but their env and schedule entries are wiped.

You should be able to repro this by running fly m status [id] -d before and after a deploy. The image assigned to the machines is the same as you see if you run fly image show. Creating a new machine with fly machine run --env=… --schedule=… registry.fly.io/[image_id] --region=… shows the expected env and schedule entries when running fly m status [id] -d, and I can confirm that the scheduler is working.

1 Like

It is totally possible that flyctl deploy wipes off schedule; I haven’t see if wipe off envs though (I set them in the dockerfile with the ENV directive and in the [env] section of the .toml file and these have been re-applied across deploys and runs, just fine): Bug : During Development of Machine Manager - #9 by tvdfly

For scheduled machines, it could very well be that only flyctl m run is supported. And so you were right: One would have to remove the existing machine and run a new one just to update the docker-image (iff using flyctl), since flyctl m update is broken.

I hadn’t set them in machines.toml or Dockerfile – I’ll try that.

Gotcha. In practice this isn’t a big deal for my use case atm as everything is working now, so updates to the image will be hopefully rare, and I assume flyctl m will catch up at some point.

1 Like

I wouldn’t hold my breath… :wink: If I was doing it, I’d schedule Machines from an external source like Scheduled Durable Objects or GitHub Actions.

1 Like

If this PR is merged flyctl m update <machine-id> --image <new-image> -a <machine-app> ... should carry forward the existing schedule unless overriden with the --schedule switch (which, per my reading of the code, flyctl m update already supports).

So, the PR (linked above) was merged (and hopefully, isn’t reverted before the next release).

This command should build and deploy a new image with schedule intact:

# alternatively, specify the --image switch if using a prebuilt
fly m update <machine-id> --dockerfile </path/to/file> -a <app-name>

I’ll send another PR to fix fly deploy wiping out schedule, in a bit.

1 Like

Thank you for this feature. I’m currently migrating a few Heroku apps to fly that require this. Can you provide an example on how to run a single Rake task with a schedule machine? I’m not sure I totally understand how to set it up.

The documentation for machines in general is VERY confusing. I managed to get a simple nodeJS machine setup by doing the following:

  • Create your nodejs app (just have console.log(“My test”); in index.js
  • Deploy the app: fly deploy --build-only --push
  • Copy the image URL from the response above
  • Create the initial machines app: fly apps create --machines
  • Create the machine to run YOUR app: flyctl m run <imageurl> --name your-app-name-machine -a <name-of-initial-machines-app>
  • List your machines to get its ID fly machines list -a name-of-initial-machines-app
  • Open the fly.dev dashboard and go to the machine page → “machines” in the sidebar → your machine name → monitoring
  • Start the machine fly machines start <id-of-your-machines-app> -a <name-of-initial-machines-app>
  • You should now see the log of the machine run.

I haven’t tested this yet, but I assume you can delete the machines app and create a new one, this time with the --schedule=hourly argument EDIT: or use the fly machines update command

You can condense these into one step:

fly m \
   run \
   . \
   --name <some-user-friendly-name> \
   --dockerfile </path/to/dockerfile> \
   --region <lhr> \
   -a <app-name>

Then clone these machines if we want more of them:

# fetch machine ids
fly m list -a <app-name>

# clone an active machine onto a new machine instance
fly m \
    clone \
    <from-machine-id> \
    --name <some-other-name> \
    --region <ams> \
    -a <app-name>
2 Likes

Would it be possible to update the documentation at Machines · Fly Docs or add a new document for this kind of stuff, especially with some basic examples (both docker & non-docker)? I had to look at multiple forum posts and the above documentation to get even a basic machine working.

1 Like

can relate :wink:

One can sure send a pull request to improve the docs: https://github.dev/superfly/docs/blob/main/reference/machines.html.md.erb

1 Like

Is there a way I can view my machine’s schedule status? I updated an existing machine to be scheduled hourly, but I can’t see anything in the dashboard to suggest that it will actually execute hourly, and is in state “stopped”. Logs are cleared each time I re-visit the monitor page, so I don’t know how to view it except keeping the monitor page open for an hour. I am supposed to start the machine again after setting a schedule?

$ dotenv -- bash -c 'fly machines update $MACHINE_ID --schedule=hourly -a $MACHINES_APP_ID'
Update available 0.0.424 -> v0.0.431.
Run "flyctl version update" to upgrade.
Machine 73d8d347b13789 was found and is currently in a stopped state, attempting to update...
Searching for image 'registry.fly.io/lightsats-scheduler:deployment-01GHZVBM4562H5W7D5BXA373BC' remotely...
image found: img_19gm46eq92kvx0jk
Image: registry.fly.io/lightsats-scheduler:deployment-01GHZVBM4562H5W7D5BXA373BC
Image size: 150 MB

Machine 73d8d347b13789 has been updated

Instance ID has been updated:
01GHZVEDNBTKTKXGPNE5J0DV1R -> 01GHZVFBKKKQ5ETFXZ3Q3ZAJYT

Image: registry.fly.io/lightsats-scheduler:deployment-01GHZVBM4562H5W7D5BXA373BC
State: Stopped

Monitor machine status here:
https://fly.io/apps/lightsats-machines-2/machines/73d8d347b13789
Done in 4.30s.

If you run fly m status [machine id] -d you should see JSON output under the Config: heading. That output should include a schedule key showing the currently configured schedule, e.g. daily.

2 Likes