Scheduled Machine not starting -- insufficient memory

Questions / Help
1

We have a scheduled machine which has hasn’t been getting run with the following error:

could not reserve resource for machine: insufficient memory available to fulfill request

This scheduled machine runs some important cron-like tasks for our production application so it very much needs to be run on the specified schedule. How do I fix this as we have tasks queuing up from this machine not running?

2

Hi!

This means the machine was on a host that didn’t have enough memory available when the scheduled task was scheduled to start.

Generally the way around this is to have multiple machines (they get spread around different hosts). In your case, since it’s a CRON-like behavior, you’ll likely want to do 2 things:

  1. Spin up at least 2 machines
  2. Add a “locking” mechanism is only one machine runs the actual work (assuming that’s what is appropriate for your situation).

This is often done with Redis. At one point we provided Consul for locking like this, I have to check to see if that’s available on our latest apps platform.

3

Here we go, you can enable consul like so: New commands: `fly consul attach` and `fly consul detach`

And then use this older announcement to see how to reach it: Sneak peak: global lock service

You can use it to do some locking: Commands: Lock | Consul | HashiCorp Developer

4

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.