Recurring scheduled tasks (like cron)

Is there any fly.io recommended way to run scheduled recurring tasks like you would with cron?

From a bit of googling around it seems like a lot of people are recommending running cron jobs from the host, since running cron inside a container isn’t great. Not sure how true that is, but I’m guessing someone else using fly has already built something like this.

My personal use case is having a task run every 10-30 seconds that checks for new config files for a caddy instance (or maybe for multiple caddy instances), if that helps.

3 Likes

Also interested in this! Knowing that Fly schedules tasks using Hashicorp Nomad, it wouldn’t be far-fetched to imagine exposing its built-in support for periodic jobs.

@carter we’re working on this. Should be available within a few months.

What you’ve been googling about is probably related to containers. We run Docker images as virtual machines (VM). It should be ok (but imperfect) for now to run cron inside your VMs, for your use case.

Indeed! We’ve been thinking about it for a while. It’s mostly a matter of implementing the logic in our API.

1 Like

@carter / @jerome
I’m not sure if this relates, but I just asked a question about the [[services.script_checks]] in a fly.toml, which appears to support intervals

Not a replacement for cron, but maybe enough? I know it could solve a few things for us, but I’m not 100% sure on its behavior.

1 Like

Yes, that would work right now :slight_smile:

I should mention: if you script checks fail, your instance will be “unhealthy” and our scheduler will restart it.

@jerome - so we should trap any errors from the script_check if we don’t want to cause a general restart. That makes sense.

1 Like

Hey guys, looks like it has been a few months, just checking in on this, what is the best way to run cron scripts today?

I have a new thread going here as this was “solved”:

https://community.fly.io/t/fly-cron-jobs/2856

Script checks are going away, so are not a good answer anymore for this. There’s nothing built-in yet, but you could use an external scheduler. Most runtimes have something that can run periodic jobs. Node has Bree and others.

Thanks for that resource, will definitely take a look at this, it looks perfect.

Any plans to have something a bit more native to Fly similar to Heroku?

Let me ask you this, how is Fly.io doing their internal billing with the Stripe API for metered billing?

I assume you are on a monthly basis running the VM usage calculation and then creating a metered subscription invoice, is this automated? If so, are you using billing thresholds internally? Would love to know more on how you guys are using fly internally to handle billing, etc as this pertains to a use case for us as well.

We do all our scheduling in process. Periodically triggering VMs is harder than you might think, because you run into issues where one takes too long, another starts, and things get weird.

The simplest thing to do is just run an app with an interval/timer that fires things off every so often.

Our billing is pushing Stripe way past its limits. Each organization gets a subscription with multiple metered products on it. Every hour, we compute usage for disk, memory, CPU, and bandwidth. We send those to Stripe as usage records and it handles the rest.

Stripe billing is not great for metered products. We’ll likely end up doing our own accounting and just create invoices from our code.

Looking for the best way to pull off scheduled jobs similar to Heroku (Heroku Scheduler | Heroku Dev Center).

We have a node run time, ideally we would like to run a script from package.json every hour for example.

Thanks!

When are script checks going away?

There is a heisenbug with script checks that causes our VM init to hang. They’re not going away any time soon, but we haven’t been able to track that bug down so we’ve stopped using them. If you have an app that relies on them and they haven’t caused issues, it’s safe to keep using them.

Thanks so much for the explanation, this is exactly what we are looking to achieve as well as we have a a subscription for the monthly flat rate, and another subscription that we add for the metered use (transaction fees / revenue shares). We are planning on taking advantage of billing thresholds with this so that AP doesn’t get too high mid month.

To be honest, I honestly would think spinning up a VM on a schedule, running a task, and then shutting down does seem simple from me as a customer haha - This is probably just because in the past in the rails world I would just create different rake tasks, and then use heroku scheduler to run that task on a hourly basis / daily basis / monthly basis, etc.

I would love to learn more on this subject to better understand the complexities of these processes.

Something to be said about using something like Bree is that this requires to have machines running 24/7 possibly doing nothing 99% of the time. From a billing / usage standpoint this is sort of a waste.

1 Like

This is true - but if your tasks don’t require many resources, a 24/7 small VM comes in at a negligible cost.

Another approach would be to run something like Bree in the same VM as your app. Our multiple processes guide gives you a few options to do that. I’m a fan of either bash (simple) or overmind which uses the familiar Procfile syntax.

Gotcha, makes sense. I agree that cost is negligible, just thought I would bring this up as a concern.

We just spun up a new app and run the bree process (on fly this is $2/month :partying_face: )