Is Fly.io suitable for long running jobs?

1

Hi fly community!

I am working on a project that runs background jobs that can take several minutes (+5 minutes), my implementation is more or less something like this:

  1. VM running Hono.dev that has an endpoint for triggering a background job when called
  2. When the job is running it spins up chromium with puppeteer and it starts crawling the content of hundreds/thousands of web pages (sometimes it takes 1 minute, sometimes it can take 10).

What is the problem?

After some time of no HTTP activity the hono server goes to sleep/scale down, (this is Fly’s natural behaviour), even though the background job it’s still running and doing some operations, this kills the chromium instance and everything starts to fail.

Is there a correct way to do configure a Fly VM to not go to sleep if there is a job running?

2

Take a look at GitHub - fly-apps/pdf-appliance: Auto start machines that will generate PDFs for your application - it it somewhat similar to your app in that it runs puppeteer and only goes to sleep after an idle period after the last active request completes. What you need to do is not have us automatically stop the machine, and instead exit successfully (code 0) when you want.

Key things to look at:

3

thank you very much! Will check this solution out

4

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.