Hi there! I have a project where i need to convert some video files via ffmpeg, but those tasks are sporadic, at most once a day, I was thinking of using something like huey or celery to manage those tasks, but I’m not sure if they could allow my worker to scale to zero
I’m assuming they will need to run all the time to fetch the tasks, so is there any alternative you’d suggest?
My naive idea was to spawn a fly machine to run the task, but that might be overkill?
Disappointed that no one has replied, this is something I’ll have to tackle sooner or later too!
My naive approach would be to run a cron task at an acceptable interval. But that depends on how frequently you’d want it to run…
Would the fly autoscaler work? Expose a “items on queue” metric from your main app or whatever you’re using as broker?
It depends on if it’s a hobby or production app. If it’s a hobby, you can get by with a single machine juiced enough to handle the CPU workload. Just set autostart to true and autostop to false, once the work is done, manually exit with status code 0 once the queue is empty.
If it needs to be on a schedule, something like Github Actions cron can wake up your machine (not it’s not guaranteed to be on time.)
For production grade, you’ll need to use a proper FIFO queue service and set up the autoscaler that @Ollytheninja mentioned. But instead of items on a queue, the metrics would look at the CPU/GPU usage, eg if 1 tasks consumes 75% of the machine resource, running a 2nd would likely OOM the process.
I would have configured the machine to only do as many tasks in parallel as it could handle but this would work too I guess!
Definitely a good point re: making sure machines are sized correctly for tasks!
We do something similar to this just by using the Machines API. Our front end machines monitor the queues and start up workers as needed, and the workers stop themselves when they get bored of waiting for work.