Queue/Worker architecture with Autostop/autostart Machines?

You can use metrics-based autoscaling to spin workers up or down based on queue size. Metrics-based Autoscaling

It uses the Machines API under the hood but you don’t have to think too much about that.

On Python land I’d run Celery Flower next to my Celery workers to publish queue size metrics for the auto-scaler to base its decisions on. I’m not sure how to publish queue size metrics with BullMQ, but pretty sure there’s a way to do so :slight_smile:

  • Daniel