fly-autoscaler: scale specific process groups in an app

[lewoudar]

Hello everyone,

I have a Django application running with different processes:

• web processes
• worker processes for running background jobs using django-dramatiq.
• cron processes for cron jobs

I end up with a classic fly.toml like the following:

[processes]
  cron = 'python manage.py crontab'
  web = 'granian --interface wsgi tme_woudar_bot.wsgi:application --host 0.0.0.0 --port 8000'
  worker = 'python manage.py rundramatiq -p 4 -t 2'

[[mounts]]
  source = 'faili_data'
  destination = '/data'
  processes = ['web', 'worker']

[http_service]
  internal_port = 8000
  force_https = true
  auto_stop_machines = 'stop'
  auto_start_machines = true
  min_machines_running = 0
  processes = ['web']

  [http_service.concurrency]
    type = 'requests'
    hard_limit = 50
    soft_limit = 25

  [[http_service.checks]]
    interval = '30s'
    timeout = '10s'
    grace_period = '10s'
    method = 'GET'
    path = '/ht/?format=json'

    [http_service.checks.headers]
      Host = 'faili-proxy.fly.dev'

[[vm]]
  size = 'shared-cpu-2x'
  memory = '2gb'
  processes = ['web']

[[vm]]
  size = 'shared-cpu-4x'
  memory = '4gb'
  processes = ['worker']

[[vm]]
  size = 'shared-cpu-1x'
  memory = '1gb'
  cpu_kind = 'shared'
  cpus = 1
  processes = ['cron']

The configuration for auto-scaling web processes is straightforward and well documented, but scaling worker processes is more difficult than I expected.
Yes, there is documentation on how to autoscale based on metrics, but it seems there is no way to specify the process groups to which the logic should be applied.

Does that mean, we should always separate worker processes from web processes into different apps to be able to scale efficiently based on metrics? If it is the case, it should be clearly mentionned in the official documentation.

Thanks in advance for your replies