Can celery and Django sit on the same machine?

[Radek]

Hi
I have a working web application on Django. It is using

• app for DB
• app that runs 3 machines
  ** Django
  ** celery
  ** beat for celery

the application was created / coded by somone else for me. And I always want to understand how IT works behind…

My question is if instead of 3 machines I can be using only 2 or even 1. And why.

Thank you
Radek

[Radek]

someone would know … ?

[roadmr]

Hi!

You can definitely run all services on a single machine. You’d have to “hack” things a bit, by creating a small shell script that starts your services like so, in daemon mode or in the background

#!/bin/bash
gunicorn the_django_app -D  [other parameters]
celery worker -D [other parameters]
celery beat --detach [other parameters]
sleep infinity

And then use this script in your Dockerfile’s CMD instead of calling e.g. gunicorn directly.

That said, it’s definitely recommended and best practice to run them in separate machines. There are several reasons:

1. Dedicated resources. If your workers are overworked (heh) they will not interfere with your web service, and viceversa.
2. Assymetrical vertical scaling (wow that sounded fancy but it’s really not). If the web server needs a lot of memory but the workers do not, you can scale independently; with the single-machine approach you have less flexibility in this respect. The Celery Beat machine typically can get by with little resources since it’s just queueing up jobs for the workers at intervals.
3. Horizontal scaling per workload. If your service mainly processes background tasks but doesn’t see a lot of web requests, you can scale the worker process group independently as much as you’d like while keeping only a few web servers. This works the other way too: if you’re mostly serving web requests and have the occasional long-running process, you can scale to many web servers but keep only a few worker nodes.
4. Single celery beat instance. This is actually important because if you have several beat instances you might end up with duplicated tasks from them. A single beat instance schedules the jobs, and multiple workers take them from the queue for processing.

I hope this rant is useful

• Daniel

[Radek]

hi Daniel

great, great. Thank you so much for your explanation. I know now how to do it and why not to do it too.

Any chance you can help me with these two questions of mine? Especilly the first one is quite important to me.

Is there an official way to move data between machines, apps, organization?

Hi I have a web application that does not need much of local space. Time to time it needs additonal space usually it would be like 2-10 GB usually just for a minute or few I would love to be able to mount temporatily few GB of space to my Django application or connect to Backblaze But it is not possible so I wonder if I am the only one having this need of temp data space. Now my question is how to get additional disk space. For that I created two questions Can RAM and Volumes be manage…

questoins on free allowances explanation + ssh to machines

Hi could someone please explain to me the Free allowances? Up to 3 shared-cpu-1x 256mb VMs**†** 3GB persistent volume storage (total) Currently I have an organization with 2 apps Django app - shared-cpu-1x@256MB DB appo - shared-cpu-1x@256MB Where Django is running 3 started machines (Django, celery and beat) and 2 stopped (celery and beat) I did not set it up but I want to understand. Could someone please explain to me what are the stopped machines about? Are they going to be billed? h…

Thank you in advance
Radek