How to Use Celery with Django for Background Tasks

To use Celery with Django, install Celery and a message broker like Redis, then configure Celery in your Django project by creating a celery.py file and defining tasks. Run the Celery worker alongside your Django app to execute tasks asynchronously.

Syntax

Using Celery with Django involves these parts:

• celery.py: Defines the Celery app and configures it with Django settings.
• tasks.py: Contains functions decorated with @shared_task to run asynchronously.
• Broker URL: Specifies the message broker (like Redis) Celery uses to queue tasks.
• Worker: A process that runs and executes the queued tasks.

python

from celery import Celery

app = Celery('myproject')
app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks()

# Example task in tasks.py
from celery import shared_task

@shared_task
def add(x, y):
    return x + y

Example

This example shows how to set up Celery in a Django project named myproject with Redis as the broker, define a simple task, and run the worker.

python

# myproject/celery.py
from __future__ import absolute_import, unicode_literals
import os
from celery import Celery

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'myproject.settings')

app = Celery('myproject')
app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks()

# myproject/__init__.py
from .celery import app as celery_app

__all__ = ('celery_app',)

# myapp/tasks.py
from celery import shared_task

@shared_task
def multiply(x, y):
    return x * y

# settings.py additions
CELERY_BROKER_URL = 'redis://localhost:6379/0'

# Run worker command in terminal:
# celery -A myproject worker --loglevel=info

Output

 -------------- celery@hostname v5.x.x (singularity)
--- ***** -----
--- * ***  * -- Linux-... 
--- * - *  * -- Python 3.x.x
--- *** -----
 --------------

[config]
broker_url: redis://localhost:6379/0

[tasks]
  . myapp.tasks.multiply

[INFO/MainProcess] Connected to redis://localhost:6379/0
[INFO/MainProcess] mingle: searching for neighbors
[INFO/MainProcess] mingle: all alone
[INFO/MainProcess] celery@hostname ready.

Common Pitfalls

Common mistakes when using Celery with Django include:

• Not running a message broker like Redis or RabbitMQ before starting Celery workers.
• Forgetting to import the Celery app in __init__.py, causing tasks not to register.
• Not configuring CELERY_BROKER_URL correctly in Django settings.
• Running tasks synchronously by calling them directly instead of using .delay() or .apply_async().

python

## Wrong way: calling task directly (runs synchronously)
# result = multiply(4, 5)

## Right way: call task asynchronously
result = multiply.delay(4, 5)  # returns AsyncResult object

Quick Reference

Summary tips for using Celery with Django:

• Install Celery and Redis: pip install celery redis
• Set CELERY_BROKER_URL in settings.py to your broker URL.
• Create celery.py in your project root to configure Celery.
• Define tasks with @shared_task decorator.
• Start Redis server before running Celery workers.
• Run workers with celery -A yourproject worker --loglevel=info.

Key Takeaways

Always configure Celery with a message broker like Redis in Django settings.

Define tasks using the @shared_task decorator to run them asynchronously.

Run the Celery worker process separately to execute background tasks.

Call tasks with .delay() or .apply_async() to avoid blocking your Django app.

Ensure the broker service is running before starting Celery workers.