How to Set Up Celery With Django to Manage Asynchronous Tasks

In modern web applications, handling time-consuming tasks synchronously can lead to performance bottlenecks and a poor user experience. It’s ideal to handle tasks like sending emails, generating reports, and processing large data sets asynchronously. This is where Celery, a powerful asynchronous task queue, comes into play. Combined with Django, Celery can significantly enhance the responsiveness and scalability of your application.

In this blog post, we will explore how to set up and use Celery with Django to manage asynchronous tasks efficiently.

Why Use Celery?

Celery is an open-source system for managing distributed task queues, enabling you to execute tasks asynchronously. Some key benefits include:

• Concurrency: Celery can run multiple tasks concurrently, utilizing multiple worker processes or threads.
• Scheduling: It supports scheduling tasks to run at specific times or intervals.
• Scalability: It can handle a large number of tasks and can be scaled by adding more worker nodes.
• Reliability: Celery guarantees reliable task execution and can retry tasks that fail.

Setting Up Django and Celery

Step 1: Installation of Celery and a Message Broker

Celery requires a message broker to handle task messaging. RabbitMQ and Redis are popular choices. For this example, we’ll use Redis.

First, install Celery and Redis:

bash

pip install celery[redis]
pip install redis

Ensure you have Redis running on your local machine or server. You can start Redis with:

bash

redis-server

Step 2: Configure Celery in Django

Create a celery.py file in your Django project’s main directory (where settings.py is located):

Python

# myproject/celery.py

from __future__ import absolute_import, unicode_literals
import os
from celery import Celery

# Setting default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'myproject.settings')

app = Celery('myproject')

# Here using string which means the worker will not have to
# pickle the object when using Windows.
app.config_from_object('django.conf:settings', namespace='CELERY')

# From all registered Django app configs loading task modules.
app.autodiscover_tasks()

@app.task(bind=True)
def debug_task(self):
    print(f'Request: {self.request!r}')

In your settings.py, add the Celery configuration:

Python

# myproject/settings.py

CELERY_BROKER_URL = 'redis://localhost:6379/0'
CELERY_RESULT_BACKEND = 'redis://localhost:6379/0'
CELERY_ACCEPT_CONTENT = ['json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
CELERY_TIMEZONE = 'UTC'

Update the __init__.py file in your project directory to ensure that Celery is loaded when Django starts:

Python

# myproject/__init__.py

from __future__ import absolute_import, unicode_literals

# This ensures the app is always imported when Django starts
# allowing shared_task to utilize this app.
from .celery import app as celery_app

__all__ = ('celery_app',)

Step 3: Create Tasks

Tasks in Celery are just Python functions decorated with @shared_task. Let’s create a simple task to send an email.

Python

# myapp/tasks.py

from celery import shared_task
from django.core.mail import send_mail

@shared_task
def send_welcome_email(user_email):
    send_mail(
        'Welcome!',
        'Thank you for signing up.',
        'from@example.com',
        [user_email],
        fail_silently=False,
    )

Step 4: Running Celery Workers

To execute tasks, you need to run a Celery worker. In the terminal, navigate to your Django project directory and start the Celery worker with:

bash

celery -A myproject worker --loglevel=info

Step 5: Using Tasks in Your Django Application

You can call Celery tasks from anywhere in your Django project. For example, call the send_welcome_email task after a user registers:

Python

# myapp/views.py

from django.shortcuts import render, redirect
from django.contrib.auth.models import User
from django.contrib.auth.forms import UserCreationForm
from .tasks import send_welcome_email

def register(request):
    if request.method == 'POST':
        form = UserCreationForm(request.POST)
        if form.is_valid():
            user = form.save()
            send_welcome_email.delay(user.email)  # Call the task asynchronously
            return redirect('login')
    else:
        form = UserCreationForm()
    return render(request, 'registration/register.html', {'form': form})

Advanced Celery Features

Periodic Tasks

Celery supports periodic tasks, which can be scheduled using the celery-beat scheduler. To use this feature, install the django-celery-beat package:

bash

pip install django-celery-beat

Add it to your INSTALLED_APPS in settings.py:

Python

# settings.py

INSTALLED_APPS = [
    ...
    'django_celery_beat',
]

Run the necessary migrations:

bash

python manage.py migrate django_celery_beat

Configure periodic tasks in your Django admin or directly in code. For example, to schedule a task which needs to run every minute:

Python

# myproject/celery.py

from celery import Celery
from celery.schedules import crontab

app = Celery('myproject')

app.conf.beat_schedule = {
    'send-email-every-minute': {
        'task': 'myapp.tasks.send_welcome_email',
        'schedule': crontab(minute='*/1'),
        'args': ('user@example.com',)
    },
}

Start the Celery beat scheduler:

bash

celery -A myproject beat --loglevel=info

Task Retries

Celery tasks can be configured to retry automatically on failure. For example, to retry the email task if it fails:

Python

# myapp/tasks.py

@shared_task(bind=True, max_retries=3)
def send_welcome_email(self, user_email):
    try:
        send_mail(
            'Welcome!',
            'Thank you for signing up.',
            'from@example.com',
            [user_email],
            fail_silently=False,
        )
    except Exception as exc:
        self.retry(exc=exc, countdown=60)  # Retry after 60 seconds

Monitoring and Administering Celery

Monitoring Celery can be done using tools like Flower, a real-time web-based monitoring tool for Celery. Install Flower with:

bash

pip install flower

Start Flower to monitor your Celery tasks:

bash

celery -A myproject flower

Flower provides a web interface (usually at http://localhost:5555) where you can monitor task progress, view task details, and manage workers.

Conclusion

Integrating Celery with Django enables you to delegate time-consuming tasks away from your main application thread, enhancing the overall responsiveness and scalability of your application. By setting up Celery with a reliable message broker like Redis, you can efficiently manage asynchronous tasks, schedule periodic tasks, and ensure that your application can handle increased load without compromising performance. With advanced features like task retries and monitoring, Celery provides a robust solution for asynchronous task management in Django applications.