Introduction to Django-Celery and Supervisor

1. Introducti
on to
Celery

2. What is Celery?
“Celery is an asynchronous task queue
based on distributed message
passing.”
It is a powerful, production-ready
asynchronous job queue, which allows
you to run time-consuming Python
functions in the background.

3. Why should I use
Celery?

4. User perspective
• Minimize request/response cycle
• Smoother user experience
• Difference between pleasant and unpleasant

5. Developer perspective
• Offload time/cpu intensive processes
• Scalability - add workers as needed
• Flexibility - many points of customization
• Actively developed
• Great documentation
• Lots of tutorials

6. OUT OF THE
REQUEST/RESPONSE
CYCLE
• Example: Sending emails asynchronously.

7. TASKS IN THE
BACKGROUND.
• Example: Computational heavy jobs.
• Example: Interacting with external APIs.

8. PERIODIC JOBS

10. Celery Architecture

11. PRODUCER
• Produces a task for the queue.

12. BROKER
• Stores the task backlog
• Answers, what work remains to be done?
• RabbitMQ, Redis, SQLAlchemy, Django's
ORM, MongoDB

13. WORKER
• Execute and consumes tasks.
• Distributed.

14. RESULTS BACKEND
• Stores the results from our tasks.

15. EXAMPLE

16. INTERGRATING WITH
DJANGO.

17. BEWARE OF DJANGO-
CELERY

18. IMPORTANT SETTINGS
• BROKER_URL = 'redis://localhost:6379/0'
• CELERY_RESULT_BACKEND = 'redis://localhost:6379/0'
• CELERY_TASK_RESULT_EXPIRES = 7*86400 # 7 days
• CELERYBEAT_SCHEDULER =
"djcelery.schedulers.DatabaseScheduler"
• CELERY_ACCEPT_CONTENT = ['application/json']
• CELERY_TASK_SERIALIZER = 'json'
• CELERY_RESULT_SERIALIZER = 'json'
• BROKER_BACKEND = "redis"
• BROKER_CONNECTION_TIMEOUT = 5.0
• CELERY_IMPORTS = ('<your-app-name>.tasks',)

19. Write code in celery.py

20. Intialize celery in __init__.py

21. BEST PRACTICES

22. NEVER PASS OBJECTS AS
ARGUMENTS.

23. AVOID LAUNCHING SYNCHRONOUS
SUBTASKS

25. PERIODIC TASKS

26. THINGS GO WRONG ?

27. RE-TRY!

28. Thanks!