Celery for background task processing and deferred execution

1. Celery for background
task processing and
deferred execution
Piyush Kumar & Konark Modi

2. Show of Hands : How many of us
• Have used or know about celery / djcelery .
• Have worked with queues .

3. Setup details :
System config. we'd be using:
Centos 6.2 : Hosted on Virtual box
Python version : Python 2.6.6
MySQL
Django-1.4
Packages installed :
1. Virutal env : pip install virtualenv
2. RabbitMQ
3. Celery : pip install -U celery
4. Django-Celery : pip install django-celery
5. MySQLdb module for python : easy_install MySQL , yum
install mysql-devel python-devel
6. pytz : pip install pytz

4. Tutorial sections:
• Understanding the basics – Level 0
• Getting started with celery – Level 1
• Next Steps with Celery – Level 2
• Recap of what we’ve discussed till now
• Q&A

5. Understanding the basics – Level 0
• Need of job queues
• What is celery ?
• Understanding AMQP (Advanced Message Queueing
Protocol )
• Choosing a broker : Why RabbitMQ ?
• Moving to console : Understanding RabbitMQ better.

6. Getting started with Celery – Level 1
• Celery architecture
• Setting up celery : Installation of all the components
• Hands-On : A simple application
• How are we using Celery
• Integrating Celery with a Web-Framework : Django using
djcelery
• Django-celery architecture
• Hands-On : Web application using celery

7. Next steps with Celery – Level 2
• Adding multiple worker
• Retrying tasks
• Routing to multiple queues
• Monitoring
• Security

8. Lets get started now J
Understanding the basics – Level 0

9. Need of Job Queues:
• For executing tasks asynchronusly.
• Distributed tasks.
• Scheduling.
• Retries.
• Handling errors.
• Dynamic work load.
• Enhancing user experience.
• Some of the use cases include:
• Clickstream ( No. of hotels being watched, likes etc.)
• Generate graphs based on a large data set for display on a site’s home page / Dashboard
every 15 minute
• Admin applications like
• Thumbnails generation
• Re-generate static CSS files by examining when certain items in the admin have been modified
• Connecting to third-party APIs
• Mass e-mailers

10. What is Celery ?
• Asynschronus task queue/job queue.
• Uses distributed message passing.
• Supports both real-time processing + schedule jobs.
• Tasks can be run concurrently on a single or more worker
servers taking advantage of multiprocessing.

11. AMQP (Advanced Message Queueing Protocol )
• Open standard application layer protocol for message-
oriented middleware
• Features of AMQP
• Message orientation,
• Queuing,
• Routing (including point-to-point and publish-and-subscribe),
• Reliability
• Security
• Interoperable: AMQP mandates the behaviour of the messaging
provider and client to the extent that implementations from
different vendors are truly interoperable, in the same way as
SMTP, HTTP, FTP, etc. have created interoperable systems.

12. Choosing a broker : Why RabbitMQ ?

13. Moving to console : Understanding
RabbitMQ better

14. Getting started with celery – Level 1

15. Installing Celery :
• RabbitMQ (http://www.rabbitmq.com/download.html )
• pip install -U celery

16. A simple application
• Create tasks.py
• Run celeryd and understand its various options
• Configure the result backend
• Fetch the result
• Call the task using file, python interpreter
• Delay()
• Apply_async() : Expiration, countdown
• Periodic tasks

17. Our use cases:
• Poll approx 300+ DB queries every 15 minutes.
• Custom metric collection
• Inventory management system :
• Asynchronus web-interface
• Periodic tasks

18. Integrating celery with a web-
framework : Django with djcelery
• Installation
• Understanding the DB schema djcelery uses to store
tasks/result sets
• Writing a simple web-app

19. Next steps
• Adding multiple workers
• Retrying tasks
• Monitoring various components of celery:
• Exploring various components of our celery setup.
• Setting up celery-cam Monitoring RabbitMQ
• Running celeryd as a daemon in production
• Security :
• Signals , Tracking of tasks as they transition through different
states, and inspecting return values.
• Message signing.

20. Recap of what we’ve discussed till now

21. Thanks : Q & A