2. Speakers' background
University of Jyväskylä
the largest Plone user in Finland
● 83 Plone sites
● 2.5 TB data served through Plone
● 700 content editors
5. Why message queues?
We needed a reliable, effective and
scalable solution for:
● running asynchronous tasks
● being a common integration point
between services (decoupling).
6. Why message queues?
”None of the existing ZODB-based mechanism
really scales. When you are really in need for a
real and working queuing mechanism one
would use some external queuing solution (e.g.
something AMQP compatible like RabbitMQ).”
– Andreas Jung
7. Advanced Message Queue
Protocol (AMQP)
Think of it as an highly advanced
post office which can deliver
message to multiple recipients
based on address.
8. AMQP jargon 1/2
● Message consists of label and payload.
● Producer is a program which sends
messages to exchange.
● Consumer is a program which mostly waits
to receive messages.
9. AMQP jargon 2/2
● Exchange receives the messages from
producers and pushes them to queues.
● Bindings are how the messages get routed
from the exchange to a queue.
● Queue is a buffer that stores messages.
13. RabbitMQ
● Implements AMQP (Advanced Message
Queue Protocol) open standard
● Written in Erlang
○ made for messaging
○ easy to cluster
○ fast (enough)
● Industry tested and reliable
16. Produce messages with
collective.zamqp
from zope.component import getUtility
from collective.zamqp.interfaces import IProducer
producer = getUtility(
IProducer, name="announcer")
producer.register() # for transaction
producer.publish("Hello World!")
17. Handle messages with
collective.zamqp
from five import grok
from collective.zamqp.interfaces import IMessageArrivedEvent
@grok.subscribe(IAnnouncement,
IMessageArrivedEvent)
def handleMessage(message, event):
logger.info(message.body)
message.ack()
26. Real world example
Moniviestin video publishing service
● Plone 4.1 site
● Distributed encoder servers (48 CPUs
combined)
● First version used XML-RPC based
communication - lots of error handling code.
● New version uses AMQP-messaging.
● Very fast and scalable. Recovers from
network outages.
27. Encoder
Mediainfo
memcached
FS
RabbitMQ
CherryPy
upload
service
Plone
28. Thank you
c.zamqp and examples are available at:
● https://github.com/datakurre/collective.zamqp
● https://github.com/datakurre/collective.zamqpdemos
● https://github.com/datakurre/chatbehavior
● https://github.com/datakurre/pubsubannouncements
other relevant links:
● https://www.rabbitmq.com/getstarted.html
● https://www.amqp.org/about/examples
● http://www.manning.com/videla/ (”RabbitMQ in Action”)