Presentation given in class for Cloud Computing at Universitat Politècnica de Catalunya

1. STORM
DISTRIBUTED AND FAULT-TOLERANT
REALTIME COMPUTATION
Jimmy Zöger
CLC < FIB < UPC
2013-06-03

2. INTRODUCTION
• Like Hadoop for realtime processing instead of batch
• Open Source
• Developed by BackType which was later acquired byTwitter
• Developed for analyzingTwitter data
• Similar to S4

3. STORMTOPOLOGY

4. SPOUTS

5. SPOUTS
• The component responsible for feeding messages into the
topology
• Emits tuples
• Can be reliable or unreliable (ack() and fail())

6. INTEGRATION
• Kestrel
• RabbitMQ
• Kafka
• JMS
• Integration is easy with the simple Spout abstraction

7. BOLTS

8. BOLTS
• A component that takes tuples as input and produces tuples
as output
• Can do filtering, joining, functions, aggregations etc.
• Does not have to process a tuple immediately and may hold
onto tuples to process later
• Comparison with Hadoop:
A bolt can be a mapper or a reducer (or anything)

9. STORMTOPOLOGY

10. STORMTOPOLOGY
• Spouts, bolts and streams
• Distributed
• Runs indefinitely until it is stopped
• Arbitrary complexity
• Streams requiring multiple steps also requires multiple bolts
• No intermediate queues for streams

11. FAULT-TOLERANCE
• Nimbus daemon and Supervisor
daemons are fail-fast and stateless
• Each worker sends heartbeats to Nimbus
• Transactional topologies → Guaranteed processing
Nimbus
Zookeeper
Supervisor
Supervisor
Supervisor
Supervisor
Zookeeper

12. USE CASES
• Counting words!
• Realtime analytics - trending topics onTwitter
• Online machine learning
• Continuous computation
• Distributed RPC
• Extract,Transform and Load (ETL)

13. FAST
One benchmark clocked it over
a million tuples processed
per second per node
{x,y,z} ↠ {x,y,z} ↠ {x,y,z} ↠ {x,y,z} ↠ {x,y,z} ↠

14. STORM
DISTRIBUTED AND FAULT-TOLERANT
REALTIME COMPUTATION
Jimmy Zöger
CLC < FIB < UPC
2013-06-03