Brad Anderson from MapR gave a presentation on Hadoop and Storm. He explained that Hadoop is a distributed computing platform that ships functions to where the data is located. Storm is described as "Hadoop for real-time" processing. It provides guarantees for processing data reliably at scale across clusters. Topologies in Storm define the network of spouts that read data from sources and bolts that process the data streams.

1. ©MapR Technologies
Hadoop and Storm
AJUG 5/21/2013

2. whoami
• Brad Anderson
• Solutions Architect at MapR (Atlanta)
• ATLHUG co-chair
• NoSQL East Conference 2009
• “boorad” most places (twitter, github)
• banderson@maprtech.com

3. Hadoop: A Paradigm Shift
 Distributed computing platform
– Large clusters
– Commodity hardware
 Pioneered at Google
– Google File System, MapReduce and BigTable
 Commercially available as Hadoop

4. Ship the Function to the Data
SAN/NAS
data data data
data data data
data data data
data data data
data data data
function
RDBMS
Traditional Architecture
data
function
data
function
data
function
data
function
data
function
data
function
data
function
data
function
data
function
data
function
data
function
data
function
Distributed Computing

5. MapReduce Flow
Input Map Combin
e
Shuffle
and sort
Reduc
e
Output
Reduc
e

6. Variation: No Reduce Necessary
Example: Batch File Transformation
Input Map Output
MPG M4V

7. Variation: Multiple MapReduces
Example: Fraud Detection in User Transactions
LDA training
Transaction
data
LDA scoring
HBase /
MapR M7 Edition
G2 score
Candidate
events for
analyst review
95 %-ile LDA
anomaly
MapReduce
http://en.wikipedia.org/wiki/Latent_Dirichlet_allocation

8. Pig

9. MR Equivalent to Pig Script

10. Hive

11. MapR Distribution for Apache Hadoop
Complete Hadoop
distribution
Comprehensive
management suite
Industry-standard
interfaces
Enterprise-grade
dependability
Enterprise-grade security
(US Intelligence Agency)
Patents - IP
Higher performance

12. Hadoop Use Cases
ETL/EDW Offload
Sensor / Telemetry Data
Recommendation Engine
Search
•ML algorithms
•eDiscovery
Fleet Management
Fraud Detection / Risk Management
Traffic Decongestion

13. One Platform for Big Data
…
99.999%
HA
Data
Protection
Disaster
Recovery
Scalability
&
Performance
Enterprise
Integration
Multi-
tenancy
Map
Reduce
File-Based
Applications
SQL Database Search Stream
Processing
Batc
h
Interactiv
e
Realtime
Batch
Log file Analysis
Data Warehouse Offload
Fraud Detection
Clickstream Analytics
Realtime
Sensor Analysis
“Twitterscraping”
Telematics
Process Optimization
Interactive
Forensic Analysis
Analytic Modeling
BI User Focus

14. ©MapR Technologies
Storm
“Hadoop for Realtime”

15. ©MapR Technologies
Before Storm
Queues Workers

16. ©MapR Technologies
Example
(simplified)

17. ©MapR Technologies
Storm
Guaranteed data processing
Horizontal scalability
Fault-tolerance
No intermediate message brokers!
Higher level abstraction than
message passing
“Just works”

18. ©MapR Technologies
Unbounded sequence of tuples
Tuple Tuple Tuple Tuple Tuple Tuple Tuple
Streams

19. ©MapR Technologies
Source of streams
Spouts

20. ©MapR Technologies
public interface ISpout extends Serializable {
void open(Map conf,
TopologyContext context,
SpoutOutputCollector collector);
void close();
void nextTuple();
void ack(Object msgId);
void fail(Object msgId);
}
Spouts

21. ©MapR Technologies
Processes input streams and produces new streams
Tuple Tuple Tuple Tuple
Bolts

22. ©MapR Technologies
public class DoubleAndTripleBolt extends BaseRichBolt {
private OutputCollectorBase _collector;
public void prepare(Map conf,
TopologyContext context,
OutputCollectorBase collector) {
_collector = collector;
}
public void execute(Tuple input) {
int val = input.getInteger(0);
_collector.emit(input, new Values(val*2, val*3));
_collector.ack(input);
}
public void declareOutputFields(OutputFieldsDeclarer declarer) {
declarer.declare(new Fields("double", "triple"));
}
}
Bolts

23. ©MapR Technologies
Network of spouts and bolts
Topologies

24. ©MapR Technologies
TridentTopology topology = new TridentTopology();
TridentState wordCounts =
topology.newStream("spout1", spout)
.each(new Fields("sentence"),
new Split(),
new Fields("word"))
.groupBy(new Fields("word"))
.persistentAggregate(new MemoryMapState.Factory(),
new Count(),
new Fields("count"))
.parallelismHint(6);
Trident
Cascading for Storm

25. Storm
©MapR Technologies
Hadoop
batch
processes
Apps
Busines
s
Value
Raw
Data
realtime
processesQueue(Kafka)
Parallel Cluster Ingest

26. ©MapR Technologies
Hadoop
batch
processes
Apps
Busines
s
Value
Raw
Data
realtime
processes
Storm
TailSpout
Franz
Queue(Kafka)

27. StormKafka
Twitter
Twitter API
TweetLogger
Kafka
Cluster
Kafka
Cluster
Kafka
Cluster
Kafka
API
Storm
Web Service NAS
Web
Data
Hadoop
Flume
HDFS
Data

28. Twitter
Twitter
API
Catcher Storm
Topic
Queue
Web-server
http
Web
Data
MapR
TweetLogger

29. Scaling Estimates
Twitter Firehose
 Old School – 8+ separate
clusters, 20-25 nodes
• >3 Kafka nodes
• >2 TweetLoggers
• 5-10 Hadoop
• >2 Catcher nodes
• >3 Storm
• 3 zookeepers
• NAS for web storage
• >2 web servers
 MapR – One Platform
• 5-10 nodes total
• Any node does any job
• Full HA included
• Backups included

30. ©MapR Technologies
github
• Watch TailSpout & Franz development
• https://github.com/{tdunning | boorad | pfcurtis}/mapr-spout
• And our example Twitter implementation
• https://github.com/{tdunning | boorad | pfcurtis}/mapr-spout-test

31. Demo