Paper presentation for IEEE BigData 2014 Paper 6745 Building A Massive Stream Computing Platform ForFlexible Applications

1. Building A Massive Stream
Computing Platform For
Flexible Applications
Tianjian Chen
Zhengrui Man
Hao Li
Xin Sun
Raymond K. Wong
Zhiwei Yu
Jun, 2014 IEEE BigData Congress

2. Unveil Something behind the Paper

3. Highlights
• Applications Design the System Themselves
• Complete Modularization Strategy
• Extreme Simple Stream Model

4. A More Accurate Version of
Development History

5. “Hey guys, there is a new stream
computing technology coming!”

6. Great! Can it make every process go
faster?

7. “OK , that may depend on …”

8. OP1
OP2
OP3
OP4 Mobile Devices
User Filter
Ads
Recalling
Ads
Ranking
Push
Controller
User Preference
Database
Location
Logging API
Ads POI
Database
LBS Ads
Service
DRPC
DRPC
LBS
Ads Query
Can I do this?
• Location Based
Ads Push
System
• Co-Serving With
Online Services

10. OP1
OP2
Redis Cluster
Web Page Cache
OP3 OP4
OP5 User Model
Log Filter
Data Join
Feature
Extraction
Model
Update
Web Crawling
Logging APICan I do this?
• User Preference
Tracking System

12. Stream
Application
Vortex Platform Hadoop Platform
Map-Reduce
Application
OnlineWebServices
Can I co-operate it with m/r?

14. Private Cluster Leased from Public Clusters
Can I scale-out it on leased containers?

16. Handle Fluctuating Workloads?

18. Can it be 100% reliable?

20. Complete Modularization

21. Nukua Automation System
Universal Resource Manger
Spinal DMQ
Stream Computing Core
DRPC Service Interface
Layer 1 : Computing Resources
Layer 2 : Deployment Automation
Layer 3 : Data Transmission
Layer 4 : Topology Representation
Layer 5 : Stream Application
5 Independent Sub-Systems

22. Everything is Configurable
Even for The Message Relay Model

23. Message Queuing Model
Operator 1
Queue A Queue B Queue C
Operator 2

24. Message Passing Model
Operator 1
Operator 2
Operator 3

25. uplink
uplink
uplink
downlink
downlink
downlink
Spinal DMQ Cluster
SCC Operator Cluster
SCC
Master

26. downlink
downlink
downlink
uplink
uplink
OP1 Sub-Links of
OP1
Downstream
Sub-Links of
OP2
Upstream
OP2
Operator
Cluster
Operator
Cluster
Spinal DMQ Cluster
Message Queuing Configuration

27. downlink
downlink
downlink
uplink
uplink
OP1 Sub-Links of
OP1
Downstream
Sub-Links of
OP2
Upstream
OP2
Message Passing Configuration

28. A Simple Stream Model

29. Traditional Stream Model
0 1 2 3 5 6 7 8 n
• Independent Consumer Status
• High Index Overhead
• High Snapshot Overhead

30. Vortex Stream Model
0 n
Head Tail
• Unified Status
• Minimal Index Overhead
• Minimal Snapshot Overhead

31. Lessons Learned
• Highly Configurable System For Flexible Applications
• Big Data Requires Everything Simple & Reliable

32. Thank U!
• G+: chentianjian@gmail.com
• Skype: tianjian_chen
• Linkedin: http://lnkd.in/bRN6xsh