Integrating C* and Spark gives us a system that combines the best of both worlds. The goal of this integration is to obtain a better result than using Spark over HDFS because Cassandra´s philosophy is much closer to RDD's philosophy than what HDFS is. The goal with Cassandra is to have a system that mines all the information stored in C* in a much more efficient way than having the information stored in HDFS. Cassandra data storage and Spark data mining power: an unrivalled mix.

1. AN EFFICIENT DATA MINING SOLUTION

2. Hadoop?

3. Cassandra?

4. Spark?

5. Stratio Deep
An efficient data mining solution
“Two and two are four?
Sometimes… Sometimes they are five.”
G. Orwell
#StratioB

7. Goals
•
•
•
•
#StratioB
Why do you need Cassandra?
What is the problem?
Why do you need Spark?
How do they work together?

8. Cassandra
•
•
•
•
#StratioB
Based on DynamoDB…
Replication, Key/Value, P2P
And based on Big Table…
Column oriented

9. ROBUST
FAST
EFFICENT

10. NO
BOTTLENECK
DECENTRALIZED
REPLICATE
D

11. Another
Databas
e?

12. Why?

13. Case A
One User – Lot of
data
#StratioB

14. Case B
Many User – Few data
#StratioB

15. Case C
Many user – Lot of
data
#StratioB

16. Crawler app
100
M
Indexed
pages
3k
reads
Cassandra, I choose you
#StratioB
Query time
< 1s

17. But…

18. Marketing
walks in

19. New query
“I need to find all the reference to the domain
ACME. I need the answer by Friday.”
#StratioB

20. Problem
Cassandra is not well suited to resolved this
type of queries
You need to design the schema with the query
in mind
#StratioB

21. Challenge
Accepted

22. What options do we have?
•
•
•
#StratioB
Run Hive Query on top of C*
Write an ETL script and load data into
another DB
Clone the cluster

23. What options do we have?
Run Hive Query on top of C*
Write ETL scripts and load into another DB
Clone the cluster
#StratioB

24. And now… what can we do?
“We can't solve problems by using the same kind of thinking
we used when we created them”
Albert Einstein
#StratioB

25. Spark
•
•
•
•
•
Alternative to MapReduce
A low latency cluster computing system
For very large datasets
Create by UC Berkeley AMP Lab in 2010.
May be 100 times faster than MapReduce for:


#StratioB
Interactive algorithms.
Interactive data mining

26. Logistic regression in
Spark vs Hadoop
SOURCE | http://spark.incubator.apache.org/
#StratioB

27. WHO USES SPARK?

28. Spark and
Cassandra
Integration points
#StratioB

29. Cassandra’s HDFS abstraction
layer
Advantantages:
•
Easily integrates with legacy systems.
Drawbacks:
•
•
Very high-level: no access to low level Cassandra’s features.
Questionable performance.
INTEGRATION POINTS: HDFS OVER CASSANDRA
#StratioB

30. Cassandra’s Hadoop
Interface
•
Thrift protocol
•
CQL3 (our implementation)

Uses the novel Cassandra’s
CqlPagingInputFormat
INTEGRATION POINTS: HDFS OVER CASSANDRA
#StratioB

31. CQL3 Integration
•
•
•
Supports CQL3 features
Respects data locality
Good compromise between
performance / implementation complexity
INTEGRATION POINTS: CASSANDRA’S HADOOP INTERFACE – CQL3
#StratioB

32. CQL3 Integration (II)
Provides a Java friendly API:
•
Developers map Column Families to custom serializable
POJOs
•
StratioDeep wraps the complexity of performing Spark
calculations directly over the user provided POJOs.
INTEGRATION POINTS: CASSANDRA’S HADOOP INTERFACE – CQL3
#StratioB

33. Demo

34. CQL3 Integration (III)
Drawbacks:
•
Still not preforming as well as we’d like
Uses Cassandra’s Hadoop Interface
No analyst-friendly interface:

•

#StratioB
No SQL-like query features
INTEGRATION POINTS: CASSANDRA’S HADOOP INTERFACE – CQL3

35. Future extensions
What are we currently working on?
Bring the integration to another level:
•
•
•
#StratioB
Dump Cassandra’s Hadoop Interface
Direct access to Cassandra’s SSTable(s) files.
Extend Cassandra’s CQL3 to make use of Spark’s
distributed data processing power

36. Conclusion
#StratioB

37. THANKS