This document discusses efficient data mining solutions using Hadoop, Cassandra, and Spark. It describes Cassandra as a fast, robust, and efficient key-value database but notes it has limitations for certain queries. Spark is presented as an alternative to Hadoop MapReduce that can be 100 times faster for interactive algorithms and data mining. The document demonstrates how Spark can integrate with Cassandra to allow distributed data processing over Cassandra data without needing to clone the data or use other databases. Future extensions are proposed to directly access Cassandra's SSTable files from Spark and extend CQL3 to leverage Spark.

1. AN EFFICIENT DATA MINING SOLUTION

2. Hadoop?

3. Cassandra?

4. Spark?

5. Stratio Deep
An efficient data mining solution
“Two and two are four?
Sometimes… Sometimes they are five.”
G. Orwell
#StratioBD

7. Goals
•
•
•
•
#StratioBD
Why do you need Cassandra?
What is the problem?
Why do you need Spark?
How do they work together?

8. Cassandra
•
•
•
•
#StratioBD
Based on DynamoDB…
Replication, Key/Value, P2P
And based on Big Table…
Column oriented

9. ROBUST
FAST
EFFICENT

10. NO
BOTTLENECK
DECENTRALIZED
REPLICATED

11. Another
Database?

12. Why?

13. Case A
One User – Lot of data
#StratioBD

14. Case B
Many User – Few data
#StratioBD

15. Case C
Many user – Lot of data
#StratioBD

16. Crawler app
100M
Indexed
pages
3k
reads
Cassandra, I choose you
#StratioBD
Query time
< 1s

17. But…

18. Marketing
walks in

19. New query
“I need to find all the reference to the domain ACME.
I need the answer by Friday.”
#StratioBD

20. Problem
Cassandra is not well suited to resolved this type of
queries
You need to design the schema with the query in mind
#StratioBD

21. Challenge
Accepted

22. What options do we have?
•
•
•
#StratioBD
Run Hive Query on top of C*
Write an ETL script and load data into another DB
Clone the cluster

23. What options do we have?
Run Hive Query on top of C*
Write ETL scripts and load into another DB
Clone the cluster
#StratioBD

24. And now… what can we do?
“We can't solve problems by using the same kind
of thinking we used when we created them”
Albert Einstein
#StratioBD

25. Spark
•
•
•
•
•
Alternative to MapReduce
A low latency cluster computing system
For very large datasets
Create by UC Berkeley AMP Lab in 2010.
May be 100 times faster than MapReduce for:


#StratioBD
Interactive algorithms.
Interactive data mining

26. Logistic regression in
Spark vs Hadoop
SOURCE | http://spark.incubator.apache.org/
#StratioBD

27. WHO USES SPARK?

28. Spark and Cassandra
Integration points
#StratioBD

29. Cassandra’s HDFS abstraction layer
Advantantages:
•
Easily integrates with legacy systems.
Drawbacks:
•
•
Very high-level: no access to low level Cassandra’s features.
Questionable performance.
INTEGRATION POINTS: HDFS OVER CASSANDRA
#StratioBD

30. Cassandra’s Hadoop Interface
•
Thrift protocol
•
CQL3 (our implementation)

Uses the novel Cassandra’s CqlPagingInputFormat
INTEGRATION POINTS: HDFS OVER CASSANDRA
#StratioBD

31. CQL3 Integration
•
•
•
Supports CQL3 features
Respects data locality
Good compromise between
performance / implementation complexity
INTEGRATION POINTS: CASSANDRA’S HADOOP INTERFACE – CQL3
#StratioBD

32. CQL3 Integration (II)
Provides a Java friendly API:
•
Developers map Column Families to custom serializable POJOs
•
StratioDeep wraps the complexity of performing Spark calculations
directly over the user provided POJOs.
INTEGRATION POINTS: CASSANDRA’S HADOOP INTERFACE – CQL3
#StratioBD

33. Demo

34. CQL3 Integration (III)
Drawbacks:
•
Still not preforming as well as we’d like

•
No analyst-friendly interface:

#StratioBD
Uses Cassandra’s Hadoop Interface
No SQL-like query features
INTEGRATION POINTS: CASSANDRA’S HADOOP INTERFACE – CQL3

35. Future extensions
What are we currently working on?
Bring the integration to another level:
•
•
•
#StratioBD
Dump Cassandra’s Hadoop Interface
Direct access to Cassandra’s SSTable(s) files.
Extend Cassandra’s CQL3 to make use of Spark’s distributed
data processing power

36. Conclusion
#StratioBD

37. THANKS