For some, Hadoop is synonymous with “Big Data,” but Hadoop is just one component of a successful Big Data architecture. Depending on one’s application, it may not even be the most important part. NoSQL solutions like MongoDB also play a dominant role for storage and real-time data processing, helping companies keep pace with the scale of their data requirements. But NoSQL figures even more prominently in helping enterprises consume a wide variety of data sources at speeds not currently possible in Hadoop. NoSQL, then, offers a useful complement to Hadoop, as well as the transaction-based data of traditional RDBMSs. Tackling Big Data is not a one-tool job, and so the orchestration of the appropriate NoSQL database with Hadoop and RDBMS is essential. In this session, we’ll dig deep into the different types of NoSQL, identifying how they differ and the types of Big Data workloads for which they’re best suited. We’ll also explore the trade-offs one makes in choosing NoSQL databases like MongoDB or Neo4j over an RDBMS like MySQL, and when it makes sense to use both Hadoop and NoSQL and when it’s more appropriate to use NoSQL on its own.

1. Essential Tools For
Your Big Data Arsenal
Matt Asay (@mjasay)
VP, Business Development & Strategy, MongoDB

2. The Big Data Unknown

3. Top Big Data Challenges?
Translation?
Most struggle
to know what
Big Data
is, how to
manage it and
who can
manage it
Source: Gartner
3

4. Understanding Big Data – It’s Not Very “Big”
64% - Ingest
diverse, new data in
real-time
15% - More than 100TB
of data
20% - Less than 100TB
(average of all? <20TB)
from Big Data Executive Summary – 50+ top executives from Government and F500 firms
4

5. Innovation As Iteration

6. “I have not failed. I've just found 10,000 ways that won't work.”
― Thomas A. Edison

7. Back in 1970…Cars Were Great!
7

8. So Were Computers!
8

9. Lots of Great Innovations Since 1970
9

10. Including the Relational Database
10

11. RDBMS Makes Development Hard
Code
DB Schema
Application
11
XML Config
Object Relational
Mapping
Relational
Database

12. And Even Harder To Iterate
New
Table
New
Column
New
Table
Name
Pet
Phone
New
Column
3 months later…
12
Email

13. From Complexity to Simplicity
RDBMS
MongoDB
{
_id : ObjectId("4c4ba5e5e8aabf3"),
employee_name: "Dunham, Justin",
department : "Marketing",
title : "Product Manager, Web",
report_up: "Neray, Graham",
pay_band: “C",
benefits : [
{
type :
"Health",
plan : "PPO Plus" },
{
type :
"Dental",
plan : "Standard" }
]
}
13

14. So…Use Open Source
14

15. Big Data != Big Upfront Payment
15

16. RDBMS Is Expensive To Scale
“Clients can also opt to run zEC12 without a raised
datacenter floor -- a first for high-end IBM mainframes.”
IBM Press Release 28 Aug, 2012
16

17. Spoiled for choice
DB-Engines.com Database Ranking
1 Oracle
2 MySQL
3 Microsoft SQL Server
4 PostgreSQL
5 DB2
6 MongoDB
7 Microsoft Access
8 SQLite
9 Sybase
10 Teradata
17
Relational DBMS
1583.84
Relational DBMS
1331.34
Relational DBMS
1207
Relational DBMS
177.01
Relational DBMS
175.83
NoSQL Document Store 149.48
Relational DBMS
142.49
Relational DBMS
77.88
Relational DBMS
73.66
Relational DBMS
54.41
54.23
25.58
-106.78
-5.22
3.58
-2.71
-4.21
-4.9
-1.68
3.32

18. Remember the Long Tail?
18

19. It Didn’t Work Out So Well
19

20. Use Popular, Well-Known Technologies
20
Source: Silicon Angle, 2012

21. Ask the Right Questions…
“Organizations already have people who know
their own data better than mystical data
scientists….Learning Hadoop [or MongoDB] is
easier than learning the company’s business.”
(Gartner, 2012)
21

22. Leverage Existing Skills
22

23. Search as a Sign?
23

24. When To Use Hadoop, NoSQL

25. 25
Applications
CRM, ERP, Collaboration, Mobile, BI
Data Management
Online Data
RDBMS
RDBMS
Offline Data
Hadoop
Infrastructure
OS & Virtualization, Compute, Storage, Network
EDW
Security & Auditing
Management & Monitoring
Enterprise Big Data Stack

26. Consideration – Online vs. Offline
Online
• Real-time
• Low-latency
• High availability
26
vs.
Offline
• Long-running
• High-Latency
• Availability is lower priority

27. Consideration – Online vs. Offline
Online
27
vs.
Offline

28. Hadoop Is Good for…
Risk Modeling
Recommendation
Engine
Ad Targeting
Transaction
Analysis
Trade
Surveillance
Network Failure
Prediction
28
Churn Analysis
Search Quality
Data Lake

29. MongoDB/NoSQL Is Good for…
360° View of the
Customer
Fraud Detection
User Data
Management
Content
Management &
Delivery
Reference Data
Product Catalogs
29
Mobile & Social
Apps
Machine to
Machine Apps
Data Hub

30. How To Use The Two Together?

31. Finding Waldo
31

32. Customer example: Online Travel
Travel
Algorithms
MongoDB
Connector for
Hadoop
•
•
•
•
32
Flights, hotels and cars
Real-time offers
User profiles, reviews
User metadata (previous
purchases, clicks, views)
•
•
•
•
User segmentation
Offer recommendation engine
Ad serving engine
Bundling engine

33. Predictive Analytics
Government
Algorithms
MongoDB
+ Hadoop
• Predictive analytics system
for crime, health issues
• Diverse, unstructured (incl.
geospatial) data from 30+
agencies
• Correlate data in real-time
33
• Long-form trend analysis
• MongoDB data dumped into
Hadoop, analyzed, re-inserted
into MongoDB for better realtime response

34. Data Hub
Churn
Analysis
Insurance
MongoDB
Connector for
Hadoop
•
•
•
•
•
34
Insurance policies
Demographic data
Customer web data
Call center data
Real-time churn detection
• Customer action analysis
• Churn prediction
algorithms

35. Machine Learning
Ad-Serving
Algorithms
MongoDB
Connector for
Hadoop
•
•
•
•
•
35
Catalogs and products
User profiles
Clicks
Views
Transactions
• User segmentation
• Recommendation engine
• Prediction engine

36. MongoDB + Hadoop Connector
• Makes MongoDB a Hadoop-enabled file system
• Read and write to live data, in-place
• Copy data between Hadoop and MongoDB
• Full support for data processing
– Hive
– MapReduce
– Pig
– Streaming
– EMR
36
MongoDB
Connector for
Hadoop

37. @mjasay