Roundtable 2: Big Data Analytics and NoSQL

DB Revolution: 2nd Roundtable
Wednesday, March 14, 12

Eric Kavanagh
Eric.kavanagh@bloorgroup.com

Twitter Tag: #briefr

To conduct an Open Research program that
invites the participation of both IT users and
technology vendors

To assist IT buyers in understanding database
technology and the architecture that surrounds
it.

Allow audience members to pose serious
questions... and get answers!

Publish all findings


Your Host: Eric Kavanagh
Research Leader: Mark Madsen - Third Nature
Primary Collaborator: Robin Bloor - The Bloor
Group
Guest Analyst 1: Colin White - BI Research
Guest Analyst 2: Steve Dine - DataSource
Consulting


Colin White is the president of
DataBase Associates Inc. and founder
of BI Research. He is well known for
his in-depth knowledge of data
management, information integration,
and business intelligence technologies.
He has consulted for dozens of
companies throughout the world and is
a frequent speaker at leading IT
events. For ten years he was the
conference chair of the DCI and Shared
Insights Portals, Content Management,
and Collaboration conference.


Big Data is Bigger than NoSQL

Colin White
President BI Research
March 2012


What is Big Data?

A term that represents
workloads and data
management solutions that
could not previously be
supported because of cost
considerations and/or
technology limitations
Three important technologies:
•Optimized analytic RDBMSs
•Non-relational “NoSQL” systems
•Stream processing systems

Copyright © BI Research, 2012 2


Big Data: The Business Case

Smarter Decisions
• Analyze new sources of data e.g., sensor data, web content,
systems logs, text, XML files, graph data, map data, etc.
• More sophisticated analyses - advanced analytics

Faster Decisions
• Supports workloads that were difficult to implement previously in
a timely or cost-effective manner
• Faster data analysis, e.g., analysis of large detailed data stores,
dramatic increase in analytic model execution

Faster Time to Value
• Analyze data that is outside of the enterprise data warehouse,
e.g., machine-generated data such as sensor data



Non-Relational Solutions

Some organizations have developed their
own non-relational (NoSQL) systems to
support extreme workloads
• Google: MapReduce + BigTable DBMS +
Google File System

Non-relational systems are not new, but
modern versions are often available to the
open source community
• Often support commodity hardware in a large-
scale distributed computing environment
• Several types of data stores (key value,
graph, document, indexed file/DB systems)
• A key vendor focus area is the Hadoop
distributed computing system



Hadoop versus an RDBMS
This debate is reminiscent of the object versus relational database
debates of the 1980s, and the reasons are similar
• Programmers prefer procedural programmatic approaches for accessing
and manipulating data, e.g., MapReduce
• Non-programmers prefer declarative languages, e.g., RDBMSs and SQL

Adding the Hive SQL-like language to Hadoop, and MR functions to
RDBMSs, however, complicates the debate
Key requirements are:
• The ability for organizations to easily analyze large volumes of multi-
structured data with good price/performance
• The need to make technologies for developing and running these
analyses more usable by data scientists

Organizations will likely use Hadoop and an RDBMS - the challenges
are deciding which to use when and interconnecting the systems



The Value of Big Data: McKinsey Report

www.mckinsey.com/Insights/MGI/Research/
Technology_and_Innovation/Big_data_The_next_frontier_for_innovation



Robin Bloor is Chief
Analyst at The
Bloor Group.

Robin.Bloor@Bloorgroup.com


The Hardware Landscape
CPUs go multicore
Memory/Disk cost ratio falls
Speed of random reads lag speed
of serial reads
Faster networking and fast
switches
Parallelism becomes more
important
Commodity servers
Cloud computing cuts H/W costs


That MapReduce Thing
There are two fundamental
approaches to parallelism
Data Partitioning
Process partitioning
MapReduce implements an
approach which is oriented to
data partitioning
This relates to data processing
rather than to database
Hadoop is often used for ETL


The Devil Is In The Workload
NoSQL is a distraction
Big Data can be Big US Big XML

Data or Big SDATA D
Table Store

A

Unstructured T
A Column
Store
Document
Store
workloads are rarely V
O
suited to traditional L
U
RDBMS ODBMS

RDMBS-type engines
M
E
Database Database

Analytical workloads
span both More
Structured
Less
Structured


If you don’t know the expected
workloads, you shouldn’t be
selecting a database


Steve Dine is the founder of Datasource
Consulting, LLC. He has extensive experience
delivering and managing successful, highly
scalable and maintainable data integration and
business intelligence solutions. Steve combines
hands-on technical experience across the
entire BI project lifecycle with strong business
acumen. He currently works as a consultant for
Fortune 500 companies. Steve is a faculty
member at TDWI and a judge for the Annual
TDWI Best Practices Awards. He teaches
courses and presents on many BI topics.
Contact info: Twitter: @steve_dineEmail:
sdine@datasourceconsulting.com Web: http://
www.datasourceconsulting.com


The State of NoSQL & BI
From the trenches…

“Hey
Bob,
seems
like
a
no
brainer.

So,
what’s
the
catch?”
*
Graphic
from
h=p://schri@man.wordpress.com/category/booksbook-‐reviews/c-‐s-‐lewis/page/2/
Conﬁden)al,
Datasource
Consul)ng,
LLC

19

Why NoSQL?

More
data
More
different
types
of
data
(semi-‐structured,

unstructured)
More
frequent
changes
to
the
structure
of
the
data
we

need
to
store
and
analyze
More
demand
for
the
long
tail
analysis
More
“affordable”,
commodity
hardware
available
(blade

servers,
“cheap”
storage,
cloud)
More
buzz!
*
Graphic
from
h=p://www.fredberinger.com/musings-‐on-‐nosql/
Confiden)al,
Datasource
Consul)ng,
LLC

20

Why Not Not NoSQL?

RelaCvely
immature
(0.x
–
2.x)
Difficult
to
describe
to
decision
makers
Not
fit
for
purpose
(low
latency,
update
heavy,
complex

joins)
In
many
organizaCons
it’s
a
soluCon
looking
for
a
problem

Lack
of
“BI”
support
Skills
gap!

*
Graphic
based
on
h=p://www.fredberinger.com/musings-‐on-‐nosql/
Confiden)al,
Datasource
Consul)ng,
LLC

21

BI-NoSQL Skills Gap

“SQL”
Skills NoSQL
Skills

•
GUI’s
(mostly) •
Command
Line
•
Rela)onal
Data
•
Key-‐Value
/
Column

Modeling
Family
Modeling
•
RDBMS
•
Distributed
Data

•
SQL Store
•
Stored
procedures •
Programming
(Java,

•
LDAP Jscript,
Python,
etc)
•
Javascript •
MapReduce
(Hive)
•
Batch/Shell
Scripts •
JSON
•
Shell
Scripts

*
Graphic
based
on
h=p://www.beckshome.com/index.php/2007/09/the-‐soa-‐chasm/
Conﬁden)al,
Datasource
Consul)ng,
LLC

22

Conclusions?

• Best
to
evaluate
your
true
data
size,
data
growth,
data
formats,
data

structure
and
analyCc
requirements
before
deciding
on
soluCon
• Make
sure
to
evaluate
your
available
skills
• Experienced
NoSQL
resources
with
BI
experience
not
always
easy
to

ﬁnd
• Need
to
plan
for
addiConal
technology
risk
in
project
plan

• Consider
starCng
out
with
one
part
of
your
DW
architecture
(i.e.

staging)
• POC
POC
POC
• NoSQL
maturing
quickly
and
will
likely
conCnue
to
evolve
into
a
hybrid

soluCon

Conﬁden)al,
Datasource
Consul)ng,
LLC

23

Mark Madsen is founder of Third Nature, a
research and consulting firm focused on
analytics, BI and decision-making. Mark
spent the past two decades working on
analysis and decision support in many
industries and countries. He is an award-
winning architect and former CTO whose
work has been featured in numerous
industry publications. Over the past ten
years Mark received awards for his work
from the American Productivity & Quality
Center, TDWI, and the Smithsonian Institute.
He is an international speaker, a contributing
editor at Intelligent Enterprise, and manages
the open source channel at the Business
Intelligence Network. For more information
or to contact Mark, visit http://
ThirdNature.net.


One Size Doesn’t Fit All
Choosing which big data,
NoSQL or database
technology to use

March 14, 2012

Mark R. Madsen
http://ThirdNature.net


Big
data?

Unstructured
data
isn’t
really

unstructured.
The
problem
is
that
this
data

is
unmodeled.
The
real
challenge
is

complexity.


The
holy
grail
of
databases
under
current
market
hype

A
key
problem
is
that
we’re

talking
mostly
about

computa?on
over
data

when
we
talk
about
“big

data”
and
analy?cs,
a

poten?al
mismatch
for
both

rela?onal
and
nosql.

Solving
the
Problem
Depends
on
the
Diagnosis


You
must
understand
your

workload
-‐
throughput
and

response
=me
requirements

aren’t
enough.
▪ 100
simple
queries
accessing

month-‐to-‐date
data
▪ 90
simple
queries
accessing

month-‐to-‐date
data
plus
10

complex
queries
using
two
years

of
history
▪ Hazard
calculaCon
for
the
enCre

customer
master
▪ Performance
problems
are
rarely

due
to
a
single
factor.


Workload:
One
big
query
or
many
small
queries?

Retrieval: small return set or large?
Selectivity: large volume of data scanned or small?

Important
workload
parameters
to
know
• Read-‐intensive

vs.
write-‐intensive


Important
workload
parameters
to
know

vs.
write-‐intensive
• Mutable
vs.
immutable
data


Important
workload
parameters
to
know

vs.
write-‐intensive
• Mutable
vs.
immutable
data
• Immediate
vs.
eventual
consistency


Important
workload
parameters
to
know

vs.
write-‐intensive
• Mutable
vs.
immutable
data
• Immediate
vs.
eventual
consistency
• Short
vs.
long
access
latency


Important
workload
parameters
to
know

vs.
write-‐intensive
• Mutable
vs.
immutable
data
• Immediate
vs.
eventual
consistency
• Short
vs.
long
access
latency
• Predictable
vs.
unpredictable
data
access
paEerns


Types
of
workloads
Write-‐biased:
Read-‐biased:
▪ OLTP Query
▪ OLTP,
batch Query,
simple
retrieval
▪ OLTP,
lite Query,
complex
▪ Object
persistence Query-‐hierarchical
/

▪ Data
ingest,
batch object
/
network
▪ Data
ingest,
real-‐Cme AnalyCc

Mixed?
Inline analytic execution, operational BI


Matching
to
parameters,
at
assumpCon
of
data
scale
Workload
Write-‐ Read-‐ Updateable
Eventual
Un-‐ Compute

parameters biased biased data consistency
predictable
intensive
ok query
path
Standard

RDBMS
Parallel

RDBMS
NoSQL
(kv,

dht,
obj)
Hadoop*

Streaming

database

You see the problem: it’s an intersection of multiple parameters, and this
chart only includes the first tier of parameters. Plus, workload factors can
completely invert these general rules of thumb.

Matching
to
parameters,
at
assumpCon
of
data
scale
Workload
Complex
SelecCve
Low
latency
High
High
ingest

parameters queries queries queries concurrency rate

Standard

RDBMS
Parallel
RDBMS

NoSQL
(kv,
dht,

obj)
Hadoop

Streaming

database

You have to look at the combination of workload factors: data scale,
concurrency, latency & response time, then chart the parameters.


Always
build
a
proof
of
concept!


Disection &
Discussion


March:
Vendor Research
March 14th: Second Round Table focusing on No SQL databases and
their application
DB Revolution Survey conducted

April:
Vendor Research
Publishing of Round Table Transcripts, with comments
May:
Authoring of White Paper
Publishing of White Paper
Publishing of survey activity


March Briefing Room:
Integration

April Briefing Room:
Discovery

May Briefing Room: Analytics


Thank You
For Your
Attention


Roundtable 2: Big Data Analytics and NoSQL

Recomendados

Recomendados

Más contenido relacionado

Más de Inside Analysis

Más de Inside Analysis (20)

Último

Último (20)

Roundtable 2: Big Data Analytics and NoSQL