Big data

Introduction to Big Data Survival Guide!

Luan Cestari
February 28 , 2014

1

RED HAT ENTERPRISE LINUX – FOUNDATION FOR THE OPEN HYBRID CLOUD

Please, let me ask ...
●

●

2

Who already tested a product/project related to Big
Data?
Who does work with Big Data?


What are we going to see here
●

The demystification the term ¨Big Data¨ and beyond!
●
●

What does the people claim to be Big Data
What is the relationship between Big Data and
databases
●
●

●

Some facts about database history
Why there are so many DB available?

How to clue all this stuff together?
●

3

Some well-known Hadoop ecosystem tools that cover a very
wide of Big Data issues


Why Big Data is important
●

Many companies is already dealing with Big Data
using Open Source tools
●

●

4

There is demand for people to work with those tools as
a developer and analyst
You can also work with some integration between those
system and building to improve a already existing tool or
the next Big Data Tool


●

When a company is using Big Data tools, it can grow
very fast and complex:
●

●

●

5

Many different clusters (due tenant, geo localized or
different versions)
Different technologies for very related propose (also due
different team skills or use cases)
Many many software integration, layers to segregate the
different aspects and re factoring due the the fast pace


Cool ... but what is Big Data after all?
●

Just tons of information isn't enough, it also needs to
be have:
●
●

Velocity

●

Value

●

6

Variety

And Volume


More about Volume: How Big it can be?
●

What is the size of daily batch job from Facebook? 100
GB 10000GB 100000GB?
●

7

Answer:104 857 600 gigabytes of users log


More about Variety: Where the data are from?
●

Customer generated Content

●

M2M

●

Sensors

●

B2B

●

B2C

●

Social Network

●

8

And others Devices: mobile phones, setbox, Security
Cameras


More about Value
●

The value is about the processing the data in a
reasonable period of time, so you can forecast
something. Because of that you will need some data
scientists, so they can do:
●

9

Analysts (find correlations using statistics, signal
processing, machine learning, persona, etc) using
different kind of tools (SQL, search engines, stream
processing)


More about Value
●

scientists, so they can:
●

10

Find correlations using statistical or predictive analytics,
signal processing, machine learning, natural language
processing, BI, visualization, etc using different kind of
tools (SQL, search engines, stream processing)


More about Value
●

●

11

So the value are the insights generated that may help
you to generate a better product, making better
decision or take a competitive advantage over the
other competitors
The Open Source helps also the value to enable it in a
cost effective way, instead buying tons of expensive
tools


... and the Velocity
●

This is a very interesting point due different analyzes
may require different times:
●

●

12

A traffic system may need a streaming system to
analyze and predict the actual traffic and suggest better
routes over the city
The same traffic system may need to process several
weeks to have a good prediction of the average traffic
over the road, so that could be an offline batch


●

13

The main point is that there isn't a silver bullet for this,
different store system may be required for different
services that it aims to provide


SQL History
●

●

Hierarchical Database in 60`s
Then Relational Database in 80`s and until couple
years ago was the only solution used in most of the
enterprise
●

14

Big companies used to buy expensive special DW
database system to analyze their data


... and now

15


... and now

16


Again the reason for that
●

For example the Web Analysis in Facebook:
●
●

+240 Billion photos

●

+1 Trillion connections

●

●

+1 Billion users

22% of references of the Internet

Harvard Business Review
●

●

17

A change from DW to a Big Data system made a 96
hours job run in just 4 hours
2012 2.5 exabyte create a day


We need to avoid the Golden hammer/Silver
Bullet Anti-pattern

18


Hadoop ecosystem save the day
●

●

Open Source projects that help you to deal with the Big
Data
Don't need vertical scaling (big machines), you ca use
cluster of commodity machines and archive even
better results
●

Parallel Processing

●

Fault tolerant Jobs

●

Redundant and distributed data (for disk failure and to
avoid moving data around)

●
●

19

Less complex programming model
It have low level native lib for high performance

●

●

But the Hadoop file system (HDFS) doesn't handle well
low latency requests and small files =(
Well, there isn't silver bullet, we need more tools
●

20

so this is why Hadoop is not alone, there are many
different projects which integrate with it


●

●

●

●

There are several big companies that offer Hadoop and
other projects as a big product and they help the
community, I will talk a little more about Hortonworks
and Cloudera`s projects sets as they are very wellknown and how they integrate. Find more on
http://wiki.apache.org/hadoop/Distributions%20and%20Commercial%20Support

21


●

22

Cluadera: CDH


●

Cluadera:
●

23

How to create this whole stack with minimum effort:
Cloudera Manager


●

24

Hortonworks: HDP


●

Hortonworks:
●

●

25

They use Ambari to management the cluster like
Claudera Manager does
They also have Tez to enhance the speed of the
workloads


●

And more tools:
●

●

26

You may use Apache Mesos or Hadoop 2 YARN to
better manage and sharing your services (for example
tenants/cloud)
Apache BigTop, Fuse-DFS, Apache Crunch, Apache
Whirr, Apache Hama,Apache Giraph, Open MPI,
Cascading (and its extensions), Weave, and more


●

27

There more tools for specific cases, like low latency
with Spark ecosystem


●

28

But you can also use other tools for low latency such
as Twitter Storm, Yahoo S4, Linkedin Samza (or
Kafka), Amazon Kinesis, Google Millwheel


The integration with other system will be complex
●

29

An overview:


A different approach: Lambda Architecture
●

30

Idea from Twitter Team (like Nathan Marz) about how
to deal with Big Data Systems


Questions?

31


Please, let me ask ...
●

●

2

Who already tested a product/project related to Big
Data?
Who does work with Big Data?


Scalable
Portable
On-demand
Resource Management
Measureable

What are we going to see here
●

The demystification the term ¨Big Data¨ and beyond!
●
●

What does the people claim to be Big Data
What is the relationship between Big Data and
databases
●
●

●

How to clue all this stuff together?
●

3

Some facts about database history
Why there are so many DB available?
Some well-known Hadoop ecosystem tools that cover a very
wide of Big Data issues


The difference in http://www.slideshare.net/CAinc/cloud-expo-session-fromvirtualization-to-cloud-computing-building-an-effective-pragmatic-reliable-cloud

●

Many companies is already dealing with Big Data
using Open Source tools
●

●

4

There is demand for people to work with those tools as
a developer and analyst
You can also work with some integration between those
system and building to improve a already existing tool or
the next Big Data Tool


4

●

When a company is using Big Data tools, it can grow
very fast and complex:
●

●

●

5

Many different clusters (due tenant, geo localized or
different versions)
Different technologies for very related propose (also due
different team skills or use cases)
Many many software integration, layers to segregate the
different aspects and re factoring due the the fast pace


5

Cool ... but what is Big Data after all?
●

Just tons of information isn't enough, it also needs to
be have:
●
●

Velocity

●

Value

●

6

Variety

And Volume


6

More about Volume: How Big it can be?
●

What is the size of daily batch job from Facebook? 100
GB 10000GB 100000GB?
●

7

Answer:104 857 600 gigabytes of users log


7

More about Variety: Where the data are from?
●

Customer generated Content

●

M2M

●

Sensors

●

B2B

●

B2C

●

Social Network

●

8

And others Devices: mobile phones, setbox, Security
Cameras


8

More about Value
●

scientists, so they can do:
●

9

Analysts (find correlations using statistics, signal
processing, machine learning, persona, etc) using
different kind of tools (SQL, search engines, stream
processing)


9

More about Value
●

scientists, so they can:
●

10

Find correlations using statistical or predictive analytics,
signal processing, machine learning, natural language
processing, BI, visualization, etc using different kind of
tools (SQL, search engines, stream processing)


10

More about Value
●

●

11

So the value are the insights generated that may help
you to generate a better product, making better
decision or take a competitive advantage over the
other competitors
The Open Source helps also the value to enable it in a
cost effective way, instead buying tons of expensive
tools


11

●

This is a very interesting point due different analyzes
may require different times:
●

●

12

A traffic system may need a streaming system to
analyze and predict the actual traffic and suggest better
routes over the city
The same traffic system may need to process several
weeks to have a good prediction of the average traffic
over the road, so that could be an offline batch


12

●

13

The main point is that there isn't a silver bullet for this,
different store system may be required for different
services that it aims to provide


13

SQL History
●

●

Hierarchical Database in 60`s
Then Relational Database in 80`s and until couple
years ago was the only solution used in most of the
enterprise
●

14

Big companies used to buy expensive special DW
database system to analyze their data


14

... and now

15


15

... and now

16


16

Again the reason for that
●

For example the Web Analysis in Facebook:
●
●

+240 Billion photos

●

+1 Trillion connections

●

●

+1 Billion users

22% of references of the Internet

Harvard Business Review
●

●

17

A change from DW to a Big Data system made a 96
hours job run in just 4 hours
2012 2.5 exabyte create a day


17

We need to avoid the Golden hammer/Silver
Bullet Anti-pattern

18


18

●

●

Open Source projects that help you to deal with the Big
Data
Don't need vertical scaling (big machines), you ca use
cluster of commodity machines and archive even
better results
●

Parallel Processing

●

Fault tolerant Jobs

●

Redundant and distributed data (for disk failure and to
avoid moving data around)

●
●

19

Less complex programming model
It have low level native lib for high performance

19

●

●

●

20



20

●

●

●

●

There are several big companies that offer Hadoop and
other projects as a big product and they help the
community, I will talk a little more about Hortonworks
and Cloudera`s projects sets as they are very wellknown and how they integrate. Find more on
http://wiki.apache.org/hadoop/Distributions%20and%20Commercial%20Support

21


21

●

22

Cluadera: CDH


Apache Sqoop is a tool designed for efficiently
transferring bulk data between Apache Hadoop and
structured datastores such as relational databases.

22

●

Cluadera:
●

23

How to create this whole stack with minimum effort:
Cloudera Manager


23

●

24

Hortonworks: HDP


Oozie is a workflow scheduler system to manage
Apache Hadoop jobs.
Oozie Workflow jobs are Directed Acyclical Graphs
(DAGs) of actions.
Oozie Coordinator jobs are recurrent Oozie Workflow
jobs triggered by time (frequency) and data
availabilty

24

●

Hortonworks:
●

●

25

They use Ambari to management the cluster like
Claudera Manager does
They also have Tez to enhance the speed of the
workloads


25

●

And more tools:
●

●

26

You may use Apache Mesos or Hadoop 2 YARN to
better manage and sharing your services (for example
tenants/cloud)
Apache BigTop, Fuse-DFS, Apache Crunch, Apache
Whirr, Apache Hama,Apache Giraph, Open MPI,
Cascading (and its extensions), Weave, and more


Apache Whirr is a set of libraries for running cloud
services.
The Apache Crunch Java library provides a
framework for writing, testing, and running
MapReduce pipelines. Its goal is to make pipelines
that are composed of many user-defined functions
simple to write, easy to test, and efficient to run.
Open MPI is a standardized API typically used for
parallel and/or distributed computing

26

●

27

There more tools for specific cases, like low latency
with Spark ecosystem


services.

27

●

28

But you can also use other tools for low latency such
as Twitter Storm, Yahoo S4, Linkedin Samza (or
Kafka), Amazon Kinesis, Google Millwheel


services.

28

The integration with other system will be complex
●

29

An overview:


29

A different approach: Lambda Architecture
●

30

Idea from Twitter Team (like Nathan Marz) about how
to deal with Big Data Systems


30

Big data

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (18)

Destacado

Destacado (8)

Similar a Big data

Similar a Big data (20)

Más de Luan Cestari

Más de Luan Cestari (8)

Último

Último (20)

Big data