Finding the needles in the haystack. An Overview of Analyzing Big Data with Hadoop

[object Object],An overview of analyzing big data with Hadoop Presented by Chris Baglieri Architect, Research Architecture & Development Technology Forum, August 30 th , 2010

[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Alright, so we’re not exactly close to them , but we’ll still run into similar problems sooner rather than later.

[object Object],A typical drive from 1990 could store 1370 MB and had a transfer speed of 4.4 MB/s , hence reading all the data in ~5 minutes . Twenty years later, storage has 1 TB if not more is the norm while the rate of access has only climbed to ~100 MB/s , yielding a full read time of >2.5 hours . * Graph extrapolated from data generated on Tom’s Hardware

[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

[object Object],Raw Processing Zookeeper Hadoop Commons Compiled Processing Structured Storage Unstructured Storage Pig Hive Chukwa HDFS MapReduce HBase

[object Object],HDFS is a file system designed for storing very large files (with hooks for smaller files) with streaming data access patterns, running on clusters on commodity hardware .

[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

[object Object],general form => (k1, v1)  list(k2, v2) key => offset from the beginning of a file value => line of text output => { word == Text, 1 == IntWritable } reporter => container that holds metadata about the job

[object Object],general form => (k2, list(v2))  list(k3, v3) key => offset from the start of a file value => collection of word counts where each count is 1 output => { word == Text, number of times it appeared in full text == IntWritable } reporter => container that holds metadata about the job

[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

[object Object],A more interesting example, the above Ruby script (taken from a presentation given by Twitter’s Analytics Lead) breaks down where users are Tweeting from (e.g. the API, the front page, their profile page, etc.). Note the use of scripts in Twitter’s “PiggyBank”.

[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

[object Object],In pioneer days they used oxen for heavy pulling, and when one ox couldn’t budge a log, they didn’t try to grow a larger ox. We shouldn’t be trying for bigger computers, but for more systems of computers. Grace Hopper Fun fact, she also is accredited with “It’s easier to ask for forgiveness than it is to get permission.

Finding the needles in the haystack. An Overview of Analyzing Big Data with Hadoop

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Destacado

Destacado (6)

Similar a Finding the needles in the haystack. An Overview of Analyzing Big Data with Hadoop

Similar a Finding the needles in the haystack. An Overview of Analyzing Big Data with Hadoop (20)

Último

Último (20)

Finding the needles in the haystack. An Overview of Analyzing Big Data with Hadoop