In today’s world of exponentially growing big data, enterprises are becoming increasingly more aware of the business utility and necessity of harnessing, storing and analyzing this information. Apache Hadoop has rapidly evolved to become a leading platform for managing and processing big data, with the vital management, monitoring, metadata and integration services required by organizations to glean maximum business value and intelligence from their burgeoning amounts of information on customers, web trends, products and competitive markets. In this session, Hortonworks' Himanshu Bari will discuss the opportunities for deriving business value from big data by looking at how organizations utilize Hadoop to store, transform and refine large volumes of this multi-structured information. Connolly will also discuss the evolution of Apache Hadoop and where it is headed, the component requirements of a Hadoop-powered platform, as well as solution architectures that allow for Hadoop integration with existing data discovery and data warehouse platforms. In addition, he will look at real-world use cases where Hadoop has helped to produce more business value, augment productivity or identify new and potentially lucrative opportunities.

1. © Hortonworks Inc. 2013
The Rise of Apache Hadoop…
…and its Role in Enterprise Data Architectures
Himanshu Bari
Sr. Product Manager, Hortonworks
Page 1

2. © Hortonworks Inc. 2013
Topics
Page 2
Market trends &
emergence of
Hadoop
Hadoop’s role
and future
direction in the
Enterprise
Enterprise
Hadoop use
cases

3. © Hortonworks Inc. 2013
Big Data & Big Impact
Page 3
Big Data
15x
growth rate of
machine
generated
data by 2020
Source: IDC
Big Impact
20%
Percentage by which
companies leveraging data
will outperform their peers
1.5M Data Savvy managers
needed
Source: Mckinsey

4. © Hortonworks Inc. 2013
2013: CIOs take note…
Page 4
2013 STATE OF THE CIO SURVEY (Jan 2013)
When it comes to adoption of Big Data:
• 34% of IT executives surveyed classify their organization as late majority
• 25% of IT executives surveyed classify their organization as laggards

5. © Hortonworks Inc. 2013
What is Apache Hadoop?
Page 5
D D
.... D
C C C
Petabyte scale reliable storage
management (HDFS) on commodity disks
Highly distributed computation framework
( MapReduce)
Apache Hadoop = Open Source Data Management Software

6. © Hortonworks Inc. 2013
Quick History: Hadoop at Yahoo!
Source: http://developer.yahoo.com/blogs/ydn/posts/2013/02/hadoop-at-yahoo-more-than-ever-before/
Page 6

7. © Hortonworks Inc. 2013
Topics
Page 7
Market Trends
& Emergence of
Hadoop
Hadoop’s role
and future
direction in the
Enterprise
Enterprise
Hadoop Use
Cases

8. © Hortonworks Inc. 2013
Current Data ArchitectureAPPLICATIONS
DATA
SYSTEMS
TRADITIONAL
REPOS
RDBMS
EDW
MPP
DATA
SOURCES
OLTP,
POS
SYSTEMS
OPERATIONAL
TOOLS
MANAGE
&
MONITOR
Tradi:onal
Sources
(RDBMS,
OLTP,
OLAP)
DEV
&
DATA
TOOLS
BUILD
&
TEST
Business
Analy:cs
Custom
Applica:ons
Packaged
Applica:ons
Page 8
ETL/ELT

9. © Hortonworks Inc. 2013
Current Data Architecture PressuredAPPLICATIONS
DATA
SYSTEMS
TRADITIONAL
REPOS
RDBMS
EDW
MPP
DATA
SOURCES
OLTP,
POS
SYSTEMS
OPERATIONAL
TOOLS
MANAGE
&
MONITOR
Tradi:onal
Sources
(RDBMS,
OLTP,
OLAP)
New
Sources
85% data growth
(sen:ment,
clickstream,
geo,
sensor,
…)
DEV
&
DATA
TOOLS
BUILD
&
TEST
Business
Analy:cs
Custom
Applica:ons
Packaged
Applica:ons
Page 9
ETL/ELT

10. © Hortonworks Inc. 2013
Next generation data architectureAPPLICATIONS
DATA
SYSTEMS
TRADITIONAL
REPOS
RDBMS
EDW
MPP
DATA
SOURCES
OLTP,
POS
SYSTEMS
OPERATIONAL
TOOLS
MANAGE
&
MONITOR
Tradi:onal
Sources
(RDBMS,
OLTP,
OLAP)
New
Sources
85% data growth
(sen:ment,
clickstream,
geo,
sensor,
…)
DEV
&
DATA
TOOLS
BUILD
&
TEST
Business
Analy:cs
Custom
Applica:ons
Packaged
Applica:ons
ENTERPRISE
HADOOP
PLATFORM
Page 10
ETL/ELT

11. © Hortonworks Inc. 2013
New architecture enables schema on read
Page 11
OLD WAY HADOOP WAY
Define table
with
Schema
Load only
table
conforming
data
CHANGE
?
Fight for eternity
Load
COMPLETE
data in
Hadoop
Read data
as you like
CHANGE
?
Just read differently

12. © Hortonworks Inc. 2013
OS/VM
Cloud
Appliance
ENTERPRISE
HADOOP
PLATFORM
Evolution of Enterprise Hadoop
Page 12
HADOOP
CORE
PLATFORM
SERVICES
DATA
SERVICES
S:nger
HIVE
&
HCATALOG
PIG
HBASE
SQOOP
FLUME
NFS
WebHDFS
HDFS
MAP
REDUCE
YARN
TEZ
OTHER
OPERATIONAL
SERVICES
OOZIE
AMBARI
FALCON
Enterprise Readiness
High Availability, Disaster
Recovery, Rolling Upgrades,
Security and Snapshots
KNOX
OpenStack

13. © Hortonworks Inc. 2013
YARN: General purpose resource
management framework
• Why is it needed?
– New ways of data processing graph and
stream processing have different resource
management needs than mapreduce
– Need to improve scalability & utilization of
the clusters
– Support multiple versions of mapreduce
• How does it work?
– Splits JobTracker responsibilities into a
global resource manager and a per-
application ApplicationMaster
– Provides an extendible framework HDFS
MapReduce
Redundant, Reliable Storage
YARN:
Cluster
Resource
Management
Tez
Stream
Processing
Other
…
Page 13
HADOOP
CORE

14. © Hortonworks Inc. 2013
Apache Tez (“Speed”): Alternative to
MapReduce
• Why is it needed?
– Widens the platform for Hadoop use cases beyond batch
– Crucial to improving the performance of low-latency applications
• Core idea-
– Create a pool of pre-allocated containers
– Reuse containers for multiple tasks
Page 14
pluggable
input
Pluggable
Processor
Task
Pluggable
Output
HADOOP
CORE

15. © Hortonworks Inc. 2013
Stinger: Improve Hive performance
and SQL compliance
Page 15
Improves existing
tools & preserves
investments
Enable Hive to
support interactive
workloads
Stinger Project
Simple Focus
Query
Planner
Hive
Execution
Engine
Tez
= 100X+
New File
Format
ORC file
= SQL Compliance
+
Data
Types
Windowing
&
Subqueries
+
DATA
SERVICES

16. © Hortonworks Inc. 2013
Falcon: One-stop Shop for Data
Lifecycle management(DLM)
Data Management Needs Tools
Data Processing Oozie
Replication Sqoop
Retention Distcp
Scheduling Flume
Reprocessing Map / Reduce
Multi Cluster Management Hive and Pig Jobs
Falcon provides a single interface to orchestrate data lifecycle.
Sophisticated DLM easily added to Hadoop applications.
OPERATIONAL
SERVICES
Apache Falcon
Provides Orchestrates

17. © Hortonworks Inc. 2013
Knox: Make Hadoop Security Simple
Simplify Security Aggregate Access Client Agility
Simplify security for both users
and operators.
Deliver unified and centralized
access to the Hadoop cluster to
give a ‘single application’ feel
Ensure service users are
abstracted from where services
are located and how services
are configured & scaled
PLATFORM
SERVICES
Hadoop Cluster
Authentication
&
Verification
Client
User Store
KDC, AD, LDAP
{REST}!
Knox
gateway
cluster

18. © Hortonworks Inc. 2013
• OpenStack provides operational agility and deployment choice
• Hadoop is a net new workload and a perfect app for OpenStack
• Integration marries two of the Largest Open Source Movements
– Community-driven innovation outpaces any single vendor
– Both are attracting major ecosystem players: IBM, RHT, HP, RAX, etc…
Page 18
Project Savanna to enable Hadoop
on OpenStack
CLOUD
PLATFORM
ENABLEMENT
Project Savanna
Automate deployment of
Apache Hadoop on
OpenStack

19. © Hortonworks Inc. 2013
Topics
Page 19
Market Trends
& Emergence of
Hadoop
Hadoop’s role
and future
direction in the
Enterprise
Enterprise
Hadoop Use
Cases

20. © Hortonworks Inc. 2013
Fundamental business drivers the same…
• Better
– Automation
– Transparency
– Segmentation
– Innovation & experimentation
• Faster
– Everything
• Cheaper
– Across the value chain
Page 20

21. © Hortonworks Inc. 2013
6 Common TYPES OF DATA
1. Sentiment
Understand how your customers feel about your brand and
products – right now
2. Clickstream
Capture and analyze website visitors’ data trails and
optimize your website
3. Sensor/Machine
Discover patterns in data streaming automatically from
remote sensors and machines
4. Geographic
Analyze location-based data to manage operations where
they occur
5. Server Logs
Research logs to diagnose process failures and prevent
security breaches
6. Text
Understand patterns in text across millions of web pages,
emails, and documents
Value
Page 21

22. © Hortonworks Inc. 2013
Financial services
• Industry specific drivers for Hadoop
– Increasing compliance regulatory pressure
– Bad guys never stop
– Never ending Macroeconomic volatility
– Cost pressures – more than ever
– Extreme competition
• Common use cases
– Fraud & risk reduction( eg. During new account creation)
– Sentiment based trading strategies
– Improve insurance underwriting based on usage and longer
history
– Data reservoir ( for archival, compliance inquiries etc.)
Page 22

23. © Hortonworks Inc. 2013
Retail
• Industry specific drivers for Hadoop
– Increasingly ‘value sensitive’ and SMART consumer
– Constant margin pressure
– Emergence of the multi-channel approach
• Common use cases
– 360 degree customer view ( behavior, location, sentiment etc.)
– Micro and dynamic segmentation
– Optimizations – Price, assortment, layout, supply chain
– Seasonal predictions – product styles, labor needs etc.
Page 23

24. © Hortonworks Inc. 2013
Telcos
• Industry specific drivers for Hadoop
– Infrastructure under stress with rise of smart devices
– 4G/LTE investment needs CapEx but revenue growth largely flat
– Cloud computing changing the game
– Increasing competition with non Telcos
– Sitting on a gold mine of data
• Common use cases
– Understanding customer behavior AND context (eg. location
based) in real-time
– Packaging and selling data
– Call Detail Record (CDR) & extended data record (XDR) analysis
for service quality improvement & capacity planning/optimization
– Customer churn analysis & prevention
Page 24

25. © Hortonworks Inc. 2013
Healthcare & Pharma
• Industry specific drivers for Hadoop
– Sudden data deluge
– Health data initiative (HDI) by US Govt.
– Digitization of health records and rise of sensor data
– Huge accumulation of R&D data
– Rising healthcare costs
– Payors shift to outcome based payment models with providers as
well as pharmaceutical companies
• Common use cases
– Improved patient outcome tracking
– Optimized patient recruitment for drug trials
– Reduce drug modeling time
– Improve insurance claim validation accuracy
Page 25

26. © Hortonworks Inc. 2013
Manufacturing
• Industry specific drivers for Hadoop
– Proliferation of sensors
– Globalization of the supply chain
– Ongoing miniaturization of products
• Common use cases
– Failure analysis to perform proactive maintenance
– Improving equipment quality by more frequent sample testing and
rigorous prototype testing
– Supply chain optimization
Page 26

27. © Hortonworks Inc. 2013
Hadoop Summit
• June 26-27, 2013- San Jose Convention Cntr
• Co-hosted by Hortonworks & Yahoo!
• Theme: Enabling the Next Generation
Enterprise Data Platform
• 90+ Sessions and 7 Tracks:
• Community Focused Event
– Sessions selected by a Conference Committee
– Community Choice allowed public to vote for
sessions they want to see
• Training classes offered pre event
– Apache Hadoop Essentials: A Technical
Understanding for Business Users
– Understanding Microsoft HDInsight and Apache
Hadoop
– Developing Solutions with Apache Hadoop –
HDFS and MapReduce
– Applying Data Science using Apache Hadoop
Page 27
hadoopsummit.org

28. © Hortonworks Inc. 2013
Thank You
Follow us: @hortonworks
Page 28
http://hortonworks.com/products/hortonworks-sandbox/

29. © Hortonworks Inc. 2012
Similar solution architecture across use
cases
Page
Hadoop
LOAD
SQOOP
FLUME
Web
HDFS
NFS
USE
DB
EDW
MPP
SOURCE
DATA
1
2
3
4
5
BATCH
STREAMING
STORM
Map
Reduce
PIG
INTERACTIVE
HIVE/SQL
ONLINE
HBASE
AMBARI
HCATALOG (table metadata)
PIG(data
processing)
HIVE
(data processing)
compute
&
storage
. . .
. . .
. .
compute
&
storage
.
.
YARN