Part of the core Hadoop project, YARN is the architectural center of Hadoop that allows multiple data processing engines such as interactive SQL, real-time streaming, data science and batch processing to handle data stored in a single platform, unlocking an entirely new approach to analytics. It is the foundation of the new generation of Hadoop and is enabling organizations everywhere to realize a Modern Data Architecture.

1. YARN – The Data Operation System
Tom Benton
Page 1 © Hortonworks Inc. 2011 – 2014. All Rights Reserved

2. In the beginning…
Tradi5onal
Hadoop Traditional Hadoop allowed early adopters
MapReduce
Largely
Batch
Processing
1
°
°
°
°
°
HDFS
(Hadoop
Distributed
File
System)
°
°
°
°
°
N
2006
to deal with data at scale via:
• Single purpose clusters, specific data sets
• Primarily batch-oriented applications using MapReduce
However…
• No direct way to integrate interactive and real-time
Page 2 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
applications
• Limited enterprise capabilities:
Operations, Security & Governance

3. JAN
2008
MAPREDUCE-279
Outlines a NEW architecture for Hadoop which allows
for efficient use of resources across many types of apps
Tradi5onal
Hadoop
MapReduce
Largely
Batch
Processing
1
°
°
°
°
°
HDFS
(Hadoop
Distributed
File
System)
°
°
°
°
°
N
2006
2011
Hortonworks Founded
Work accelerates on Hadoop’s
next-gen architecture
Page 3 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
…with increased adoption and
breadth of use cases,
a new approach was needed

4. Traditional Hadoop, challenges & limitations
MapReduce
Largely Batch Processing
1 ° ° ° ° °
HDFS
(Hadoop Distributed File System)
° ° ° ° ° N
SOURCES
EXISTING
Systems
Clickstream
Web
&Social
Geoloca5on
Sensor
&
Machine
Page 4 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Server
Logs
Unstructured
Architectural Limitations
• Primarily a batch system using MapReduce
• Single purpose clusters, specific data sets
Enterprise Challenges
• Limited enterprise capabilities:
Operations, Security & Governance
• Created additional Silos
Interoperability Challenges
• Difficult to natively integrate existing applications
DATA SYSTEM APPLICATIONS
Business
Analytics
Custom
Applications
Packaged
Applications
RDBMS EDW MPP

5. Page 5 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
YARN Has
Fundamentally
Changed Hadoop
YARN enables:
• More Workloads
From batch to interactive & real-time
• More Data
Multiple data sets of varying types
and structures
• More Value
Hosting multiple business cases
in a single Hadoop cluster
Enterprise Hadoop Enables…
• More Workloads
From batch to interactive & real-time
• More Data
Multiple data sets of varying types
and structures
• More Value
Hosting multiple business cases
in a single Hadoop cluster

6. 2006 2008
MAPREDUCE-­‐279
Tradi5onal
Hadoop
MapReduce
Largely
Batch
Processing
1
°
°
°
°
°
HDFS
(Hadoop
Distributed
File
System)
°
°
°
°
°
N
2011
Page 6 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Enterprise Hadoop Era Begins
October 23, 2013
Core of Enterprise Hadoop
Hadoop 2 & YARN
Batch
Interactive
Real-Time
YARN : Data Operating System
1
°
°
°
°
°
°
°
°
°
°
°
°
°
°
°
°
°
°
°
N°
HDFS
(Hadoop Distributed File System)
Architected &
led development
of YARN to enable
the Modern Data
Architecture

7. Benefits Enabled by MDA and YARN
SOLUTION: A single set of data across the entire cluster with multiple
access methods using “zones” for processing
1
°
°
°
°
°
°
°
°
°
°
°
°
°
°
°
°
°
°
°
°
°
Page 7 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
°
n
Interactive
Hive
Storm
Real
Time
Streams
Single Cluster,
Multiple Workloads
• Maximize compute
resources to lower TCO
• No standalone,
siloed clusters
• Simple management
& operations
…all enabled by YARN
Batch
Pig
Real Time
HBase
Spark
In
Memory

8. YARN and HDP Enables the Modern Data Architecture
HDP
Hortonworks Data Platform
BATCH, INTERACTIVE & REAL-TIME SECURITY
DATA ACCESS
In-Memory
Spark
Page 8 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Provision,
Manage &
Monitor
Ambari
Zookeeper
Scheduling
Oozie
Data Workflow,
Lifecycle &
Governance
Falcon
Sqoop
Flume
NFS
WebHDFS
YARN: Data Operating System
DATA MANAGEMENT
GOVERNANCE
& INTEGRATION
Authentication
Authorization
Accounting
Data Protection
Storage: HDFS
Resources: YARN
Access: Hive, …
Pipeline: Falcon
Cluster: Knox
OPERATIONS
Script
Pig
Search
Solr
SQL
Hive
HCatalog
NoSQL
HBase
Accumulo
Stream
Storm
Other
ISVs
1 ° ° ° ° ° ° ° ° °
° ° ° ° ° ° ° ° ° °
° ° ° ° ° ° ° ° ° °
°
°
N
HDFS
(Hadoop Distributed File System)
YARN is the architectural center of
Hadoop and HDP
• YARN enables a common data set
across all applications
• Batch, interactive & real-time
workloads
• Support multi-tenant access &
processing
HDP enables Apache Hadoop to
become Enterprise Viable Data
Platform with centralized services
• Security
• Governance
• Operations
• Productization
Enabled broad ecosystem
adoption
TezTez
Hortonworks drove this innovation of Hadoop through YARN

9. Page 9 © Hortonworks Inc. 2011 – 2014. All Rights Reserved

10. Batch Interactive Real-Time
YARN: Data Operating System
1 ° ° ° ° ° ° ° ° °
1 ° ° °
HDFS
(Hadoop Distributed File System)
°
° ° ° ° ° ° ° ° N
° ° ° Thank You! Questions?
Page 10 © Hortonworks Inc. 2011 – 2014. All Rights Reserved