Presentation at Data Innovation Summit 2016 in Stockholm How to build a modern data architecture supporting data in motion and data at rest with Hortonworks Data Flow and Data Platform.

1. Data in Motion
Data at Rest
Mats Johansson
Solutions Engineer - EMEA
2016-03-22

2. 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
About Hortonworks
Customer Momentum
 +800 customers
 Public company - NASDAQ: HDP
The Leader in Connected Data Platforms
 Hortonworks DataFlow for data in motion
 Hortonworks Data Platform for data at rest
Partner for Customer Success
 Leader in open-source community, focused
on innovation to meet enterprise needs
 Unrivaled support & professional services
Founded in 2011
Original 24 Architects, Developers,
Operators of Hadoop from Yahoo!
850+
E M P L OY E E S
1600+
E C O S Y S T E M
PA R T N E R S

3. 3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
EMBRACE AN OPEN
APPROACH
MASTER THE VALUE
OF DATA
EVERY BUSINESS
IS A DATA BUSINESS

4. 4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Blind Spots Block Your Ability to Use All the Data
GROUP 3
GROUP 2 GROUP 4
GROUP 1
INTERNET
OF
ANYTHING
Fragmented Data-at-rest
increases the cost of insight
Data-in-motion streams
through your blind spots
Various Data types and
massive Volumes

5. 5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
The Future of Data
Actionable Intelligence
D A T A I N M O T I O N
STORAGE
STORAGE
GROUP 2GROUP 1
GROUP 4GROUP 3
D ATA AT R E S T
INTERNET
OF
ANYTHING

6. 6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Hortonworks® customers leverage our Connected Data Platforms to transform their
industries – renovating their IT architectures and innovating with their Data in Motion
or Data at Rest to power actionable intelligence through Modern Data Applications.
Social
Mapping
Payment
Tracking
Factory
Yields
Defect
Detection
Call Analysis
Machine
Data
Product
Design
M & A
Due
Diligence
Next Product
Recs
Cyber
Security
Risk
Modeling
Ad
Placement
Proactive
Repair
Disaster
Mitigation
Investment
Planning
Inventory
Predictions
Customer
Support
Sentiment
Analysis
Supply Chain
Ad
Placement
Basket
Analysis
Segments
Cross-
Sell
Customer
Retention
Vendor
Scorecards
Optimize
Inventories
OPEX
Reduction
Mainframe
Offloads
Historical
Records
Data
as a Service
Public
Data
Capture
Fraud
Prevention
Device Data
Ingest
Rapid
Reporting
Digital
Protection
6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

7. 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
DATA AT RESTDATA IN MOTION
ACTIONABLE
INTELLIGENCE
Modern Data Applications
PERISHABLE
INSIGHTS
HISTORICAL
INSIGHTS
INTERNET
OF
ANYTHING
Hortonworks
DataFlow
Hortonworks
Data Platform
Hortonworks
Connected Data Platforms

8. 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Store Data
Process and Analyze
Data
Acquire Data
Data in Motion : Easy, Simple, Definitive
Dataflow

9. 9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Reality of Data in Motion: Complex, Chaotic, Messy
Store Data
Process and Analyze
Data
Acquire Data
Store DataStore Data
Store Data
Store Data
Acquire Data
Acquire Data
Acquire Data
Dataflow

10. 10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Enterprise DataFlow Challenges
• Variable Protocols, Formats , Schemas
• Data Size and Speed
• Security at Data Plane
• Traceability (Data Lineage)
• Prioritization of Resources
• Multi-Directional Flow
• Recoverability and Replay
• Transparency of DataFlow
• Scaling Down
• Enrichment/Transformation
• Unreliable Comms
GATHER
DELIVER
PRIORITIZE
Track from the edge Through the datacenter

11. 11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
HDF makes Data in Motion Easy
Complicated, messy, and takes weeks to
months to move the right data into Hadoop
HDP
HORTONWORKS
DATA PLATFORM
Streamlined, Efficient, Easy

12. 12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Secure
Real-time
Adaptive
Integrated
Hortonworks DataFlow for Data in Motion
Powered by Apache NiFi

13. 13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
SOURCES REGIONAL
INFRASTRUCTURE
CORE
INFRASTRUCTURE
HDF Manages Dataflow
Constrained
High-Latency
Localized Context
Hybrid – Cloud/On-Premise
Low-Latency
Global Context

14. 14 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Real-Time, Visual Control of Data Flows
Add and Adjust Data Sources
to maximize the opportunity that you
capture from perishable insights
Visually Trace the Data Path
to manage the what, who, where and
how around data in motion
Dynamically Adjust the Pipeline
to match the dataflow with your bandwidth
H O R TO N W O R K S
DATA F LO W Add and adjust
data sources
Visually trace
the data path
Dynamically
adjust the
pipeline

15. 15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ecosystem: 130+ Processors
HTTP
Syslog
Email
HTML
Image
Hash Encrypt
Extract
TailMerge
Evaluate
Duplicate Execute
Scan
GeoEnrich
Replace
ConvertSplit
Translate
HL7
FTP
UDP
XML
SFTP
Route Content
Route Context
Route Text
Control Rate
Distribute Load
AMQP

16. 16 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Graphical

17. 17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Integrated Processes and Control
 Optimize Your Architecture
• Reduce cost and complexity with the
most efficient data collection
technologies
 Assure Efficient Operations
• Via real-time control of data inputs,
outputs, transportation and
transformations
 Rely on a Common Foundation
• Eliminating dependence on multiple
customized systems
C O M M O N A R C H I T E C T U R E
W I T H O U T H O R T O N W O R K S D A T A F L O W
W I T H H O R T O N W O R K S D A T A F L O W
Ingest
Scripts
Messaging
Scripts
HORTONWORKS DATAFLOW

18. 18 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Secure Flows with Chain of Custody and Provenance
H O R TO N W O R K S
DATA F LO W
End-to-End Security
Apply security rules to encrypt, decrypt,
filter and replace data from the point of
collection at the jagged edge to its final
destination
Granular Control and Sharing
Move beyond role-based access and
dynamically share an entire dataflow
Real-Time Traceability
Rich metadata and contextual detail
helps troubleshoot security issues and
informs timely decisions

19. 19 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Adapt to a Broad Range of Data Flow Demands
Automated
Bi-directional communication between source and destination
adapts data flows automatically, according to current priorities
On-Demand
Operational control to adapt to changing conditions and requirements
Scalable
By incorporating data from any device—small machine sensors to
enterprise data centers—HDF connects you to the broadest set of
disparate data sources

20. 20 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Hortonworks Data Platform for Data at Rest
Powered by Open Enterprise Hadoop
Open
Interoperable
Ready
Central

21. 21 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
YARN : Data Operating System
DATA ACCESS SECURITY
GOVERNANCE &
INTEGRATION
OPERATIONS
1 ° ° ° ° ° ° ° ° °
° ° ° ° ° ° ° ° ° °
°
N
Administration
Authentication
Authorization
Auditing
Data Protection
Ranger
Knox
Atlas
HDFS Encryption
Data Workflow
Sqoop
Spark
Flume
Kafka
NFS
WebHDFS
Provisioning,
Managing, &
Monitoring
Ambari
Cloudbreak
Zookeeper
Scheduling
Oozie
Falcon
Batch
MapReduce
Script
Pig
Search
Solr
SQL
Hive
NoSQL
HBase
Accumulo
Phoenix
Stream
Storm
In-memory
Spark
Others
ISV Engines
TezTez Tez Slider Slider
HDFS Hadoop Distributed File System
DATA MANAGEMENT
Hortonworks Data Platform 2.4
Deployment ChoiceLinux Windows On-Premise Cloud
Data Lifecycle
& Governance
Falcon
Atlas
Source Systems
Clickstream
Social/Web
Geolocation
Machine/Sensor
Server Log
Unstructured
CRM/ERP
ODS
EDW
Target Systems
ODS
EDW
Visualization
& Reporting
Business
Applications
Data
Marts
Hortonworks Data Platform

22. 22 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
OPERATIONS SECURITY
GOVERNANCE
STORAGE
STORAGE
Machine
Learning
Batch
StreamingInteractive
Search
Ready for Consistent Operations
OPERATIONS
YA R N
D A T A O P E R A T I N G S Y S T E M
Shared Big Data Platform
across applications, business groups,
functions and users
Centralized
management and monitoring of Hadoop
clusters
Automated Provisioning
either on-premises or in the cloud with
the Cloudbreak API for clusters in minutes
Managed Services
for high availability and consistent lifecycle
controls, with dashboards and alerts

23. 23 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ready for Trusted Governance
OPERATIONS SECURITY
GOVERNANCE
STORAGE
STORAGE
Machine
Learning
Batch
StreamingInteractive
Search
GOVERNANCE
YA R N
D A T A O P E R A T I N G S Y S T E M
Data Management
along the entire data lifecycle with integrated
provenance and lineage capability
Modeling with Metadata
enables comprehensive data lineage through
a hybrid approach with enhanced tagging
and attribute capabilities
Interoperable Solutions
across the Hadoop ecosystem, through a
common metadata store

24. 24 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ready for Comprehensive Security
OPERATIONS SECURITY
GOVERNANCE
STORAGE
STORAGE
Machine
Learning
Batch
StreamingInteractive
Search
SECURITY
YA R N
D A T A O P E R A T I N G S Y S T E M
Comprehensive Security
through a platform approach
Fine-Grained, Flexible Authorization
controlling access based on roles
or data tags
Encrypt Data
at rest and in motion
Centralized Administration
of security policies and user authentication

25. 25 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Fast SQL with Apache Hive at Scale
H I V E O N YA R N
OPERATIONS SECURITY
GOVERNANCE
STORAGE
STORAGE
Pluggable Architecture
supports Apache Hive, Pivotal HAWQ and other
leading SQL engines
Familiar SQL Query Semantics
enable transactions and SQL:2011 Analytics
for rich reporting
Unprecedented Speed at Extreme Scale
returns query results in interactive time,
even as data sets grow to petabytes

26. 26 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Agile Analytics with Enterprise Spark 1.6 at Scale
S PA R K O N YA R N
OPERATIONS SECURITY
GOVERNANCE
STORAGE
STORAGE
Powering Agile Analytics
via Zeppelin data science notebooks and automation
for most common analytics (including Geospatial
analysis and entity resolution)
Seamless Data Access
that brings together as many data types as possible
Unmatched Economics
combining the speed of in-memory processing with
HDP’s cost efficiencies at scale
Ready for the Enterprise
with robust security, governance and operations
coordinated centrally by Apache Hadoop and YARN

27. 27 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

28. 28 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
We Power The Future of Data
1600+ Partners
4000+
members
15,000+
Unique Weekly visitors
A PA C H E H A D O O P
C O M M I T T E R S

29. Thank You
@matsjo66
mjohansson@hortonworks.com