Hadoop operations started on-prem primarily driven by Apache Ambari. However, due to the agility and flexibility of the cloud, it has driven many Hadoop cluster operations to the cloud and to hybrid environments. Cloud is enabling many ephemeral on-demand use cases which is a game-changing opportunity for analytic workloads. But all of this comes with the challenges of running enterprise workloads in the cloud securely and with ease. Apache Ambari is used by thousands of Hadoop Operators to manage the deployment, lifecycle, and automation of DevOps for Hadoop ecosystem projects. Starting out, Apache Ambari installed a handful of Apache Hadoop ecosystem projects, on a few operating systems, and helped with the most basic Hadoop operational tasks. Today, the product manages over 20 different services, runs on multiple major operating systems and versions, and automates many of the most challenging Hadoop operational tasks in the most secure customer environments. In this session, we will also take you through Cloudbreak as a solution to simplify provisioning and managing enterprise workloads while providing an open and common experience for deploying workloads across clouds. We will discuss the challenges (and opportunities) to run enterprise workloads in the cloud and will go through a live demo of how the latest from Cloudbreak enables enterprises to easily and securely run Apache Hadoop. This includes deep-dive discussion on Ambari Blueprints, recipes, custom images, and enabling Kerberos -- which are all key capabilities for Enterprise deployments. As part of this talk, will walk you through what we've learned, the challenges we've overcome, and how the Apache Ambari and Cloudbreak community has changed the product to handle them. The future is fast approaching, and with it comes new on-premise and cloud deployment architectures. See how Apache Ambari and Cloudbreak are being re-imagined to handle these new challenges. Speaker: Santosh Gowda, Principal Solutions Engineer, Hortonworks

1. 1 © Hortonworks Inc. 2011–2018. All rights reserved
Hadoop Operations –
Past, Present, and Future
Santhosh B Gowda
Feb 2019

2. 2 © Hortonworks Inc. 2011–2018. All rights reserved
Disclaimer
This document may contain product features and technology directions that are under
development, may be under development in the future or may ultimately not be developed.
Project capabilities are based on information that is publicly available within the Apache Software
Foundation project websites ("Apache"). Progress of the project capabilities can be tracked from
inception to release through Apache, however, technical feasibility, market demand, user
feedback and the overarching Apache Software Foundation community development process can
all effect timing and final delivery.
This document’s description of these features and technology directions does not represent a
contractual commitment, promise or obligation from Hortonworks to deliver these features in
any generally available product.
Product features and technology directions are subject to change, and must not be included in
contracts, purchase orders, or sales agreements of any kind.
Since this document contains an outline of general product development plans, customers should
not rely upon it when making purchasing decisions.

3. 3 © Hortonworks Inc. 2011–2018. All rights reserved
Agenda
• Hadoop Operations: Ambari
• Hadoop Operations: Data Challenge
• Cloud Key Considerations
• Cloudbreak
• What is Cloudbreak ?
• Custom Images
• Kerberos Security
• Recipes
• Auto Scaling

4. 4 © Hortonworks Inc. 2011–2018. All rights reserved
What Is Apache Ambari?
A completely open source
management platform for
provisioning, managing,
monitoring and securing
Apache Hadoop clusters.
Apache Ambari takes the
guesswork out of operating
Hadoop.

5. 5 © Hortonworks Inc. 2011–2018. All rights reserved
Hadoop Operations - Ambari
Simplified Installation,
Configuration and Management
Centralized Security Setup
Full Visibility into Cluster Health
Highly Extensible and
Customizable
• Wizard-driven and automated cluster provisioning
• Smart Configurations and Cluster Recommendations
• Automated Rolling and Express cluster upgrades
• Reduce complexity to administer security across the
platform
• Automate setup Kerberos
• Simplify the configuration of Apache Ranger
• Predefined alerts based on operational best practices
• Advanced metrics visualization with Grafana
• Integrated with SmartSense for proactive issues prevention
• Seamlessly fit into your enterprise environment
• Bring custom Services under management via Ambari
Stacks
• Customize the UI with Ambari Views

6. 6 © Hortonworks Inc. 2011–2018. All rights reserved
Early Adopters
Ambari
HDFS
Atlas, Ranger,
Metastore, Knox
Hive Spark
YARN
10101
10101010101
010101010101010
Public Cloud Storage
Public Cloud
Compute
Large Shared Workloads, supported by Shared
Services, On-Premise
Ambari
HDFS
Atlas, Ranger,
Metastore, Knox
Hive Spark
YARN
10101
10101010101
01010101010101
01010101010101010
10
Long-Running Cluster on Cloud IaaS
10101
10101010101
01010101010101
0101010101010101010

7. 7 © Hortonworks Inc. 2011–2018. All rights reserved
Hadoop Operations: Data Challenge
• Data is becoming more and more distributed…
• Across data center and cloud environments…
• Accessed using multi- and single-workload clusters…
• But must be discoverable and accessed by all who seek it.
Cluster
Cluster
Cluster
Cluster
ClusterCluster ClusterClusterClusterClusterCluster ClusterClusterCluster
DATA
CENTER CLOUDS
The Virtual Data Lake
Business User
Very difficult to find data
(leading to inefficient use of time)
Platform Operator
Hard to secure and hard to
operate (can be time consuming
and prone to error)

8. 8 © Hortonworks Inc. 2011–2018. All rights reserved
Cloud: Key Considerations
• Cloud is infrastructure… need a Data Strategy
• Hybrid (on-premise & cloud) requirements are real.
• Multi-Cloud (i.e. portability) is a key emerging requirement
• Logistics & Physics
• Regulatory & Compliance
• Economic arbitrage
• Consistent and familiar Security & Governance across on-premise & cloud environments
• Free movement of data, regardless of origin or destination
• Global data catalog, regardless of location

9. 9 © Hortonworks Inc. 2011–2018. All rights reserved
Data Management across On-Prem & Multi-Cloud
Large Shared Workloads, supported by
Shared Services, On-Premise
Ambari
HDFS
Atlas, Ranger,
Metastore, Knox
Hive Spark
YARN
10101
10101010101
01010101010101
010101010101010
Multiple Ephemeral Workloads,
supported by Shared Services, Multi-
Cloud.
Hortonworks DataPlane Service
Public Cloud A
Storage
Public Cloud A
Compute
Atlas, Ranger, Metastore, Knox
Hive LLAP
Ambari Ambari Ambari
NiFi
Spark
Cloudbreak
YARN YARN
Public Cloud B
Storage
Public Cloud B
Compute
Atlas, Ranger, Metastore, Knox
Hive LLAP
Ambari Ambari Ambari
NiFi
Spark
Cloudbreak
YARN YARN
Multiple Ephemeral Workloads,
supported by Shared Services, Multi-
Cloud.
10101
10101010101
01010101010101
010101010101010
1010
10101
10101010101
01010101010101
010101010101010
1010

10. 10 © Hortonworks Inc. 2011–2018. All rights reserved
Hortonworks: Architecting and Optimizing for the Cloud
CLOUD STORAGE WORKLOADS
Durable Ephemeral
When data resides in cloud object
stores (e.g. Amazon S3), Hadoop
optimizes reads/writes and acts as
an intermediate cache to increase
performance and decrease latency.
Metastore
SCHEMA
Long Running
Security access to workload
clusters via a Protected Gateway
enabled for AuthN and HTTPS.
Define your data schema, security
policies, and metadata catalog
once for your ephemeral and
always-on workloads.
Atlas
CATALOG
Ranger
POLICY
SHARED DATA LAKE SERVICES

11. 11 © Hortonworks Inc. 2011–2018. All rights reserved
Cloudbreak

12. 12 © Hortonworks Inc. 2011–2018. All rights reserved
What Is Cloudbreak ?
Cloudbreak is a tool for provisioning Hadoop
clusters on any cloud infrastructure
Simplified cluster provisioning - prescriptive
setup, simple automation

13. 13 © Hortonworks Inc. 2011–2018. All rights reserved
Cloudbreak: Harness the Agility of Cloud with Ease
Cloudbreak
• Declarative workload
provisioning across
multiple cloud providers
• Flexible topologies and
security configuration
options
• DevOps friendly, easy setup
and simple to automate
• Built-in elasticity and auto-
scaling
• Prescriptive integration
with cloud services

14. 14 © Hortonworks Inc. 2011–2018. All rights reserved
Cloudbreak Building Blocks
• Cloud Credentials
• Ambari Blueprints
• Auto Scaling
• Custom Recipes
• Custom Images
• Network
• Gateway
• Kerberos Security
• Dynamic Blueprints
• Cloud Storage
Simple and Flexible Prescriptive Secure

15. 15 © Hortonworks Inc. 2011–2018. All rights reserved
Custom Images

16. 16 © Hortonworks Inc. 2011–2018. All rights reserved
Background: Cloudbreak
1. Cloudbreak creates VM instances using a default base image.
2. Cloudbreak installs Ambari on a VM instance.
3. Cloudbreak instructs Ambari to install a cluster on the remaining VM instances.
Cloudbreak
Node
VM
Node
VM
Node
VM
Node
VM
Node
VM
Node
VM
Cluster

17. 17 © Hortonworks Inc. 2011–2018. All rights reserved
Custom Images Overview
Create the
Custom Image
Register the
Custom Image
Use the
Custom Image
when Creating
a Cluster
1 2 3

18. 18 © Hortonworks Inc. 2011–2018. All rights reserved
Recipes

19. 19 © Hortonworks Inc. 2011–2018. All rights reserved
Background: Recipes
• Cloudbreak lets you provision cluster using Ambari Blueprint however not all use-cases
can be addressed.
• Install additional software.
• System config changes.
• A recipe is a script that runs on all nodes of a selected node group at a specific time.
• Support for bash and python scripts.
• Available hooks
• Pre-ambari-start
• Post-ambari-start
• Post-cluster-install
• Pre-termination

20. 20 © Hortonworks Inc. 2011–2018. All rights reserved
Cloudbreak: Add Recipes
• Cluster Extensions > Recipes > Create
• Add recipe as File, Url or Text

21. 21 © Hortonworks Inc. 2011–2018. All rights reserved
Cloudbreak: Add Recipes
• Clusters > Create Cluster >
Cluster Extensions

22. 22 © Hortonworks Inc. 2011–2018. All rights reserved
Kerberos Security

23. 23 © Hortonworks Inc. 2011–2018. All rights reserved
Background: Kerberos
• Strongly authenticating and establishing a user’s identity is the basis for secure access in
Hadoop. Users need to be able to reliably “identify” themselves and then have that
identity propagated throughout the Hadoop cluster.
• Once this is done, those users can access resources (such as files or directories) or
interact with the cluster (like running MapReduce jobs).
• Besides users, Hadoop cluster resources themselves (such as Hosts and Services) need
to authenticate with each other to avoid potential malicious systems or daemon’s
“posing as” trusted components of the cluster to gain access to data.

24. 25 © Hortonworks Inc. 2011–2018. All rights reserved
Cloudbreak: Enable Kerberos Security
• Create Cluster > Security > Advanced
• [ ] Enable Kerberos Security

25. 26 © Hortonworks Inc. 2011–2018. All rights reserved
Options: Use Existing KDC or Use Test KDC
Use Existing
KDC
Use Test KDC
Advanced
Basic
- Not for production use. For testing and
evaluation purposes only.
- Installs and configures an MIT KDC on the
master node.
- Configures the cluster to leverage that
KDC.
- Provide basic information
about your existing KDC.
- Ambari Kerberos descriptors
are generated automatically.
- Provide basic information
about your existing KDC.
- Provide your own Ambari
Kerberos descriptors.

26. 27 © Hortonworks Inc. 2011–2018. All rights reserved
Auto Scaling

27. 28 © Hortonworks Inc. 2011–2018. All rights reserved
Auto-Scaling
• Alerts: Create metric or time-based alerts for cluster scaling
• Policies: Scaling policies adjust cluster size based on activity and workload alerts
• General Configurations: Boundaries and cooldown period

28. 29 © Hortonworks Inc. 2011–2018. All rights reserved
Auto-Scaling Time-Based Alert
Fire at 10:15 am everyday

29. 30 © Hortonworks Inc. 2011–2018. All rights reserved
Auto-Scaling Metric-Based Alert
Fire after NodeManagers are
CRITICAL for 10 minutes

30. 31 © Hortonworks Inc. 2011–2018. All rights reserved
Auto-Scaling Policies
• Define the Scale Adjustment (Node Count, Percentage, Exact)
• Select the Host Group (to Scale)
• Select Alert (which when fired, executes the Policy)

31. 37 © Hortonworks Inc. 2011–2018. All rights reserved
Learn More
• Try Ambari
• https://docs.hortonworks.co
m/HDPDocuments/Ambari/A
mbari-2.7.0.0/index.html
• Try Cloudbreak 2.8 (TP)
• https://docs.hortonworks.co
m/HDPDocuments/Cloudbre
ak/Cloudbreak-
2.8.0/index.html

32. 38 © Hortonworks Inc. 2011–2018. All rights reserved
Questions ?

33. 39 © Hortonworks Inc. 2011–2018. All rights reserved
Thank you !