Introduction This workshop is a hands-on session to quickly deploy Hadoop and Streaming on AWS / Azure / Google Cloud. Cloudbreak simplifies the deployment of Hadoop in cloud environments. It enables the enterprise to quickly run big data workloads in the cloud while optimizing the use of cloud resources. Format A short introductory lecture about Cloudbreak. This is followed by a walk through and lab leveraging Hadoop and Streaming in the Cloud with Cloudbreak. Objective To provide a quick and short hands-on introduction to Hadoop on the cloud. Review key benefits of cluster deployment automation. This lab will use Cloudbreak to quickly and effortlessly stand up Hadoop and Streaming clusters in a cloud provider of your choice. The lab shows the use of Ambari blueprints that are your declarative definitions of your Hadoop or Streaming clusters. Steps to dynamically change these blueprints and use external databases and external authentication sources and in essence showing a way to provide Shared Authentication, Authorization and Audit across ephemeral and long-lasting clusters. However it is not limited to only custom blueprints, the lab also shows how Cloudbreak provides easy to use custom scripts called recipes that can be executed before or after Ambari start or after cluster installation.

1. 1 © Hortonworks Inc. 2011–2018. All rights reserved
Data in the Cloud
DataWorks Summit - Singapore
Oct 2018

2. 2 © Hortonworks Inc. 2011–2018. All rights reserved
Who Am I ?
Santhosh B Gowda
Engineer @ Hortonworks

3. 3 © Hortonworks Inc. 2011–2018. All rights reserved
Agenda
• Introduction
• Primary Use Cases
• Cloudbreak Architecture
• Cloudbreak Core Concepts
• Credentials
• Blueprints
• Recipes
• Images
• CLI
• Streaming in Cloudbreak
• Shared Services Data Lake
• Hands-On Lab

4. 4 © Hortonworks Inc. 2011–2018. All rights reserved
No Upfront
HW Costs
$0
Unlimited
Elastic Scale
Ephemeral &
Long-Running
IT &
Business Agility
Why Data on Cloud ?

5. 5 © Hortonworks Inc. 2011–2018. All rights reserved
What Is Cloudbreak ?
Cloudbreak is a tool for provisioning Hadoop
clusters on any cloud infrastructure
Simplified Cluster Provisioning - prescriptive
setup, simple automation

6. 6 © Hortonworks Inc. 2011–2018. All rights reserved
Cloudbreak: Harness the Agility of Cloud with Ease
Cloudbreak
• Declarative workload
provisioning across
multiple cloud providers
• Flexible topologies and
security configuration
options
• DevOps friendly, easy setup
and simple to automate
• Built-in elasticity and auto-
scaling
• Prescriptive integration
with cloud services

7. 7 © Hortonworks Inc. 2011–2018. All rights reserved
Dev / Test
(all HDP services)
BI / Analytics
(Hive)
Data Science
(Spark)
IoT Apps
(Storm, HBase, Hive)
Cloudbreak
Cloudbreak
1. Pick a Blueprint
2. Choose a Cloud
3. Launch HDP/HDF!
Example Ambari Blueprints:
IoT Apps, BI / Analytics, Data Science,
Dev / Test

8. 8 © Hortonworks Inc. 2011–2018. All rights reserved
Create Cluster
Manage
Credentials
Manage
Templates
Manage
Network
Manage
Security
Groups
Manage
Blueprints
Advanced
Configurati
on
Configure Credentials for
Cloudbreak for connecting
to a Cloud Provider
Configure size of VM
images
Manage who can access
HDP clusters
Define network in which
HDP cluster will run Define HDP Services
Configure HDP Repository
Define Failure Actions
Cloud Provider Specific Configurations
HDP Specific Configurations

9. 10 © Hortonworks Inc. 2011–2018. All rights reserved
Cloudbreak UI

10. 11 © Hortonworks Inc. 2011–2018. All rights reserved
Manage Cluster

11. 12 © Hortonworks Inc. 2011–2018. All rights reserved
Cloudbreak CLI
DevOps

12. 13 © Hortonworks Inc. 2011–2018. All rights reserved
Cloudbreak CLI

13. 14 © Hortonworks Inc. 2011–2018. All rights reserved
“SHOW CLI COMMAND”

14. 15 © Hortonworks Inc. 2011–2018. All rights reserved
Recipes
DevOps

15. 16 © Hortonworks Inc. 2011–2018. All rights reserved
Background: Recipes
• Cloudbreak lets you provision cluster using Ambari Blueprint however not all use-cases
can be addressed.
• Install additional software.
• System config changes.
• A recipe is a script that runs on all nodes of a selected node group at a specific time.
• Support for bash and python scripts.
• Available hooks
• Pre-ambari-start
• Post-ambari-start
• Post-cluster-install
• Pre-termination

16. 17 © Hortonworks Inc. 2011–2018. All rights reserved
Cloudbreak: Add Recipes
• Cluster Extensions > Recipes >
Create
• Add recipe as File, Url or Text

17. 18 © Hortonworks Inc. 2011–2018. All rights reserved
Cloudbreak: Add Recipes
• Clusters > Create Cluster >
Cluster Extensions

18. 19 © Hortonworks Inc. 2011–2018. All rights reserved
Streaming in the Cloud

19. 20 © Hortonworks Inc. 2011–2018. All rights reserved

20. 21 © Hortonworks Inc. 2011–2018. All rights reserved
HDF in CloudBreak 2.7
Apache NiFi and Apache NiFi Registry

21. 22 © Hortonworks Inc. 2011–2018. All rights reserved
HDF in CloudBreak 2.7
Apache Kafka

22. 23 © Hortonworks Inc. 2011–2018. All rights reserved
Auto Scaling

23. 24 © Hortonworks Inc. 2011–2018. All rights reserved
Auto-Scaling
• Alerts: Create metric or time-based alerts for cluster scaling
• Policies: Scaling policies adjust cluster size based on activity and workload alerts
• General Configurations: Boundaries and cooldown period

24. 25 © Hortonworks Inc. 2011–2018. All rights reserved
Auto-Scaling Time-Based Alert
Fire at 10:15 am everyday

25. 26 © Hortonworks Inc. 2011–2018. All rights reserved
Auto-Scaling Metric-Based Alert
Fire after NodeManagers are
CRITICAL for 10 minutes

26. 27 © Hortonworks Inc. 2011–2018. All rights reserved
Auto-Scaling Policies
• Define the Scale Adjustment (Node Count, Percentage, Exact)
• Select the Host Group (to Scale)
• Select Alert (which when fired, executes the Policy)

27. 28 © Hortonworks Inc. 2011–2018. All rights reserved
Data Lake Shared
Services

28. 29 © Hortonworks Inc. 2011–2018. All rights reserved
Why Data Lake Shared Services
• Customers have a need to secure ephemeral workload clusters
• Customers need a single metadata repository for Hive schema
• Customers want a single pane of glass to define users, groups and authorization policies

29. 30 © Hortonworks Inc. 2011–2018. All rights reserved
Introducing “Data Lake” in Cloudbreak
CLOUD STORAGE WORKLOAD
S
Durable Ephemeral
When data resides in cloud object
stores (e.g. S3, ADLS, WASB, GCS),
Hadoop optimizes reads/writes
and acts as an intermediate cache
to increase performance and
decrease latency.
Metastore
SCHEM
A
Long Running
Security access to workload
clusters via a Protected Gateway
enabled for AuthN and HTTPS.
Define your data schema,
security policies, and metadata
catalog once for your ephemeral
and always-on workloads.
Atlas
CATALOG
Ranger
POLIC
Y
SHARED DATA LAKE SERVICES

30. 31 © Hortonworks Inc. 2011–2018. All rights reserved
Data Lake: Flyover
LDAP/AD
Hive
Database
Ranger
Database
Cloud
Storage
Data Lake Workload
Cluster(s)
Ranger
Hive
Metastore
Hive, Spark,
Zeppelin
Attach

31. 32 © Hortonworks Inc. 2011–2018. All rights reserved
Learn More
• Try Cloudbreak 2.8 (TP)
• http://docs.hortonworks.com

32. 33 © Hortonworks Inc. 2011–2018. All rights reserved
Thank You

33. 35 © Hortonworks Inc. 2011–2018. All rights reserved
Launch HDP from
Cloudbreak UI
(Demo#1)

34. 36 © Hortonworks Inc. 2011–2018. All rights reserved
Launch HDF from
Cloudbreak UI
(Demo#2)

35. 37 © Hortonworks Inc. 2011–2018. All rights reserved
Launch HDP from CLI
(Demo#3)