What are the tools that we can find to day to manage Hadoop cluster and its ecosystem? There are two tools ready today: Cloudera Manager and Ambari from Hortonworks. In this presentation I explain what they do and why to use them, as well as Pros. and Cons.

1. by Ran Silberman
DevOps for Big Data
Cluster management tools
20.4.2015
Hosted by:
FullStack Developers Israel

2. Ran Silberman,
Big Data Architect
...and amateur birder

3. ● Explain Cluster Management
tools by example
● Demo Cloudera Management
● Pros and Cons
Agenda

4. Birds of Brazil Wiki application

6. ● Input photos and locations
● Batch: Display statistics on bird,
location & photographer.
● Real-time: Count how many birds
were seen in the last minute from
each species
Application
requirements

7. ● Volume growth
● Velocity of Streaming and Batch
● Same env from DEV to PROD
● Data from PROD to test on DEV
● Manage Deployment of many
applications on many nodes
Big Data lifecycle
considerations

8. ● HDFS for storing the data
● Hive for batch processing
● Solr/elasticsearch for search
● Spark for streaming
● ...Home-grown applications
Choosing the
Infrastructures

9. Many Infrastructures

10. How can we manage all those
infrastructures?

11. ● Hortonworks Ambari
or
● Cloudera Manager
Choosing the
Management
tool

12. ● All platforms & infrastructures are
installed by the tool
● Monitoring, Audits & logs are
built-in
● Easy installation and upgrade
● Save scripting work
What are the
news for DevOps
pipeline?

13. ● Manage cluster with GUI or API
● Hadoop installation and setup
● System monitoring & alerts
● Built-in systems: Zookeeper,
Spark, Hive Impala and more
● Ability to add parcels
CM features

14. ● Monolithic packages
● Relocatable
● sudo-less installs
● Rolling upgrade
Parcels

15. Custom Service Descriptors
● CSD is a descriptor for a service
used by CM
● Defines how to install start/stop
a service and the logic used by
CM
CSD

16. Demo

17. ● Archive data in Hadoop
● Growing data affects DWH
performance & capabilities
● Creating realistic testing data
● Dev and Prod env. may differ in
cluster size (dev may be 1 node)
More DevOps
considerations

18. Tools Comparison
CM Ambari
Licence Paid Ent edition Free Apache Open Source
Technology Cloudera puppet, ganglia, nagios
Dependency CDH HDP
Manage cluster Parcels Yum
REST API + +
Extra Features Rolling Upgrade, 3rd-
parties Mngt,
Extendable by REST API

19. CM features
Express Enterprise
Subscription Free Annual
Deployment &
Configuration
+ +
Management + +
Monitoring + +
Diagnostic + +
Extra Features Reports, Rollbacks, Rolling
Upgrade, AD Kerberos, Kerberos
wizard, Backup & DR

20. ● Fast Deploy
● Easy management by GUI
● Built in monitoring and alerts
● Simple upgrades
● Same management and deploy
in Dev and Prod
Pros. of Hadoop
Management
tools

21. ● Tied to specific vendor
proprietary system
● Tied to system version by
Parcels
● Less flexibility to low-level
management
Cons. of Hadoop
Management
tools

22. THANK YOU
Ran Silberman
Email: ran@tikalk.com