Discover.hdp2.2.h base.final[2]

© Hortonworks Inc. 2014
Discover HDP 2.2:
Apache HBase with YARN & Slider for Fast NoSQL Access
Hortonworks. We do Hadoop.

Speakers
Justin Sears
Hortonworks Product Marketing Manager
Carter Shanklin
Hortonworks Director of Product Management & PM for
Apache HBase in Hortonworks Data Platform
Enis Soztutar
Hortonworks Engineer, Apache HBase Committer & PMC Member

Agenda
•  Introduction to Apache HBase
•  New HBase Innovation in HDP 2.2
–  HBase HA
–  Support for rolling upgrades
–  HBase on YARN using Apache Slider
•  Q & A
We’ll move quickly:
•  Attendee phone lines are muted
•  Text any questions to Enis Soztutar using Webex chat
•  Questions answered at the end
•  Unanswered questions and answers in upcoming blog post

Big Data, Hadoop & Data Center Re-platforming
Business Drivers
•  From reactive analytics
to proactive interactions
•  Insights that drive
competitive advantage
& optimal returns
Financial Drivers
•  Cost of data systems, as
% of IT spend,
continues to grow
•  Cost advantages of
commodity hardware
& open source software
$
Technical Drivers
•  Data is growing
exponentially & existing
systems overwhelmed
•  Predominantly driven by
NEW types of data that
can inform analytics
There is an inequitable balance between vendor and customer in the market

Clickstream
Capture and analyze
website visitors’ data
trails and optimize
your website
Sensors
Discover patterns in
data streaming
automatically from
remote sensors and
machines
Server Logs
Research logs to
diagnose process
failures and prevent
security breaches
New Types of DataHadoop Value:
Sentiment
Understand how
your customers feel
about your brand
and products –
right now
Geographic
Analyze location-
based data to
manage operations
where they occur
Unstructured
Understand patterns
in files across millions
of web pages, emails,
and documents

A Shift from Reactive to Proactive Interactions
HDP and Hadoop allow
organizations to use
data to shift interactions
from…
Reactive
Post Transaction
Proactive
Pre Decision
…to Real-time PersonalizationFrom static branding
…to repair before breakFrom break then fix
…to Designer MedicineFrom mass treatment
…to Automated AlgorithmsFrom Educated Investing
…to 1x1 TargetingFrom mass branding
A shift in Advertising
A shift in Financial Services
A shift in Healthcare
A shift in Retail
A shift in Telco

Enterprise Goals for the Modern Data Architecture
•  Consolidate siloed data sets structured
and unstructured
•  Central data set on a single cluster
•  Multiple workloads across batch
interactive and real time
•  Central services for security, governance
and operation
•  Preserve existing investment in current
tools and platforms
•  Single view of the customer, product,
supply chain
APPLICATIONSDATASYSTEM
Business
Analytics
Custom
Applications
Packaged
Applications
RDBMS
EDW
MPP
YARN: Data Operating System
1 ° ° ° ° ° ° ° ° °
° ° ° ° ° ° ° ° ° N
Interactive Real-TimeBatch
CRM
ERP
Other
1 ° ° °
° ° ° °
HDFS
(Hadoop Distributed File System)
SOURCES
EXISTING

Systems

Clickstream
Web

&Social

Geoloca9on
Sensor

&
Machine

Server

Logs

Unstructured

YARN Transformed Hadoop & Opened a New Era
YARN
The Architectural
Center of Hadoop
•  Common data platform, many applications
•  Support multi-tenant access & processing
•  Batch, interactive & real-time use cases
(Cluster Resource Management)
1 ° ° ° ° ° ° °
° ° ° ° ° ° ° °
Script
Pig
SQL
Hive
Tez
Tez
Java
Scala
Cascading
Tez
° °
° °
° ° ° ° °
° ° ° ° °
Others
ISV
Engines
HDFS
Stream
Storm
Search
Solr
NoSQL
HBase
Accumulo
Slider
Slider
BATCH, INTERACTIVE & REAL-TIME DATA ACCESS
In-Memory
Spark

YARN Extends Hadoop to Other Data Center Leaders
YARN
The Architectural
Center of Hadoop
•  Common data platform, many applications
•  Support multi-tenant access & processing
•  Batch, interactive & real-time use cases
•  Supports 3rd-party ISV tools
(ex. SAS, Syncsort, Actian, etc.)
YARN Ready Applications
Facilitates ongoing innovation and enterprise adoption via
ecosystem of new and existing “YARN Ready” solutions
1 ° ° ° ° ° ° °
° ° ° ° ° ° ° °
Script
Pig
SQL
Hive
Tez
Tez
Java
Scala
Cascading
Tez
° °
° °
° ° ° ° °
° ° ° ° °
Others
ISV
Engines
HDFS
Stream
Storm
Search
Solr
NoSQL
HBase
Accumulo
Slider
Slider
In-Memory
Spark

Enterprise Hadoop: Central Set of Services
1 ° ° ° ° ° ° °
° ° ° ° ° ° ° °
° °
° °
° ° ° ° °
° ° ° ° °
Enables Apache Hadoop to be
an Enterprise Data Platform
with centralized services for:
•  Governance
•  Operations
•  Security
Everything that plugs into
Hadoop inherits these services
Provision,
Manage &
Monitor
Ambari
Zookeeper
Scheduling
Oozie
Load data and
manage
according
to policy
Deploy and
effectively
manage the
platform
Provide layered
approach to
security through
Authentication,
Authorization,
Accounting, and
Data Protection
SECURITYGOVERNANCE OPERATIONS
Script
Pig
SQL
Hive
Java
Scala
Cascading
Stream
Storm
Search
Solr
NoSQL
HBase
Accumulo
In-Memory
Spark
Others
ISV
Engines
HDFS
Tez
Slider
Slider
Tez
Tez

Hortonworks Data Platform 2.2
HDP Delivers Enterprise Hadoop
1 ° ° ° ° ° ° °
° ° ° ° ° ° ° °
Script
Pig
SQL
Hive
Tez
Tez
Java
Scala
Cascading
Tez
° °
° °
° ° ° ° °
° ° ° ° °
HDFS
Stream
Storm
Search
Solr
NoSQL
HBase
Accumulo
Slider
Slider
SECURITYGOVERNANCE OPERATIONSBATCH, INTERACTIVE & REAL-TIME DATA ACCESS
In-Memory
Spark
Provision,
Manage &
Monitor
Ambari
Zookeeper
Scheduling
Oozie
Data Workflow,
Lifecycle &
Governance
Falcon
Sqoop
Flume
Kafka
NFS
WebHDFS
Authentication
Authorization
Audit
Data Protection
Storage: HDFS
Resources: YARN
Access: Hive
Pipeline: Falcon
Cluster: Ranger
Cluster: Knox
Deployment ChoiceLinux Windows Cloud
YARN is the architectural
center of HDP
•  Common data set across all
applications
•  Batch, interactive & real-time
workloads
•  Multi-tenant access & processing
Provides comprehensive
enterprise capabilities
•  Governance
•  Security
•  Operations
Enables broad
ecosystem adoption
•  ISVs can plug directly into Hadoop
The widest range of deployment options
•  Linux & Windows
•  On premises & cloud
Others
ISV
Engines
On-Premises

Hortonworks Data Platform 2.2
HDP Delivers Enterprise Hadoop
1 ° ° ° ° ° ° °
° ° ° ° ° ° ° °
Script
Pig
SQL
Hive
Tez
Tez
Java
Scala
Cascading
Tez
° °
° °
° ° ° ° °
° ° ° ° °
HDFS
Stream
Storm
Search
Solr
Slider
SECURITYGOVERNANCE OPERATIONSBATCH, INTERACTIVE & REAL-TIME DATA ACCESS
In-Memory
Spark
Provision,
Manage &
Monitor
Ambari
Zookeeper
Scheduling
Oozie
Data Workflow,
Lifecycle &
Governance
Falcon
Sqoop
Flume
Kafka
NFS
WebHDFS
Authentication
Authorization
Audit
Data Protection
Storage: HDFS
Resources: YARN
Access: Hive
Pipeline: Falcon
Cluster: Ranger
Cluster: Knox
YARN is the architectural
center of HDP
•  Common data set across all
applications
•  Batch, interactive & real-time
workloads
•  Multi-tenant access & processing
Provides comprehensive
enterprise capabilities
•  Governance
•  Security
•  Operations
Enables broad
ecosystem adoption
•  ISVs can plug directly into Hadoop
The widest range of deployment options
•  Linux & Windows
•  On premises & cloud
Others
ISV
Engines
Deployment ChoiceLinux Windows CloudOn-Premises
NoSQL
HBase
Accumulo
Slider

Introduction to Apache HBase

What Is Apache HBase?
Flexible
Schema

Extreme
Low
Latency

SQL
and
NoSQL
Interfaces

Store
and
Process
Petabytes
of
Data

Scale
out
on
Commodity
Servers

Integrated
with
YARN

100%
Open
Source

YARN
:
Data
Opera9ng
System

HBase

RegionServer

1
°
°
°
°
°
°
°
°
°
°

°
°
°
°
°
°
°
°
°
°
N

HDFS

(Permanent
Data
Storage)

HBase

RegionServer

HBase

RegionServer

Flexible Schema
Extreme Low Latency
Directly Integrated with Hadoop

New in HDP 2.2: HBase HA

Primary
Keys:

(Read
Write)

1-‐100

Standby
Keys:

(Read
Only)

101-‐200

201-‐300

Primary
Keys:

(Read
Write)

101-‐200

Standby
Keys:

(Read
Only)

201-‐300

301-‐400

Primary
Keys:

(Read
Write)

201-‐300

Standby
Keys:

(Read
Only)

301-‐400

1-‐100

Primary
Keys:

(Read
Write)

301-‐400

Standby
Keys:

(Read
Only)

1-‐100

101-‐200

HBase

RegionServer
1

HBase

RegionServer
2

HBase

RegionServer
3

HBase

RegionServer
4

HDFS

(3
Copies
of
All
Data,
Available
to
all
RegionServers)

1
2
3
1 HBase
Keys
are
range
parVVoned
across
servers,
node
failure
aﬀects
1
key
range,
rest
remain
available.

2 HBase
HA
stores
read-‐only
copies
in
separate
RegionServers.
Data
can
sVll
be
read
if
a
node
fails.

3 3
copies
of
all
data
stored
in
HDFS.
Data
from
failed
nodes
automaVcally
recovered
on
other
nodes.

HBase
HA:
3
Levels
of
Protec9on

Comparing HBase HA Phase 1 Versus 2
Item
HA
Phase
1
/
HDP
2.1
HA
Phase
2
/
HDP
2.2

Data
Staleness
>
30s
Near
Zero

HA
in
Scans
Unsupported
Supported

Region
Split/Merge
Disabled
Supported

META
Table
Highly
Available
Unsupported
Supported

HBCK
check
for
common
HA
problems
Unsupported
Supported

New in HDP 2.2: Rolling Upgrades

Rolling Upgrade Goals
Zero downtime upgrades
Roll forward and roll backward
Update clients and servers independently

HBase Rolling Upgrade: Component Overview
New
Package

Format

Install
mulVple
versions
of

Hadoop
so`ware
on
a
single

node
or
cluster.

hdp-‐select
U9lity

Choose
the
component

version
you
want,
roll

forward
or
backward.

Decoupled
Clients

and
Servers

Upgrade
servers

independently
of
clients.

HBase Rolling Upgrade: Directory Layout
Directory
Layout:
/usr/hdp

[root@cluster1
current]#
pwd

/usr/hdp/current

[root@cluster1
current]#
ls
-‐l
|
grep
hbase

lrwxrwxrwx.
1
root
root
27
Dec

6
22:57
hbase-‐client
-‐>
/usr/hdp/2.2.0.0-‐1995/hbase

lrwxrwxrwx.
1
root
root
27
Dec

6
22:57
hbase-‐master
-‐>
/usr/hdp/2.2.0.0-‐1995/hbase

lrwxrwxrwx.
1
root
root
27
Dec

6
22:57
hbase-‐regionserver
-‐>
/usr/hdp/2.2.0.0-‐1995/hbase

[root@cluster1
hdp]#
pwd

/usr/hdp

[root@cluster1
hdp]#
ls
-‐l

drwxr-‐xr-‐x.
19
root
root
4096
Nov
15
07:26
2.2.0.0-‐1995

drwxr-‐xr-‐x.

2
root
root
4096
Dec

7
01:22
2.2.0.1-‐2217

drwxr-‐xr-‐x.

2
root
root
4096
Dec

6
22:57
current

Multiple versions of
the HDP stack.
Within
/usr/hdp/current

HBase Rolling Upgrade: Upgrade One Component
hdp-‐select

[root@cluster1
hdp]#
hdp-‐select
status
|
grep
hbase

hbase-‐client
-‐
2.2.0.0-‐1995

hbase-‐master
-‐
2.2.0.0-‐1995

-‐
2.2.0.0-‐1995

Upgrade
Servers
Before
Clients

[root@cluster1
hdp]#
hdp-‐select
set
hbase-‐master
2.2.0.1-‐2217

[root@cluster1
current]#
pwd

/usr/hdp/current

[root@cluster1
current]#
ls
-‐l
|
grep
hbase

lrwxrwxrwx.
1
root
root
27
Dec

6
22:57
hbase-‐client
-‐>
/usr/hdp/2.2.0.0-‐1995/hbase

lrwxrwxrwx.
1
root
root
27
Dec

7
02:23
hbase-‐master
-‐>
/usr/hdp/2.2.0.1-‐2217/hbase

lrwxrwxrwx.
1
root
root
27
Dec

6
22:57
-‐>
/usr/hdp/2.2.0.0-‐1995/hbase

Rolling Upgrade Contracts
Rolling Upgrade works for minor upgrades.
•  Example: HDP 2.2.0 to HDP 2.2.1.
Wire compatibility guaranteed between clients and servers.
Binary compatibility guaranteed, e.g. for coprocessors.
Data format compatibility guaranteed.

Rolling Upgrade Benefits
Rolling
Upgrade
Beneﬁt

Upgrade
with
zero
downVme.

Roll
forward
and
roll
backward.

Instant
switchover
/
restart
preserve
data
locality
when
upgrading
HBase.

Update
servers
and
clients
independently.

New in HDP 2.2: HBase on YARN via Slider

Deploying HBase with Slider
What is it?
•  Deploy HBase into the Hadoop cluster using YARN.
Benefit Details
Simplified Deployment No need to deploy HBase or its configuration to individual cluster nodes.
Lifecycle Management Start / stop / process management handled automatically.
Multitenancy Different users can run HBase clusters within one Hadoop cluster.
Multiple Versions Run different versions of HBase (e.g. 0.98 and 1.0) on the same cluster.
Elasticity Cluster size is a parameter and easily changed.
Co-located Analytics HBase resource usage is known to YARN, nodes running HBase will not
be used as heavily to satisfy MapReduce or Tez jobs.

HBase / Slider Sample
Configure HBase settings in appConfig.json and resources.json
Sample Slider Command:
•  slider
create
mycluster

-‐-‐template
appConfig.json

-‐-‐resources
resources.json

{

"schema":
"http://example.org/specification/v2.0.0",

"metadata":
{

},

"global":
{

"site.hbase-‐site.hbase.hstore.flush.retries.number":
"120",

"site.hbase-‐site.hbase.client.keyvalue.maxsize":
"10485760",

"site.hbase-‐site.hbase.hstore.compactionThreshold":
"3",

"site.hbase-‐site.hbase.rootdir":
"${DEFAULT_DATA_DIR}/data",

"site.hbase-‐site.hbase.stagingdir":
"${DEFAULT_DATA_DIR}/staging",

"site.hbase-‐site.hbase.regionserver.handler.count":
"60”,

...

Q & A

Thank you!
Learn more at:
hortonworks.com/hadoop/hbase/
Register for the last
Discover HDP 2.2 Webinar
Hortonworks.com/webinars

Discover.hdp2.2.h base.final[2]

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Similar a Discover.hdp2.2.h base.final[2]

Similar a Discover.hdp2.2.h base.final[2] (20)

Más de Hortonworks

Más de Hortonworks (20)

Último

Último (20)

Discover.hdp2.2.h base.final[2]