1. Apache Hadoop and HBase in the Real World
Joey Echeverria
@fwiffo
#novahug

2. The Plug
We're Training!
Developer Training July 25 to 27
Admin Training July 28 to 29
http://www.cloudera.com/training
We’re Hiring!
Solution Architects, Trainers,
Distributed Systems Engineers
http://www.cloudera.com/careers
Copyright 2011 Cloudera Inc. All rights reserved 2

3. 1 Minute Hadoop Recap
HDFS
Distributed file system
Optimized for streaming reads and writes
Block-level replication
MapReduce
Distributed processing framework
Reads/writes data in HDFS (typically)
Operates over (key, value) view of data
Copyright 2011 Cloudera Inc. All rights reserved 3

4. Where does HBase come in?
Google
Google invented GFS and MapReduce
GFS optimized for streaming reads and writes
BigTable
Google's answer to random read/write workloads
Copyright 2011 Cloudera Inc. All rights reserved 4

5. HBase: BigTable-like storage (for Hadoop)
Copyright 2011 Cloudera Inc. All rights reserved 5

6. What is HBase?
Key/value column family store
Data stored in HDFS
ZooKeeper for coordination
Access model is get/put/del
Plus range scans and versions
Copyright 2011 Cloudera Inc. All rights reserved 6

7. Architecture
Image courtsey Lars George, Licensed uner Creative Commons Attribution-Noncomm
Copyright 2011 Cloudera Inc. All rights reserved 7

8. Tables and Column Families
Static part of the schema
Column families also form locality groups
One Store per family
Multiple HFiles per Store
Tables split into regions
Continuous range of row keys
Unit of distribution
Automatically split
Pre-split for performance
Copyright 2011 Cloudera Inc. All rights reserved 8

9. Why use HBase?
Variable schema in each record
Collections of data for each key
Atomic control of per-key data
Row access to each column family
Copyright 2011 Cloudera Inc. All rights reserved 9

10. HBase Applications
“Smart Data, at Scale, made Easy”
http://www.lilyproject.org
“Distributed, scalable Time Series Database (TSDB)”
http://opentsdb.net
Copyright 2011 Cloudera Inc. All rights reserved 10

11. Real-time ad optimizations
Capturing impressions and serving ads
HBase front-end – to serve models (via memcached)
HBase back-end – to serve pixels and capture cookies
MapReduce to compute models between the two
Copyright 2011 Cloudera Inc. All rights reserved 11

12. Click stream sessionization
Key on userid and time
Seperate table for significant events (e.g. purchase)
Load data using HBase importtsv tool
Sessionization performed by simple scans
Copyright 2011 Cloudera Inc. All rights reserved 12

13. Mozilla - Soccorro
When Firefox crashes, where do reports go?
The Mozilla team gathers those crashes in HBase
Crashes varry widely and change format often
Processors take each individually and parse it out
http://code.google.com/p/socorro
Copyright 2011 Cloudera Inc. All rights reserved 13

14. Navteq
Location based content serving
All served out of HBase, location makes a great key
Content is variable – Maps, POI, User Data
Preprocessing is all done via MR jobs
Copyright 2011 Cloudera Inc. All rights reserved 14

15. Cloudera
Gathers data about customer clusters
Each customer node is a key with Avro values
Easy to browse, quick to find issues on Nodes
Dump to HDFS and process with Pig
Copyright 2011 Cloudera Inc. All rights reserved 15

16. Copyright 2011 Cloudera Inc. All rights reserved 16