The document summarizes HBase use at Facebook, including its development and future work. HBase is used for incremental updates to data warehouses, high frequency analytics, and write-intensive workloads. Development includes Hive integration, master high availability, and random read optimizations. Future work focuses on coprocessors, intelligent load balancing, and cluster performance.

2. HBase at Facebook
Jonathan Gray
Hadoop World
October 12, 2010

3. 1 Data at Facebook
2 HBase Development
3 Future Work
Agenda

4. Data at Facebook

5. 500 Million active monthly users
>500 Billion page views per month
25 Billion pieces of content per month

6. Cache
OS Web server Database Language
Data analysis

7. Daily Federated MySQLSilver Cluster
Platinum
Hadoop
Cluster
5-15 minutes5-15 minutes
Cluster 1 Cluster 2
Scribe/Hadoop
cluster
Scribe/Hadoop
cluster

8. Cache
OS Web server Database Language
Data analysis

9. Cache
OS Web server Database Language
Data analysis

10. HBase
Key Properties
▪ Linearly scalable
▪ Fast indexed writes
▪ Tight integration with Hadoop
Bridges gap between online and offline

11. Use Case #1
Incremental Updates into Data Warehouse
▪ Currently
▪ Nightly dumps of UDBs into Warehouse
▪ With HBase
▪ Tail UDB replication logs into HBase
UDB to Warehouse in minutes

12. Use Case #2
High Frequency Counters and Realtime Analytics
▪ Currently
▪ Scribe to HDFS, periodically aggregate to UDB
▪ With HBase
▪ Scribe to HBase, read in realtime with API or MR
Storage, serving, and analysis in one

13. Use Case #3
User-facing Database for Write Intensive Workloads
▪ Currently
▪ Constantly expanding UDB and Memcache tiers
▪ With HBase
▪ Fast writes, automatic partitioning, linear scaling
Fast and scalable writes, just add nodes

14. HBase Development

15. Hive Integration
HBase and Hive
▪ HBase Tables usable as Hive Tables
▪ ETL data target
▪ Query data source
▪ Support for different read/write patterns
▪ API random write or MR bulk load
▪ API random read or MR table scan

16. HBase Master
Re-architected for HA and Testability
▪ Increased usage of ZooKeeper for failover
▪ Region transitions in ZK
▪ Working master failover in all cases
▪ Refactor/Redesign of major components
▪ Load balancer, cluster startup, failover redesigned
▪ Emphasis on independent testability

17. Random Read Optimizations
Performance degrades with lots of files
▪ Bloom filters
▪ Dynamic Row or Row+Column as HFile metadata
▪ Skip files on disk that don’t match
▪ Timestamp ranges
▪ Stored as HFile metadata
▪ Skip files on disk that don’t cover time range

18. Random Read Optimizations
Performance degrades with wide rows
▪ Aggressively seek/reseek
▪ Use query and block index to skip blocks
▪ Stop processing as soon as we finish query
▪ Expose seeking to Filter API
▪ Allow specialized optimizations
▪ Millions of versions in a row, grab 10

19. Administration Tools
Detect and repair potential issues
▪ HBCK
▪ FSCK for HBase
▪ Detect and repair cluster issues
▪ Cluster Verification
▪ Ensure cluster can be written to, read from
▪ Tables can be created/disabled/dropped

20. Hadoop Improvements
HDFS Appends
▪ Hadoop 0.20
▪ Widely deployed but no support for appends
▪ Hadoop 0.20 with append support
▪ Apache Hadoop 0.20-Append
▪ Cloudera’s CDH version 3
▪ Facebook’s version of Hadoop 0.20
▪ http://github.com/facebook/hadoop-20-append

21. Hadoop Improvements
HDFS rolling upgrades and NameNode HA
▪ HDFS in online application
▪ Need to support upgrades without downtime
▪ More sensitive to NameNode SPOF
▪ Hadoop AvatarNode
▪ Hot standby pair of NameNodes
▪ Failover to new version of NameNode
▪ Failover to hot standby in seconds under failure

22. Coming Soon
New Features
▪ East coast / west coast replication
▪ Asynchronous replication between data centers
▪ Faster recovery
▪ Distributed log splitting
▪ Master controlled rolling restart
▪ Fast and retaining assignment information

23. Future Work

24. Coprocessors
Complex server-side operations
▪ Dynamically loaded server-side logic
▪ Hook into read/write and cluster operations
▪ Endless possibilities
▪ Server-side merges and joins
▪ Lightweight MapReduce for aggregation
▪ Efficient secondary indexing

25. Intelligent Load Balancing
Complex notion of load
▪ Currently based only on region count
▪ Different regions have different access patterns
▪ And data locality equally important
▪ Next generation load balancing algorithms
▪ Consider complex notion of read load / write load
▪ And HDFS block locations for locality
▪ Retain assignment information between restarts

26. Other Future Work
Cluster Performance
▪ Quality of service
▪ One MapReduce job can take down cluster
▪ Dynamic configuration changes
▪ Change important parameters on running cluster
▪ HDFS performance
▪ Critical target for long-term HBase performance

27. (c) 2007 Facebook, Inc. or its licensors. "Facebook" is a registered trademark of Facebook, Inc.. All rights reserved. 1.0