Hw09 Hadoop Development At Facebook Hive And Hdfs

Hadoop and Hive Development at Facebook Dhruba Borthakur Zheng Shao {dhruba, zshao}@facebook.com Presented at Hadoop World, New York October 2, 2009

Who generates this data? ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Data Usage ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Where is this data stored? ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Data Flow into Hadoop Cloud Web Servers Scribe MidTier Network Storage and Servers Hadoop Hive Warehouse Oracle RAC MySQL

Hadoop Scribe: Avoid Costly Filers Web Servers Scribe Writers Realtime Hadoop Cluster Hadoop Hive Warehouse Oracle RAC MySQL Scribe MidTier http://hadoopblog.blogspot.com/2009/06/hdfs-scribe-integration.html

HDFS Raid ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],A A B B A+B+C A B http://hadoopblog.blogspot.com/2009/08/hdfs-and-erasure-codes-hdfs-raid.html C C C A file with three blocks A, B and C

Archival: Move old data to cheap storage Cheap NAS Hadoop Archival Cluster Hadoop Archive Node NFS Hive Query Hadoop Warehouse http://issues.apache.org/jira/browse/HDFS-220

Dynamic-size MapReduce Clusters ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Resource Aware Scheduling (Fair Share Scheduler) ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Hive – Data Warehouse ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Hive Architecture HDFS Map Reduce Web UI + Hive CLI + JDBC/ODBC Browse, Query, DDL Hive QL Parser Planner Optimizer Execution SerDe CSV Thrift Regex UDF/UDAF substr sum average FileFormats TextFile SequenceFile RCFile User-defined Map-reduce Scripts

Hive DDL ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Hive Query Language ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Group By ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Join ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Sampling ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Multi-table Group-By/Insert ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

File Formats ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

SerDe ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

UDF/UDAF ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Hive – Performance ,[object Object],[object Object],[object Object],[object Object],[object Object],Date SVN Revision Major Changes Query A Query B Query C 2/22/2009 746906 Before Lazy Deserialization 83 sec 98 sec 183 sec 2/23/2009 747293 Lazy Deserialization 40 sec 66 sec 185 sec 3/6/2009 751166 Map-side Aggregation 22 sec 67 sec 182 sec 4/29/2009 770074 Object Reuse 21 sec 49 sec 130 sec 6/3/2009 781633 Map-side Join * 21 sec 48 sec 132 sec 8/5/2009 801497 Lazy Binary Format * 21 sec 48 sec 132 sec

Hive – Future Works ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Hw09 Hadoop Development At Facebook Hive And Hdfs

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Destacado

Destacado (20)

Similar a Hw09 Hadoop Development At Facebook Hive And Hdfs

Similar a Hw09 Hadoop Development At Facebook Hive And Hdfs (20)

Más de Cloudera, Inc.

Más de Cloudera, Inc. (20)

Último

Último (20)

Hw09 Hadoop Development At Facebook Hive And Hdfs

Notas del editor