Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.

Managing multi tenant resource toward Hive 2.0

1.553 visualizaciones

Publicado el

Migration architecture toward Hive 2.0 in Treasure Data.

Publicado en: Software
  • Sé el primero en comentar

Managing multi tenant resource toward Hive 2.0

  1. 1. Managing multi tenant resource toward Hive 2.0 Kai Sasaki Treasure Data Inc.
  2. 2. About Me • Kai Sasaki (佐々木 海) • @Lewuathe (Twitter) • Software Engineer 
 at Treasure Data Inc. • Maintaining and develop 
 Hadoop/Presto infrastructure
  3. 3. Topic • Treasure Data infrastructure • Hive 2.0 change • Migration architecture • Resource management for multi tenancy • Performance comparison
  4. 4. • Live Data Management Platform • Original creator of Fluentd/Embulk/Digdag • 70+ integrations with • BI tools • Mobile/IoT • Cloud Storage • and more
  5. 5. • Hive/Pig/Presto data processing interface • 40000+ Hive queries / day • 130000+ Presto queries / day • Plazma Cloud Storage • 450000+ records/sec imported
  6. 6. Hive 1.x Hive 2.x Any change?
  7. 7. Hive 2.0 • Include major new features • Fixed 600+ bugs • 140+ improvements or new features • Backward compatible as much as possible • Hive 1.x stable line • 2.1.0 is available from June 20th, 2016 http://www.slideshare.net/HadoopSummit/apache-hive-20-sql-speed-scale
  8. 8. Hive 2.0 • HPLSQL • LLAP • HBase metastore • Improvements of Hive on Spark • CBO improvements http://www.slideshare.net/HadoopSummit/apache-hive-20-sql-speed-scale
  9. 9. HPLSQL • Procedural SQL like Oracle’s PL/SQL • Cursor • loops (WHILE, FOR, LOOP) • branches (IF) • External library which communicates through JDBC • http://www.hplsql.org/doc http://www.slideshare.net/HadoopSummit/apache-hive-20-sql-speed-scale
  10. 10. LLAP • Sub-second Queries in Hive • Save JVM container launch time • Data caching • Fit to Adhoc or interactive use case • Beta in 2.0 http://hortonworks.com/wp-content/uploads/2014/09/Screen-Shot-2014-09-02-at-5.03.47-PM.png
  11. 11. LLAP • Sub-second Queries in Hive http://hortonworks.com/wp-content/uploads/2014/09/Screen-Shot-2014-09-02-at-5.03.47-PM.png
  12. 12. HBase metastore • Use HBase as metastore of Hive • Fetching thousands of partitions • Limitation of concurrent connection • Will support transaction with Apache Omid • Alpha in Hive 2.0 http://hortonworks.com/wp-content/uploads/2014/09/Screen-Shot-2014-09-02-at-5.03.47-PM.png
  13. 13. Many fixes and 
 Cutting edge features
  14. 14. That’s all? • Operation cost of migration • Manage multiple cluster • Test and verify multiple packages • Difference of configuration and parameter
  15. 15. That’s all? • Operation cost of migration • Manage multiple cluster • Test and verify multiple packages • Difference of configuration and parameter • Need to reduce operation cost at the same time
  16. 16. Now migration
  17. 17. Challenge • NO DOWNTIME • NO HARMFUL OPERATION • Change package easily • Separate from other components (Micro service) • NO DEGRADATION • Automatic query test and validation
  18. 18. NO DOWNTIME • Hadoop cluster Blue-Green deployment • Reliable queue system separated from Hadoop → PerfectQueue • Reliable storage system separated from Hadoop → Plazma
  19. 19. PerfectQueue • Distributed queue built on top of RDBMS • At-least-once semantics • Graceful and live restarting • State consistency by transaction • https://github.com/treasure-data/perfectqueue
  20. 20. Plazma • Distributed cloud-based storage • PostgreSQL + S3/Riak CS • Enable time-index push down for Hive/Pig/Presto • Column-oriented IO (mpc1) • Data consistency with transactional API
  21. 21. Plazma x PQ PQ App request
  22. 22. Plazma x PQ PQ App request pull submit
  23. 23. Plazma x PQ PQ App request pull submit fetch
  24. 24. Plazma x PQ PQ App request pull submit fetch disposable components
  25. 25. Plazma x PQ PQ App request pull submit fetch v1 v2
  26. 26. Plazma x PQ PQ App request pull submit fetch v1 v2
  27. 27. Plazma PQ PQ App request pull submit fetch v2
  28. 28. NO HARMFUL OPS • Automatic package version up • Chef server specifies the version • Hadoop package repository • S3 remote package repository • Hadoop as a REST service • elephant-server
  29. 29. elephant-server • Hadoop as REST service • Pluggable executor • Hive • Pig • Embulk MapReduce executor • Distributed on-memory queue (Hazelcast)
  30. 30. PQ PQ App request pull REST elephant server
  31. 31. PQ PQ App request pull REST elephant server elephant server elephant server
  32. 32. PQ PQ App request pull REST elephant server hazelcast elephant server elephant server
  33. 33. PQ PQ App request pull REST elephant server hazelcast elephant server elephant server service discovery
  34. 34. PQ PQ App request pull REST elephant server hazelcast elephant server elephant server service discovery x x
  35. 35. PQ PQ App request pull REST elephant server hazelcast elephant server elephant server service discovery package distribution S3 x x
  36. 36. PQ PQ App request pull REST elephant server hazelcast elephant server elephant server request x x fetch submit service discovery package distribution S3
  37. 37. NO DEGRADATION • Validation in • Parameter difference • Query result difference • Performance deterioration • Automatic testing and persistent result tables
  38. 38. PQ PQ App request pull REST elephant server S3 1. upload param 
 and configurations
  39. 39. PQ PQ App request pull REST elephant server S3 1. upload param 
 and configurations x submit v1
  40. 40. PQ PQ App request pull REST elephant server S3 1. upload param 
 and configurations 2. upload query result Plazma x submit v1 3. send metrics
  41. 41. PQ PQ App request pull REST elephant server S3 1. upload param 
 and configurations 2. upload query result Plazma x submit v1 3. send metrics S3 Plazma x v2
  42. 42. elephant server S3 1. upload param 
 and configurations 2. upload query result Plazma x submit v1 3. send metrics S3 Plazma x v2 Verification between 
 persistent result set PQ PQ App request pull REST
  43. 43. Resource management • Define 1 resource per 1 account • Workload type of an account varies • Batch, Adhoc, BI tool… • Require high level resource management 
 across clusters • An account can have multiple resource pools • For service and internal purpose
  44. 44. request queue1 queue2 cluster1 cluster2 cluster1 cluster2 Hadoop queue A Hadoop queue B Hadoop queue A Hadoop queue B Hadoop queue A Hadoop queue B Hadoop queue A Hadoop queue B
  45. 45. request queue1 queue2 cluster1 cluster2 cluster1 cluster2 Hadoop queue A Hadoop queue B Hadoop queue A Hadoop queue B Hadoop queue A Hadoop queue B Hadoop queue A Hadoop queue B Enables us to define which resource the request can use
  46. 46. PQ PQ App request REST elephant server x
  47. 47. PQ PQ App request REST elephant server PQ PQ x 1. multiple job queue
  48. 48. PQ PQ App request REST elephant server x x PQ PQ 1. multiple job queue 2. multiple Hadoop cluster
  49. 49. PQ PQ App request REST elephant server x q1 q2 q3 x PQ PQ q1 q2 q3 1. multiple job queue 2. multiple Hadoop cluster 3. multiple Hadoop queue
  50. 50. Briefly performance comparison
  51. 51. 130GB+ 70B+ records Elapsedtime(sec) 0 200 400 600 800 COUNT Hive 1.x + MapReduce Hive 2.x + Tez + Vectorization
  52. 52. 130GB+ 70B+ records Elapsedtime(sec) 0 250 500 750 1000 GROUP BY Hive 1.x + MapReduce Hive 2.x + Tez + Vectorization
  53. 53. 130GB+ 70B+ records Elapsedtime(sec) 0 275 550 825 1100 JOIN Hive 1.x + MapReduce Hive 2.x + Tez + Vectorization
  54. 54. Recap • Hadoop architecture in Treasure Data 
 for Hive 2.0 and beyond • Resource management for multi tenancy
  55. 55. We’re hiring!

×