Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.

[RakutenTechConf2014] [D-4] The next step of LeoFS and Introducing NewDB Project

44.917 visualizaciones

Publicado el

Rakuten Technology Conference 2014
"The next step of LeoFS and Introducing NewDB Project"
Yosuke Hara w/Paras Patel (Rakuten)

Publicado en: Tecnología
  • DOWNLOAD FULL BOOKS, INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Responder 
    ¿Estás seguro?    No
    Tu mensaje aparecerá aquí

[RakutenTechConf2014] [D-4] The next step of LeoFS and Introducing NewDB Project

  1. 1. The Next Step Yosuke Hara - @yosukehara A Researcher of R.I.T. and Tech Lead LeoFS with Paras Patel, An engineer of Rakuten DU 1
  2. 2. Overview Brief Benchmark Report v1.0 -Multi Data Center Replication v1.1 - NFS Support LeoFS Administration at Rakuten Future Plan 2
  3. 3. Overview 3
  4. 4. 2008 2010 2012 2016 2018 2014 2020 LeoFS - Background Terabyte Petabyte Exabyte 4
  5. 5. The Lion of Storage Systems HIGH Availability 3 Vs in 3 HIGHs HIGH Cost Performance Ratio HIGH Scalability LeoFS Non Stop Velocity: Low Latency Minimum Resources Volume: Petabyte / Exabyte Variety: Photo, Movie, Unstructured-data 5
  6. 6. Storage Engine/Router Metadata Object Storage Monitor GUI Console ( Erlang RPC) LeoFS - Architecture Storage (storage cluster) Manager ( Erlang RPC) Gateway ( TCP/IP,SNMP ) S3-API / NFS - Software Defined Storage - Satisfied the 3 HIGH - Easy Administration Storage Engine/Router Metadata Object Storage Storage Engine/Router Metadata Object Storage Clients ..... Monitoring System Huge Capacity Storage 6
  7. 7. LeoFS Gateway 7
  8. 8. LeoFS Overview - Gateway HTTP Request and Response Built in Object Cache Mechanism REST-API / S3-API Stateless Proxy + Object Cache [ Memory Cache, Disc Cache ] Use Consistent Hashing for decision of a primary node Clients Gateway(s) Storage Cluster Storage Cluster Fast HTTP Server - Cowboy API Handler Object Cache Mechanism 8
  9. 9. LeoFS Storage 9
  10. 10. LeoFS Overview - Storage Gateway Storage (Storage Cluster) WRITE: Auto Replication READ : Auto Repair of an Inconsistent Object with Async Use Consistent Hashing for Data Operation in the Storage Cluster Choosing Replica Target Node(s) RING 2 ^ 128 (MD5) # of replicas = 3 KEY = “bucket/leofs.key” Hash = md5(Filename) Secondary-1 Secondary-2 Primary Node P2P 10
  11. 11. Storage consists of Object Storage and Metadata Storage Includes Replicator and Recoverer for the eventual consistency Request From Gateway LeoFS Overview - Storage ... LeoFS Storage Replicator Recoverer ... Storage Engine Storage Engine, Metadata Gateway + Object Storage Metadata Storage Object Storage 11
  12. 12. LeoFS Overview - Storage - Data Structure Metadata Storage Object Storage for retrieving an object Offset Version Time- Key stamp for Sync Checksum KeySize DataSize Offset Version Time-stamp Key User-Meta Footer Actual File Robust and High Performance Necessary for GC Metadata Checksum KeySize Custom Meta Size File Size Footer (8B) User-Meta Size Header (Metadata - Fixed length) Body (Variable Length) Needle Super-block Needle-1 Needle-2 Needle-3 Object Container Needle-4 Needle-5 Log Structure File Format 12
  13. 13. LeoFS Manager 13
  14. 14. LeoFS Overview - Manager Storage Cluster Monitor RING, Node State Operate status, suspend, resume, detach, whereis, ... Gateway(s) Gateway(s) Storage Cluster Manager(s) Operates LeoFS - Gateway and Storage Cluster RING Monitor and NodeState Monitor 14
  15. 15. Brief Benchmark Report 15
  16. 16. Brief Benchmark Report 1st Case: Group of Value Ranges Storage:5, Gateway:1, Manager:2 R:W = 9:1 source: https://github.com/leo-project/notes/tree/master/leofs/benchmark/leofs/20140605/tests/1m_r9w1_240min 2nd Case: Group of Value Ranges Storage:5, Gateway:1, Manager:2 R:W = 8:2 source: https://github.com/leo-project/notes/tree/master/leofs/benchmark/leofs/20140605/tests/1m_r8w2_120min 16
  17. 17. Brief Benchmark Report Server Spec - Gateway: CPU Intel(R) Xeon(R) CPU X5650 @ 2.67GHz * 2 (12 cores / 24 threads) Memory 96GB Disk HDD - 240GB RAID0 Network 10G-Ether Server Spec - Storage x5: CPU Intel(R) Xeon(R) CPU X5650 @ 2.67GHz * 2 (12 cores / 24 threads) Memory 96GB Disk HDD - 240GB RAID0 (System) HDD - 2TB RAID0 (Data) Network 10G-Ether 17
  18. 18. Brief Benchmark Report - 1st Case (R:W=9:1) Environment: Network 10Gbps OS CentOS release 6.5 (Final) Erlang OTP R16B03-1 LeoFS v1.0.2 System Consistency Level: [ N:3, W:2, R:1, D:2 ] Benchmark Configuration: Duration 4.0h R:W 9:1 # of Concurrent Processes 64 # of Keys 100,000 Value Size Range (byte) Percentage 1,024 10,240 24% 10,241 102,400 30% 102,401 819,200 30% 819,201 1,572,864 16% 18
  19. 19. Brief Benchmark Report - 1st Case (R:W=9:1) The range of the data size of 80% is between 1KB and 1.5MB Range (byte) Percentage 1,024 10,240 24% 10,241 102,400 30% 102,401 819,200 30% 819,201 1,572,864 16% 19
  20. 20. Brief Benchmark Report - 1st Case (R:W=9:1) source: https://github.com/leo-project/notes/tree/master/leofs/benchmark/leofs/20140601/tests/1m_r9w1_240min 50ms 50ms 1,500ops No Errors OPS Latency 20
  21. 21. Brief Benchmark Report - 1st Case / Network Traffic 1,500,000 1,350,000 1,200,000 1,050,000 900,000 750,000 600,000 450,000 300,000 150,000 0 0s 500s 1000s 1500s 2000s 2500s 3000s 3500s 4000s 4500s 5000s 5500s 6000s 6500s 6.0Gbps 7000s 7500s 8000s 8500s 9000s 9500s 10000s 10500s gateway rxbyt/s gateway txbyt/s storage-1 rxbyt/s storage-1 txbyt/s storage-2 rxbyt/s storage-2 txbyt/s storage-3 rxbyt/s storage-3 txbyt/s storage-4 rxbyt/s storage-4 txbyt/s storage-5 rxbyt/s storage-5 txbyt/s 11000s 11500s 12000s 12500s 13000s 13500s 14000s 10.0Gbps 7.0Gbps 5.0Gbps Storage Gateway 60% 21
  22. 22. Brief Benchmark Report - 1st Case / Memory and CPU 14000s 13500s 13000s 12500s 12000s 11500s 11000s 10500s 10000s 9500s 9000s 8500s 8000s 7500s 7000s 6500s 6000s 5500s 5000s 4500s 4000s 3500s 3000s 2500s 2000s 1500s 1000s 500s 0 0.1 0.3 0.4 0.6 0.7 0.9 1.0 1.1 1.3 1.4 1.6 1.7 1.9 2.0 0s Memory Usage CPU Load 5min 1.0 14000s 13500s 13000s 12500s 12000s 11500s 11000s 10500s 10000s 9500s 9000s 8500s 8000s 7500s 7000s 6500s 6000s 5500s 5000s 4500s 4000s 3500s 3000s 2500s 2000s 1500s 1000s 500s 0 10 20 30 40 50 60 70 80 90 100 0s gateway storage-1 storage-2 storage-3 storage-4 storage-5 22
  23. 23. Brief Benchmark Report - 2nd Case (R:W=8:2) Environment: Network 10Gbps OS CentOS release 6.5 (Final) Erlang OTP R16B03-1 LeoFS v1.0.2 System Consistency Level: [ N:3, W:2, R:1, D:2 ] Benchmark Configuration: Duration 2.0h R:W 8:2 # of Concurrent Processes 64 # of Keys 100,000 Value Size Range (byte) Percentage 1,024 10,240 24% 10,241 102,400 30% 102,401 819,200 30% 819,201 1,572,864 16% 23
  24. 24. Brief Benchmark Report - 2nd Case (R:W=8:2) 65ms 85ms 1,000ops No Errors OPS Latency 24
  25. 25. Compare 1st case with 2nd case 25
  26. 26. Brief Benchmark Report 1,500,000 1,350,000 1,200,000 1,050,000 900,000 750,000 600,000 450,000 300,000 150,000 0 0s 500s 1000s 1500s 2000s 2500s 3000s 7.0Gbps 3500s 4000s 4500s 5000s gateway rxbyt/s gateway txbyt/s storage-1 rxbyt/s storage-1 txbyt/s storage-2 rxbyt/s storage-2 txbyt/s storage-3 rxbyt/s storage-3 txbyt/s storage-4 rxbyt/s storage-4 txbyt/s storage-5 rxbyt/s storage-5 txbyt/s 5500s 6000s 6500s 7000s 3,000,000 2,700,000 2,400,000 2,100,000 1,800,000 1,500,000 1,200,000 900,000 600,000 300,000 0 0s 500s 1000s 1500s 2000s 2500s 3000s minus 0.7Gbps 7.0Gbps 3500s 4000s 4500s 5000s 5500s 6000s 6500s 7000s 6.0Gbps 6.0Gbps 1st Case - Network Traffic 2nd Case - Network Traffic 26
  27. 27. Brief Benchmark Report 600.0 550.0 500.0 450.0 400.0 350.0 300.0 250.0 200.0 150.0 100.0 50.0 600.0 550.0 500.0 450.0 400.0 350.0 300.0 250.0 200.0 150.0 100.0 50.0 0 0s 500s 1000s 1500s 2000s 2500s 3000s 3500s 4000s 4500s storage-1 storage-2 storage-3 storage-4 storage-5 5000s 5500s 200 6000s 6500s 7000s 0 0s 500s 1000s 1500s 2000s 2500s 3000s 3500s 4000s 4500s 5000s 5500s 6000s 6500s 7000s 100 100 2nd Case - Disk util% 200 1st Case - Disk util% 1.8x high 27
  28. 28. Brief Benchmark Report 1st Case - CPU Load 5min 3.0 2.8 2.6 2.4 2.2 2.0 1.8 1.6 1.4 1.2 1.0 0.8 0.6 0.4 0.2 0 0s 500s 1000s 1500s 2000s 2500s 3000s 3500s 4000s 4500s 5000s 1.00 5500s 6000s 6500s 7000s gateway storage-1 storage-2 storage-3 storage-4 storage-5 3.0 2.8 2.6 2.4 2.2 2.0 1.8 1.6 1.4 1.2 1.0 0.8 0.6 0.4 0.2 0 0s 500s 1000s 1500s 2000s 2500s 3000s 3500s 4000s 4500s 5000s 5500s 6000s 6500s 7000s 1.00 1.6x high 2nd Case - CPU Load 5min 28
  29. 29. Brief Benchmark Report Conclusion: LeoFS kept in a stable performance through the benchmark Bottleneck is Disk I/O The cache mechanism contributed to reduce network traffic between Gateway and Storage 29
  30. 30. LeoFS v1.0 Multi Data Center Replication 30
  31. 31. LeoFS - For Disaster Recovery Storage cluster Manager cluster For Disaster Recovery = High Scalability + High Availability w/o SPOF and Performance Degradation 31
  32. 32. Multi Data Center Replication Designes it as simple as possible 1. Easy Operation to build multi clusters. 2. Asynchronous data replication between clusters Stacked data is transferred to remote cluster(s) 3. Eventual consistency 32
  33. 33. join cluster DC-2 and DC-3 [# of replicas:3] [# of replicas:1] [# of replicas:1] DC-2 DC-3 Client Storage cluster Manager cluster DC-1 leo_rpc leo_rpc Monitors and Replicates each “RING” and “System Configuration” Leo Storage Platform Multi Data Center Replication Executing “Join Cluster” on Manager Console Preparing the MDC Replication 33
  34. 34. [# of replicas:1] [# of replicas:1] DC-2 DC-3 Multi Data Center Replication Stacking objects Client Storage cluster Manager cluster Monitors and Replicates each “RING” and “System Configuration” Leo Storage Platform Application(s) Request to the Target Region [# of replicas:3] DC-1 Temporally Stacking objects - One container's capacity is *32MB - When capacity is full, send it to remote cluster(s) * 32MB: default capacity - able to set optional value leo_rpc leo_rpc 34
  35. 35. DC-2 DC-3 Client Storage cluster Manager cluster Monitors and Replicates each “RING” and “System Configuration” Leo Storage Platform Application(s) DC-1 Stacked an object with a metadata Compress it with LZ4 Replicated an object Request to the Target Region leo_rpc leo_rpc leo_rpc Multi Data Center Replication Transferring stacked objects Stacked objects 35
  36. 36. leo_rpc Multi Data Center Replication Investigating stored objects DC-2 DC-3 Client Storage cluster Manager cluster leo_rpc Monitor and Replicate each “RING” and “System Configuration” Leo Storage Platform Application(s) Request to the Target Region DC-1 1) Receive metadata of stored objects 2) Compare them at the local cluster 3) Fix inconsistent objects leo_rpc leo_rpc 36
  37. 37. LeoFS v1.1 NFS Support 37
  38. 38. LeoFS v1.1 - NFS Support Offline - NFS / S3-API Online - REST / S3-API PaaS / IaaS BigData Analysis Log / Event / Sensor data BigData Analysis Platform Document / Contents Photo / Movie Various Kind and Huge Amount Data Various Kind Data Any Services Log / Event / Sensor data 38
  39. 39. LeoFS Administration at Rakuten Presented by Paras Patel An engineer of Rakuten DU 39
  40. 40. LeoFS Administration at Rakuten Storage Platform File Sharing Service End-User Servers Portal Site Photo Storage Background Storage of OpenStack 40
  41. 41. 41 Rakuten Services are using LeoFS 41
  42. 42. External Users Project FiFo is an open-source Cloud Management and Orchestration system for SmartOS virtualization environments. The components of FiFo are written entirely in Erlang which gives the suite excellent stability and fault recovery as it continues maturing to a production quality release. Project FiFo uses LeoFS packages for SmartOS in their repository. 42
  43. 43. Storage Platform 43
  44. 44. Storage Platform - Scaling the Storage Platform (Movie) Reduce Costs High Reliability Easy to Scale S3-API 44
  45. 45. Storage Platform - Scaling the Storage Platform Using Various Services Total Usage: 450TB/600TB # of Files: 600Million Daily Growth: 100GB Daily Reqs: 13Million E-Commerce Review Photo Blog Insurance Calendar share Portal Contents Recruiting B Bookmark Storage Platform (Movie) 45
  46. 46. Monitor GUI Console Storage Platform - System Layout ( Erlang RPC) ( Erlang RPC) ( TCP/IP,SNMP ) Gateway x 4 Storage x 14 Manager x 2 Requests from Web Applications / Browsers w/HTTP over S3-API Load Balancer / Cache Servers Total disk space: 600TB Number of Files: 600Million Access Stats: 800Mbps (MAX) 400Mbps (AVG) 46
  47. 47. Monitor GUI Console Ganglia Agent ( Erlang RPC) ( Erlang RPC) ( TCP/IP,SNMP ) Gateway x 4 Storage x 14 Manager x 2 Storage Platform - Monitor Send Mail Alert Status Collection (Ganglia) Status Check (Nagios) Port + Threshold Check 47
  48. 48. Storage Platform - Spreading Globally Covering All Services with Multi DC Replication 48
  49. 49. + = + https://owncloud.com/ http://roma-kvs.org/ 49
  50. 50. File Sharing Service - Required Targets + Reduce Costs Handle Confidential Files Store Large Files Scale Easily 50
  51. 51. File Sharing Service - Usage + Share Docs and Videos with Group Companies Over 20 Companies, Over 10 Countries Over 10,000 Users, Over 4,000 Teams 51
  52. 52. LDAP Monitor Manager x 2 GUI Console ( Erlang RPC) ( Erlang RPC) ( TCP/IP,SNMP ) Authenticate Users Manage Configurations Manage Login Session (KVS) File Sharing Service - System Layout Web GUI File Browser 52
  53. 53. LeoFS Continuous Automated Testing Continuous Integration with Jenkins AWS SDK for PHP AWS SDK for Ruby AWS SDK for Java Boto(Python) erlcloud(Erlang) s3cmd s3fuse LeoFS Client Test: https://github.com/leo-project/leofs_client_tests 53
  54. 54. File Sharing Service - Future Plans Cover 25 Countries/Regions Over 20,000 Users + 54
  55. 55. Enhancing the Services and Empowering the Users through the Cloud Storage 55
  56. 56. LeoFS v2.0 Beyond Storage System, Toward Cloud Storage 56
  57. 57. LeoFS v2.0 - Cloud Storage Always keeps best condition of LeoFS Autonomic Operation Automatic Rebalance Automatic Data Compaction QoS Statistics of internal LeoFS Watchdog Request / Traffic Control 57
  58. 58. LeoFS v2.0 - Cloud Storage Hybrid Storage Centralizes Huge Amount and Various Kind data in LeoFS Supports Online and Offline Access PaaS / IaaS BigData Analysis Log / Event / Sensor data BigData Analysis Platform Document / Contents Photo / Movie Various Kind and Huge Amount Data Various Kind Data Any Services Log / Event / Sensor data 58
  59. 59. LeoFS v2.0 - Cloud Storage PaaS / IaaS BigData Analysis Log / Event / Sensor data BigData Analysis Platform Document / Contents Photo / Movie Various Kind and Huge Amount Data Various Kind Data Any Services Log / Event / Sensor data REST-API (JSON) Operate LeoFS Notify a message of over # of req threshold NewDB Proj for LeoFS Insight + Retrieve metrics and stats from NewDB Proj's Agents - Autonomic operation - NFS Support - Erasure Code - NewDB Integration - OpenStack Integration and more... 59
  60. 60. Introducing NewDB Project 60
  61. 61. NewDB Proj - Background We need to get Value in Data in realtime Typical System Transaction Analysis Cache NewDB Proj is a Hybrid DB Handle Atomic Data and Analyze Data in realtime Typical Systems still realise Semi-realtime Data Analysis + Database Statistics / ML NewDB Proj 61
  62. 62. NewDB Proj - Concept Analyzing Data in Realtime Retrieving Value in Exponential Data Ad-hoc Query Pluggable QL ODBC/JDBC, REST, CLI,... NewDB Proj ML-lib Integration Pluggable ML/Statistics Mechanism High Realiability High Scalability 62
  63. 63. Analyzing Data Handle Event, Sensor and Metrics Data Online Transaction Statistics, ML Clustering Data Finding Patterns in Data Calculate and Generate Data Then Grow Value of Data Pluggable QL Ease of Operation and The QL Express Intuition Directly NewDB Proj - Concept SQL Snapshot, Backup Restore Visualizing Data 63
  64. 64. Early 2015 64
  65. 65. Set Sail for the new Leo land LeoProject Website: leo-project.net Twitter: @LeoFastStorage 65

×