Recomendados
Recomendados
Más contenido relacionado
Más de Michael Stack
Más de Michael Stack (20)
Último
Último (20)
HBaseConEast2016: OpenTSDB+BigTable
- 1. OpenTSDB + Bigtable Integrating time series database with Google Cloud Bigtable Danil Zburivsky, Big Data Practice Lead, Pythian — zburivsky@pythian.com Max Luebbe, SRE, Google — maxluebbe@google.com
- 2. Pythian specializes in design, implementation, and management of systems that directly contribute to revenue and business success. History 19 years in business Growing at 30+% per year 400+ employees 300+ customers worldwide HQ Ottawa, Canada - global reach Technology agnostic = trusted advisor Deep expertise: Oracle, Oracle Apps, MySQL, AWS, SQL Server, Cassandra/DataStax, Azure, PostgreSQL, Cloudera, MapR, Hortonworks etc. Google Premier Partner Status (as of end Aug) 5 Certified Developers (soon to be 12) Dedicated Google Technical Champion Launch partner for: Kubernetes, Dataflow, Cloud SQL, Dataproc Integrated OpenTSDB with Bigtable DW Explorers Program Partner Upcoming BigQuery & Cloud ML Launch Partner
- 3. • (time, metric, value) • OS and apps metrics • Industrial equipment • Web traffic Time series data
- 4. • Volume can be explosive • Data arrival and access patterns are different Storing time series data is a challenge
- 5. • Volume can be explosive • Data arrival and access patterns are different Storing time series data is a challenge
- 6. OpenTSDB Architecture Server Server Server Server TSD TSD Hbase or Bigtable TSD RPC Hbase RPC or Hbase API Web UI Scripts/Alerting http TSD RPC
- 7. • Open source • Uses HBase as a data store • Data model optimized for TS • REST API OpenTSDB <metric_uid><timestamp><tagk1><tagv1>[...<tagkN><tagvN>] <col_t+1>[...<col_t+N>]
- 8. Origins of Bigtable • Updating search index in bulk: too slow! • Rather than a file system (GFS), needed random access • Generic & scalable, now powers many diverse products
- 9. Bigtable decouples compute & storage... Node Distributed filesystem Node Client Client Client Client Client Client Processing Storage Clients A B C D E
- 10. ...making scalability fast & seamless Node Distributed filesystem Node Client Client Client Client Client Client Processing Storage Clients A B C D E Node
- 11. Google Cloud Bigtable Cloud Bigtable Bigtable Service
- 12. async-bigtable ● Uses standard HBase 1.0 API ● BufferedMutator ● Thread pool https://github.com/OpenTSDB/asyncbigtable
- 13. Future work ● Native Bigtable API ● Fully asynchronous ● Performance
- 14. Demo time!