Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.

MongoDB SoCal 2020: Best Practices for Working with IoT and Time-series Data

559 visualizaciones

Publicado el

Time series data is increasingly at the heart of modern applications - think IoT, stock trading, clickstreams, social media, and more. With the move from batch to real time systems, the efficient capture and analysis of time series data can enable organizations to better detect and respond to events ahead of their competitors or to improve operational efficiency to reduce cost and risk. Working with time series data is often different from regular application data, and there are best practices you should observe.

This talk covers:

Common components of an IoT solution
The challenges involved with managing time-series data in IoT applications
Different schema designs, and how these affect memory and disk utilization – two critical factors in application performance.
How to query, analyze and present IoT time-series data using MongoDB Compass and MongoDB Charts
At the end of the session, you will have a better understanding of key best practices in managing IoT time-series data with MongoDB.

Publicado en: Tecnología
  • Inicia sesión para ver los comentarios

MongoDB SoCal 2020: Best Practices for Working with IoT and Time-series Data

  1. 1. @ #MDBlocal Best practices: Working with IoT and Time-Series Data Robert Walters, Director of IoT, MongoDB @RobsCranium SOCAL LinkedIn: https://www.linkedin.com/in/robwaltersprofile/
  2. 2. #MDBLocal
  3. 3. #MDBLocal Agenda Define time-series data Review typical IoT Architecture Designing time-series schemas Query and visualize time-series data
  4. 4. #MDBLocal What is time series data?
  5. 5. Data in an IoT Architecture
  6. 6. #MDBLocal https://tinyurl.com/Mongodb-Houston-2019 https://www.mongodb.com/white-papers
  7. 7. #MDBLocal Common IoT Architecture
  8. 8. #MDBLocal Common IoT Architecture
  9. 9. #MDBLocal Common IoT Architecture
  10. 10. #MDBLocal Common IoT Architecture
  11. 11. #MDBLocal Common IoT Architecture
  12. 12. #MDBLocal Common IoT Architecture
  13. 13. #MDBLocal MongoDB IoT Reference Architecture
  14. 14. #MDBLocal Data at the edge
  15. 15. Designing time-series schemas
  16. 16. #MDBLocal Breaking the relational mindset….
  17. 17. #MDBLocal MongoDB Schema Design for Time Series data
  18. 18. #MDBLocal Effects on Document count Per second vs Per Minute
  19. 19. #MDBLocal Effects on data storage Per second vs Per Minute
  20. 20. #MDBLocal Effects on memory Per second vs Per Minute
  21. 21. #MDBLocal Sample code to add to the size based bucketMongoDB Schema Design for Time Series data
  22. 22. #MDBLocal Case Study: https://www.ampcontrolgroup.com/ Leaders in industrial electrical and electronic engineering Selected MongoDB for Gas-Detection Mining Application
  23. 23. #MDBLocal Case Study:
  24. 24. #MDBLocal Case Study: One Document per Device per Parameter per Hour Pre-aggregated data updated for every update After 3 months samples array removed Data archived for 7 years
  25. 25. #MDBLocal Keys to successful time series applications Know your application requirements Writes Reads Data Retention Security
  26. 26. #MDBLocal What to do with old data • Pre-aggregation • Offline archival • db.remove • TTL Indexes • Dropping the collection • Online archival • Queryable backups • Sharding archival data • Offload to data warehouse / S3
  27. 27. #MDBLocal MongoDB Atlas Data Lake Analyze data in any format on S3 using MongoDB Query Language Archive IoT and Time-series data for long term storage and analysis
  28. 28. How to query and visualize time-series data
  29. 29. #MDBLocal Real-time Analytics One database for operational and analytical data Easy to query unstructured data MongoDB Query Language (MQL) Aggregation Framework
  30. 30. #MDBLocal Real-time Analytics Easily query unstructured data with Aggregation Framework
  31. 31. #MDBLocal Using MQL Demo
  32. 32. What is the largest value for a given sensor id? db.SensorData.aggregate( [ {$match: { "sensorid":3003 }}, {$unwind: { path: '$samples' }}, {$group: { _id: '$day', MaxValue: { $max: '$samples.val' } }}, {$sort: { "id": 1 }} ])
  33. 33. What is the largest value for a given sensor id? db.SensorData.aggregate( [ {$match: { "sensorid":3003 }}, {$unwind: { path: '$samples' }}, {$group: { _id: '$day', MaxValue: { $max: '$samples.val' } }}, {$sort: { "id": 1 }} ])
  34. 34. What is the largest value for a given sensor id? db.SensorData.aggregate( [ {$match: { "sensorid":3003 }}, {$unwind: { path: '$samples' }}, {$group: { _id: '$day', MaxValue: { $max: '$samples.val' } }}, {$sort: { "id": 1 }} ])
  35. 35. What is the largest value for a given sensor id? db.SensorData.aggregate( [ {$match: { "sensorid":3003 }}, {$unwind: { path: '$samples' }}, {$group: { _id: '$day', MaxValue: { $max: '$samples.val' } }}, {$sort: { "id": 1 }} ])
  36. 36. Cloud – Real-time Analytics What is the largest value for a given sensor id? db.SensorData.aggregate( [ {$match: { "sensorid":3003 }}, {$unwind: { path: '$samples' }}, {$group: { id: '$day', MaxValue: { $max: '$samples.val' } }}, {$sort: { "id": 1 }} ]) { "_id" : ISODate("2018-12-10T00:00:00Z"), "MaxValue" : 40.4 } { "_id" : ISODate("2018-12-11T00:00:00Z"), "MaxValue" : 40.43 } { "_id" : ISODate("2018-12-12T00:00:00Z"), "MaxValue" : 40.09 } { "_id" : ISODate("2018-12-13T00:00:00Z"), "MaxValue" : 39.7 } …
  37. 37. #MDBLocal Cloud – Visualization MongoDB Charts
  38. 38. #MDBLocal Cloud – Visualization MongoDB BI Connector
  39. 39. #MDBLocal Integration with
  40. 40. Why MongoDB for IoT and Time- Series Data?
  41. 41. #MDBLocal Why MongoDB for IoT Data?
  42. 42. #MDBLocal Best practices summary Know your requirements 1 sensor reading per document works but not the most efficient Data retention policy Data access requirements For very large data consider multiple collections Faster to archive/drop collections than db.remove() Recommended design fixed-based schema https://www.mongodb.com/collateral/time-series-best-practices Archive data to S3 / Atlas Data Lake for long term storage and analysis
  43. 43. #MDBLocal Resources MongoDB IoT Reference Architecture Time Series Data and MongoDB: Best Practices Guide Get Started with your project in MongoDB Atlas. Use code “IOT200” for $200 in credit! Questions about your IoT Solution? Robert.Walters@MongoDB.COM
  44. 44. THANK YOU

×