Time series data is increasingly at the heart of modern applications - think IoT, stock trading, clickstreams, social media, and more. With the move from batch to real time systems, the efficient capture and analysis of time series data can enable organizations to better detect and respond to events ahead of their competitors or to improve operational efficiency to reduce cost and risk. Working with time series data is often different from regular application data, and there are best practices you should observe. This talk covers: Common components of an IoT solution The challenges involved with managing time-series data in IoT applications Different schema designs, and how these affect memory and disk utilization – two critical factors in application performance. How to query, analyze and present IoT time-series data using MongoDB Compass and MongoDB Charts At the end of the session, you will have a better understanding of key best practices in managing IoT time-series data with MongoDB.

2. Best practices:
Working with IoT and time-series data
Robert Walters, Director of IoT, MongoDB

3. Robert Walters
Director of IoT, MongoDB
@RobsCranium
LinkedIn: https://www.linkedin.com/in/robwaltersprofile/

5. Agenda
Define time-series data
Review typical IoT Architecture
Designing time-series schemas
Query and visualize time-series data

6. What is time series data?

7. Data in an IoT Architecture

8. Common IoT Architecture

9. Common IoT Architecture

10. Common IoT Architecture

11. Common IoT Architecture

12. Common IoT Architecture

13. Common IoT Architecture

14. MongoDB IoT Reference Architecture

15. Data at the edge

16. Data requirements in IoT solution

17. Data Platform Requirements
Agility Schema Flexibility
Real-time Data Archiving
Global Presence Data Protection
IoT data is a Big Data problem - Volume, Velocity, Variety

18. Freedom to deploy wherever you need

19. Designing time-series schemas

20. Breaking the relational mindset….

21. MongoDB Schema Design
for Time Series data

22. Effects on Document count
Per second vs Per Minute

23. Effects on data storage
Per second vs Per Minute

24. Effects on memory
Per second vs Per Minute

25. Sample code to add to the size based bucket
MongoDB Schema Design
for Time Series data

26. Case Study:
https://www.ampcontrolgroup.com/
Leaders in industrial electrical and electronic engineering
Selected MongoDB for Gas-Detection Mining Application

27. Case Study:

28. Case Study:
One Document per Device per Parameter per Hour
Pre-aggregated data updated for every update
After 3 months samples array removed
Data archived for 7 years

29. Keys to successful time series applications
Know your application requirements
Writes Reads Data Retention Security

30. What to do with old data
• Pre-aggregation
• Offline archival
• db.remove
• TTL Indexes
• Dropping the collection
• Online archival
• Queryable backups
• Sharding archival data
• Offload to data warehouse / S3

31. MongoDB Atlas Data Lake
Analyze data in any format on S3 using MongoDB Query Language
Archive IoT and Time-series data for long term storage and analysis

32. How to query and visualize time-series data

33. Real-time Analytics
One database for operational and analytical data
Easy to query unstructured data
MongoDB Query Language (MQL)
Aggregation Framework

34. Real-time Analytics
Easily query unstructured data with Aggregation Framework

35. What is the largest value for a given sensor id?
db.SensorData.aggregate( [
{$match: { "sensorid":3003 }},
{$unwind: { path: '$samples' }},
{$group: { _id: '$day', MaxValue: { $max:
'$samples.val' } }},
{$sort: { "id": 1 }} ])

36. What is the largest value for a given sensor id?
db.SensorData.aggregate( [
{$match: { "sensorid":3003 }},
{$unwind: { path: '$samples' }},
{$group: { _id: '$day', MaxValue: { $max:
'$samples.val' } }},
{$sort: { "id": 1 }} ])

37. What is the largest value for a given sensor id?
db.SensorData.aggregate( [
{$match: { "sensorid":3003 }},
{$unwind: { path: '$samples' }},
{$group: { _id: '$day', MaxValue: { $max:
'$samples.val' } }},
{$sort: { "id": 1 }} ])

38. What is the largest value for a given sensor id?
db.SensorData.aggregate( [
{$match: { "sensorid":3003 }},
{$unwind: { path: '$samples' }},
{$group: { _id: '$day', MaxValue: { $max:
'$samples.val' } }},
{$sort: { "id": 1 }} ])

39. Cloud – Real-time Analytics
What is the largest value for a given sensor id?
db.SensorData.aggregate( [
{$match: { "sensorid":3003 }},
{$unwind: { path: '$samples' }},
{$group: { id: '$day', MaxValue: { $max: '$samples.val' }
}},
{$sort: { "id": 1 }} ])
{ "_id" : ISODate("2018-12-10T00:00:00Z"), "MaxValue" : 40.4 }
{ "_id" : ISODate("2018-12-11T00:00:00Z"), "MaxValue" : 40.43 }
{ "_id" : ISODate("2018-12-12T00:00:00Z"), "MaxValue" : 40.09 }
{ "_id" : ISODate("2018-12-13T00:00:00Z"), "MaxValue" : 39.7 }
…

40. Cloud – Visualization
MongoDB Charts

41. Cloud – Visualization
MongoDB BI Connector

42. Integration with

43. Why MongoDB for IoT and Time-Series Data?

44. Why MongoDB for IoT Data?

45. Best practices summary
Know your requirements
1 sensor reading per document works but not the most efficient
Data retention policy
Data access requirements
For very large data consider multiple collections
Faster to archive/drop collections than db.remove()
Recommended design fixed-based schema
https://www.mongodb.com/collateral/time-series-best-practices
Archive data to S3 / Atlas Data Lake for long term storage and analysis

46. Resources
MongoDB IoT Reference Architecture
Time Series Data and MongoDB: Best Practices Guide
Get Started with your project in MongoDB Atlas.
Use code “IOT200” for $200 in credit!
Thinking about an IoT Solution? Robert.Walters@MongoDB.COM

47. Thank You!