Classification of incidents in big IoT data analytics in the Cloud

1. Characterizing Incidents in Cloud-
based IoT Data Analytics
Hong-Linh Truong and Manfred Halper
Faculty of Informatics, TU Wien, Austria
hong-linh.truong@tuwien.ac.at
http://rdsea.github.io
@linhsolar
The 42th IEEE COMPSAC 2018, 24 July 2018, Tokyo 1

2. Outline
 Motivation
 Cloud-based Big IoT data analysis pipelines
 Characterization of incidents
 Classification of incidents
 Conclusions and future work
The 42th IEEE COMPSAC 2018, 24 July 2018, Tokyo 2

3. Motivation
 Big IoT data analytics pipelines and software
services are complex
 But most researches have not been focused on
service management w.r.t. end-to-end incidents
 We tend to focus on ML, analytics algorithms
 Many types of incidents might occur
 Typical system incidents
 Data-centric incidents
 But they are interdependent and across multiple
providers and layers
 Incidents are based on customer-specific constraints
The 42th IEEE COMPSAC 2018, 24 July 2018, Tokyo 3

4. Public cloud infrastructures
Private cloud infrs.
Base Transceiver Station (BTS)
Case Study BTS
 Large-scale systems (1K+ BTS)
 Flexible back-end clouds
 Generic enough for other applications (e.g., in smart agriculture)
 With weak infrastructures for IoT and connectivity
The 42th IEEE COMPSAC 2018, 24 July 2018, Tokyo 4
Sensor
IoT
Gateway
MQTT
Broker
BigQuery
Influxdb
Hadoop FS
G. StorageActuator
Optimizer Analytics
Analytics
Analytics

5. Apache Nifi
Big data storage (Hadoop
FS/Google Storage)
analytics
result
BTS
Monitoring
SFTP
Apache Spark
Enrichment
Service
Kibana
Visualization
analyticsanalytics
result
result
result
result
ElasticSearch
result
result
result
result
resultdata
notification
analytics
results
Web
services
Client
BTS
Monitoring
MQTT
RabbitMQ
BatchAnalytics
Manager
Analytics Web
Service
Planner
Streaming Data
Processing
Ingestion
Service
BigQuery
Analytics
Service
The complexity of software stacks
and subsystems
Source: Simplified version of the
design from I & A Computing Lab, VN
www.inacomputing.com
The 42th IEEE COMPSAC 2018, 24 July 2018, Tokyo 5

6. Some real examples of incidents
 Due to a minor error in the location data,
several stations are wrongly clustered by
machine learning algorithms.
 Integrating and analyzing logs from specific
systems of different vendors, including Huawei,
Ericsson and Nokia
• Due to some mistakes in tasks for extracting logs
and merging data, tasks for data analytics have
produced some wrong results.
The 42th IEEE COMPSAC 2018, 24 July 2018, Tokyo 6

7. Approach and contribution
 Approach:
 To identify and classify potential incidents in big IoT
data analytics
 To determine important, focused data-centric
incidents in IoT data analysis systems
 Solutions in this paper:
 Understand big IoT data analytics pipelines
 Characterization of incidents
 Classification of incidents and relevant concepts
The 42th IEEE COMPSAC 2018, 24 July 2018, Tokyo 7
The long version of the paper: https://bit.ly/2v0hAtk

8. Analysis/
Transformation
Task
IoT
Sensor
Data
Storage
Resulting
analytics
Message
Broker/Data
Logistics
Service
….
Large number
of data
sources (e.g.,
IoT devices)
Large-scale
brokers & data
transfer/logistics
services
Complex big data
processing
frameworks
Other
systems in
the pipeline
IoT
Gateway
Analysis/
Transformation
Task
Abstracting the complex software
pipelines for incident analytics
 Group complex software services into categories
 Focus on common phases in big data: Acquisition,
Preparation, Analysis and Delivery
The 42th IEEE COMPSAC 2018, 24 July 2018, Tokyo 8

9. Examples of
preparation
phase
The 42th IEEE COMPSAC 2018, 24 July 2018, Tokyo 9
Merging
data errors
in select and
join creates
incidents

10. Example of data delivery phase
A problem in the network could prevent us to store
data, leading to missing data for analytics.
The 42th IEEE COMPSAC 2018, 24 July 2018, Tokyo 10

11. W3H: what, when,
where and how for
incidents
The 42th IEEE COMPSAC 2018, 24 July 2018, Tokyo 11
Where – Location of incidents
What – types of incidents
When – the incident happens
How – the incident happens
Phases and Stakeholders in Incident Analytics
 Address diverse types of Stakeholders
 Dependencies among incidents from different
subsystems and providers

12. Examples of incident propagation
The 42th IEEE COMPSAC 2018, 24 July 2018, Tokyo 12
Monitor alarms in BTSs:
What could be happened?
Acquisition incidents  Analysis incidents
But where did acquisition incidents occur?

13. Points of instrumentation
The 42th IEEE COMPSAC 2018, 24 July 2018, Tokyo 13
 Capture monitoring data from different subsystems and
at different layers
 Systems and data quality behavior
 Different ways of instrumentation and monitoring

14. Classification of incidents
The 42th IEEE COMPSAC 2018, 24 July 2018, Tokyo 14

15. Snapshot of incident classification
information model
One implementation in graph database: Neo4J.
The 42th IEEE COMPSAC 2018, 24 July 2018, Tokyo 15

16. Providing guidelines for development
of incident monitoring and analytics
The 42th IEEE COMPSAC 2018, 24 July 2018, Tokyo 16
Check: https://github.com/rdsea/bigdataincidentanalytics
A concrete example of implementation

17. Conclusions and future work
 Conclusions
 Incidents monitoring and analytics in IoT data analytics
are under researched and supported
 We present a framework for characterization and
classification of incidents in big IoT data analytics
 Future work
 Improve the classification and characteristics of incidents
 Develop monitoring and analysis techniques for capturing
and evaluating incidents.
 Results are available at
https://github.com/rdsea/bigdataincidentanalytics.
The 42th IEEE COMPSAC 2018, 24 July 2018, Tokyo 17

18. Thanks for your
attention!
Hong-Linh Truong
Faculty of Informatics
TU Wien, Austria
rdsea.github.io
The 42th IEEE COMPSAC 2018, 24 July 2018, Tokyo 18