Despite the increasing automation levels in an Industry 4.0 scenario, the tacit knowledge of highly skilled manufacturing workers remains of strategic importance. Retaining this knowledge by formally capturing it is a challenge for industrial organisations. This paper explores research on automatically capturing this knowledge by using methods from activity recognition and process mining on data obtained from sensorised workers and environments. Activity recognition lifts the abstraction level of sensor data to recognizable activities and process mining methods discover models of process executions. We classify the existing work, which largely neglects the possibility of applying process mining, and derive a taxonomy that identifies challenges and research gaps.

1. A TAXONOMY FOR COMBINING
ACTIVITY RECOGNITION
AND
PROCESS DISCOVERY
IN INDUSTRIAL ENVIRONMENTS
Felix Mannhardt, Riccardo Bovo, Manuel Fradinho Oliveira, Simon Julier
1SINTEF, Trondheim, Norway , 2University College London, UK
This research has received funding from the European
Unions H2020 research and innovation programme under grant
agreement no. 723737 (HUMAN).

2. What is Process Mining?
2
• Activity (What?)
• Ordering (When?)
• Case / Context
• SAP, MES, …
• Sensors (IoT)
• …
• Assembly
• Logistics
• …
• Work instructions
• Assembly sequence
• …

3. 3
Operators
Sensor
data
discover
Process
Models
capture
Activities
recognise
analyse
tacit knowledge
deliver explicit knowledge
Process
Engineer
Process Discovery in Industrial Environments

4. What is Activity Recognition (ARC)?
4 Some input sensor signal segmented according to the ground truth label

5. 5
Example: Supervised detection using a LSTM

6. 6
Some further typical examples
Ekaterina H. Spriggs*, Fernando De La Torre, and Martial Hebert, "Temporal segmentation and activity
classification from first-person sensing," IEEE Workshop on Egocentric Vision, CVPR 2009, June, 2009.
Michael S. Ryoo Greg Mori Kris Kitani.
CVPR1 Tutorial: Paper 18: Human Activity Recognition. 2018

7. Project Context – HUMAN
• Human MANufacturing Project (H2020)
• Sensorised manufacturing operator (Wearables etc.)
• Stress and fatigue detection through sensor data
• Support operators in their workflow through technology
• Augmented reality (AR)
• Exoskeletons
• Use of Process Mining on collected data for
long-term diagnostics and optimisation of work
7
Challenge: Automatically detect manual shop-floor activities
to re-discover the actual work process

8. Exploratory search for existing work
8
Google Scholar Scopus
Search terms
• Manufacturing
• Industrial
• Activity recognition
• Sensors
• Process discovery
• Process elicitation
• Process analysis
• Events
• Tacit knowledge

9. Categorized 26 relevant papers
9

10. Preliminary taxonomy for ARC & Process discovery
10
Time
Predictive
Online
Post-mortem

11. 11
Knoch, S., Ponpathirkoottam, S., Fettke, P., Loos, P.:
Technology-enhanced process elicitation of sworker activities in manufacturing.
In: Business Process Management Workshops,
pp. 273–284. Springer International Publishing (2018)
Stiefmeier, T., Roggen, D., Ogris, G., Lukowicz, P., Tr, G.:
Wearable activity tracking in car manufacturing.
IEEE Pervasive Computing 7(2), 42–50 (2008)
Ambient vs. Wearable | Video vs. Motion

12. Preliminary taxonomy for ARC & Process discovery
12
Data
Capture
Sensor type
Vision-based
Motion-based
Sound-based
Radiowave-based
Sensor location
Wearable
Objects
Ambient
Storage
Batch
Streaming
Processing
Supervised
Unsupervised

13. Very little unsupervised learning
13
Boettcher, S., Scholl, P.M., Laerhoven, K.V.:
Detecting process transitions from wearable sensors.
In: iWOAR 2017. ACM Press (2017)

14. Some semi-supervised works
14
Blanke, U., Schiele, B.:
Remember and transfer what you have learned - recognizing composite
activities based on activity spotting.
In: ISWC 2010. IEEE (2010)

15. Preliminary taxonomy for ARC & Process discovery
15
Processcontext
Activity type
Granularity
Coarse
Fine
Complexity
Simple
Complex
Control-flow type
Sequence of steps
Sequential
workflow
Collaborative
workflow
Process size
Small
Large

16. 16
Stiefmeier, T., Roggen, D., Ogris, G., Lukowicz, P., Tr, G.:
Wearable activity tracking in car manufacturing.
IEEE Pervasive Computing 7(2), 42–50 (2008)
Complexity & Control-Flow
Blanke, U., Schiele, B.:
Remember and transfer what you have learned - recognizing composite
activities based on activity spotting.
In: ISWC 2010. IEEE (2010)

17. Preliminary taxonomy for ARC & Process discovery
17
Environment
Setting
Laboratory
Factory
Layout
Flexible
Structured
RigidInterference
Privacy

18. 18
Voulodimos, A.S., Kosmopoulos, D.I., Doulamis, N.D., Varvarigou, T.A.:
A top-down event driven approach for concurrent activity recognition.
Multimedia Tools and Applications
69(2), 293–311 (2012)
Environment
Knoch, S., Ponpathirkoottam, S., Fettke, P., Loos, P.:
Technology-enhanced process elicitation of sworker activities in manufacturing.
In: Business Process Management Workshops,
pp. 273–284. Springer International Publishing (2018)

19. Conclusion
• Contribution
• A structured overview of the work on activity recognition
relevant to process discovery in industrial environments
• Preliminary taxonomy derived from our literature search
• Challenges identified for process discovery in industrial environments
• Future work
• Rigerous description of the finalised taxonomy
• Comprehensive literature review based on the completed taxonomy
• Validation of the taxonomy?
• Apply process discovery paired with activity recognition in real industrial environments
19

20. Mannhardt F., Bovo R., Oliveira M.F., Julier S. (2018)
A Taxonomy for Combining Activity Recognition and Process Discovery in Industrial Environments.
In: Yin H., Camacho D., Novais P., Tallón-Ballesteros A. (eds)
Intelligent Data Engineering and Automated Learning – IDEAL 2018. IDEAL 2018.
Lecture Notes in Computer Science, vol 11315. Springer, Cham