1. © Know-Center GmbH, www.know-center.at
Reflection Guidance
Angela Fessl
STATE-OF-THE-ART AND CHALLENGES IN INTELLIGENT
MENTORING SYSTEMS
JTELSS 2019, Bari, Italy, 3rd June 2010
1

2. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
THINK ABOUT YESTERDAY‘S EXCURSION ….
2
• What was your highlight? And why?
• What could be improved?
Grotte di Castellana Alberobello Polignano a Mare

3. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
REFLECTION GUIDANCE
3

4. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
DATA-DRIVEN AND REFLECTIVE LEARNING
• Problem setting
• Design for lifelong learning from experience
• Approach
• Use data as basis for learning from experience via adaptive
learning support
• Expected results
• Socio-technical concepts that combine working and learning to
improve work performance on all levels

5. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
REFLECTIVE LEARNING …
5
… is the re-evaluation of past experiences with the goal to
learn from them to drive future behaviour.

6. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
TECHNOLOGIES: PROMPTS
6
Data-Driven and Reflective Learning
• Reflection Interventions vs. Reflection amplifiers
• Generic prompts vs. Directed prompts
• Prompts in conversation (ChatBots) Prompts
Prompts
Misc.
Diaries
Visuals
(Kocielniketal.,2018)
(Davis, 2000)

7. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
TECHNOLOGIES: PROMPTS
7
Data-Driven and Reflective Learning
• Advantages:
• Well investigated in learning environments, less investigated in
work-place settings
• If reflective learning is induced depends on the type of prompt
• Disadvantages:
• Timing: when should they be presented in order to be perceived as
useful and not as disruptive?
• How: How should they pre presented? Silent notification vs.
Popup?
Prompts
Misc.
Diaries
Visuals

8. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
TECHNOLOGIES: DIARIES/JOURNALS/E-
PORTFOLIOS
8
Data-Driven and Reflective Learning
• Diaries: personal notes not to be shared
• Journals/ePortfolios: shared with
trainer/teacher/colleagues
Prompts
Misc.
Diaries
Visuals
From: http://ultranet.stdoms.ac.nz/WebSpace/4865/

9. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
TECHNOLOGIES: DIARIES/JOURNALS/E-
PORTFOLIOS
9
Characters
Timeline
Scribble
Bluetooth devices
Incoming/
Outcoming SMS
Photos
(Ståhl, et al. 2009)

10. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
TECHNOLOGIES: DIARIES/JOURNALS/E-
PORTFOLIOS
10
Data-Driven and Reflective Learning
• Advantages
• Works well and lead to reflective learning
• Disadvantages
• Difficulties in writing
• Time consuming in keeping and maintaining
• Well consideration necessary, where to introduce
Prompts
Misc.
Diaries
Visuals

11. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
TECHNOLOGIES: VISUALS
11
Data-Driven and Reflective Learning
• Smart Indicators, videos, images, charts, …Prompts
Misc.
Diaries
Visuals
(Glahn et al., 2009)
(Malacria et al., 2013)
(Fessl et al., 2019)

12. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
TECHNOLOGIES: VISUALS
12
Data-Driven and Reflective Learning
• Advantages
• Easy to understand
• Motivation through comparison via gamification
• Fotos and videos, easy to interpret
• Disadvantages
• Photos, videos -> GDPR
• It needs to be well considered what to present worth being
reflected on
Prompts
Misc.
Diaries
Visuals

13. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
TECHNOLOGIES: MISCELLANEOUS
13
Data-Driven and Reflective Learning
Prompts
Misc.
Diaries
Visuals (Prilla et al., 2013)

14. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
TECHNOLOGIES: MISCELLANEOUS
14
Data-Driven and Reflective Learning
• Advantages
• Everything that might be worth being reflected on can be used and
presented
• Disadvantages
• Decision of what is worth being reflected upon?
Prompts
Misc.
Diaries
Visuals

15. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
IN-APP REFLECTION GUIDANCE CONCEPT
15
Reflection-in-action Reflection-on-action
How often?At what time? Which type?
Pop-up automatically Self-regulated reflection
Reflection
Intervention
Reflection
Amplifier
Application usage during work
Usage
Reminder
Awareness
of team
usage
Awareness
of individ.
data
Awareness
of team
data
Guidance
with Context
Reports and
Feedback
Reflection
Journal/Diary
Ask for
context
Statistical
information
Significant
Patterns
Insert
outcomes
or insights
Reflect on
entries
Context
information
Intertwined
Provide
options for
context

16. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
DATA-DRIVEN AND REFLECTIVE LEARNING
16
Motivation & Challenges
„What is in for me“?
• Timing for reflection: When?
• Context of reflection: In which?
• Topic/data as object for reflection: Which?
• Motivation for reflection: Why?

17. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
DATA
17
Data-Driven and Reflective Learning
• Can be anything and is not limited to
• … mood data
• … work activities
• … search activities
• … sports/fitness data
• … knowledge
• …
• … your data?

18. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
Reflect about your…
Work
Activities
Knowledge
Mood
Search
Behaviour
Learning
Progress
… working activities
to become an expert
in time management
and self-organisation
… knowledge to
become an expert in
your domain.
… mood and your
team‘s mood to
become aware of how
emotions affect our
work.
… search behaviour to
become an effective and
efficient search expert.
… learning progress to
become expert in
information literacy and
digital competences.
Demo
Demo
Demo

19. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
DEMOS ….
19
• Medical Quiz
• https://www.know-center.tugraz.at/en/showcase/medical-quiz-2/
• MOVING
• https://moving.mz.tu-dresden.de/

20. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
INDIVIDUAL ASSIGNMENT
20
• How does this relate to your PhD?
• Reflective Learning Technologies …
• Data-driven reflection…
• Adaptive/context aware systems…
• Intelligent mentoring/tutoring…
• User models/learner models …
• Note down your ideas
• Present and discuss it afterwards with the groups

21. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
TIME MANAGEMENT ….
21
• Participate in the Study on „TimeManagement“
• http://tiny.cc/timeTool
What‘s in it for you? Reflect on your time management – and if you
have no need for that, do share your secret with everyone this
week.

22. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
References
• (Brusilovsky & Peylo, 2003) Brusilovsky, P. & Peylo, C. (2003) Adaptive and Intelligent Web-based Educational Systems. International Journal of Artificial Intelligence in
Education (IJAIED), 2003, 13, pp.159-172. hal-00197315.
• (Brusilovsky & Millán, 2007) Brusilovsky, P., and Millán, E. User models for adaptive hypermedia and adaptive educational systems. In The Adaptive Web, P. Brusilovsky,
A. Kobsa, and W. Nejdl, Eds. Springer-Verlag, Berlin, Heidelberg, 2007, ch. User Models for Adaptive Hypermedia and Adaptive Educational Systems, 3–5.
• (Brusilovsky, 2012) Brusilovsky, P. Adaptive hypermedia for education and training. In Adaptive Technologies for Training and Education, P. J. Durlach and A. M. Lesgold,
Eds. Cambridge University Press, 2012, 46–66. Cambridge Books Online.(Dimitrova et al., 2016) Dimitrova, V., Poulovassilis, A., Van Labeke, N., Treasure-Jones, T.,
Zukas, M., & Brna, P. (2016). Intelligent Mentoring Systems for Making Meaning from Work Experience. In Proceedings of First International Workshop on Intelligent
Mentoring Systems (IMS 2016). Leeds.
• (Dimitrova et al., 2016) Dimitrova, V., Poulovassilis, A., Van Labeke, N., Treasure-Jones, T., Zukas, M., & Brna, P. (2016). Intelligent Mentoring Systems for Making
Meaning from Work Experience. In Proceedings of First International Workshop on Intelligent Mentoring Systems (IMS 2016). Leeds.
• (Dimitrova & Brna, 2016) Dimitrova, V., & Brna, P. (2016). From Interactive Open Learner Modelling to Intelligent Mentoring: STyLE-OLM and Beyond. International
Journal of Artificial Intelligence in Education, 26(1), 332-349.
• (Fessl et al., 2017a) Fessl, A., Blunk, O., Prilla, M., & Pammer, V. (2017). The known universe of reflection guidance: a literature review. International journal of
technology enhanced learning, 9(2-3), 103-125.
• (Fessl et al., 2017b) Fessl, A., Wesiak, G., Rivera-Pelayo, V., Feyertag, S., & Pammer, V. (2017). In-app reflection guidance: lessons learned across four field trials at the
workplace. IEEE Transactions on Learning Technologies, 10(4), 488-501.
• (Kocielnik et al., 2018) Kocielnik, R., Xiao, L., Avrahami, D., & Hsieh, G. (2018). Reflection Companion: A Conversational System for Engaging Users in Reflection on
Physical Activity. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 2(2), 70.
• (Davis, 2000) Davis, E. A. (2000). Scaffolding students' knowledge integration: Prompts for reflection in KIE. International Journal of Science Education, 22(8), 819-837.
• (Lindström et al., 2006) Lindström, M., Ståhl, A., Höök, K., Sundström, P., Laaksolathi, J., Combetto, M., ... & Bresin, R. (2006, April). Affective diary: designing for bodily
expressiveness and self-reflection. In CHI'06 extended abstracts on Human factors in computing systems (pp. 1037-1042). ACM.
• (Ståhl, et al. 2009) Ståhl, A., Höök, K., Svensson, M., Taylor, A. S., & Combetto, M. (2009). Experiencing the affective diary. Personal and Ubiquitous Computing, 13(5),
365-378.
• (Glahn et al., 2009) Glahn, C., Specht, M., & Koper, R. (2009). Visualisation of interaction footprints for engagement in online communities. Educational Technology &
Society, 12 (3), 44–57.
• (Malacria et al., 2013) Malacria, S., Scarr, J., Cockburn, A., Gutwin, C., & Grossman, T. (2013, October). Skillometers: reflective widgets that motivate and help users to
improve performance. In Proceedings of the 26th annual ACM symposium on User interface software and technology (pp. 321-330). ACM.
• (Prilla et al., 2013) Prilla, M., Nolte, A., Blunk, O., Liedtke, D., & Renner, B. (2015). Analyzing collaborative reflection support: a content analysis approach. In ECSCW
2015: Proceedings of the 14th European Conference on Computer Supported Cooperative Work, 19-23 September 2015, Oslo, Norway (pp. 123-142). Springer, Cham.
22

23. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
References
• (Fessl et al., 2017a) Fessl, A., Blunk, O., Prilla, M., & Pammer, V. (2017). The known universe of reflection guidance: a
literature review. International journal of technology enhanced learning, 9(2-3), 103-125.
• (Fessl et al., 2017b) Fessl, A., Wesiak, G., Rivera-Pelayo, V., Feyertag, S., & Pammer, V. (2017). In-app reflection guidance:
lessons learned across four field trials at the workplace. IEEE Transactions on Learning Technologies, 10(4), 488-501.
• (Rivera-Pelayo et al., 2017) Rivera-Pelayo, V., Fessl, A., Müller, L., & Pammer, V. (2017). Introducing mood self-tracking at
work: Empirical insights from call centers. ACM Transactions on Computer-Human Interaction (TOCHI), 24(1), 3.
• (Kocielnik et al., 2018) Kocielnik, R., Xiao, L., Avrahami, D., & Hsieh, G. (2018). Reflection Companion: A Conversational
System for Engaging Users in Reflection on Physical Activity. Proceedings of the ACM on Interactive, Mobile, Wearable and
Ubiquitous Technologies, 2(2), 70.
• (Davis, 2000) Davis, E. A. (2000). Scaffolding students' knowledge integration: Prompts for reflection in KIE. International
Journal of Science Education, 22(8), 819-837.
• (Lindström et al., 2006) Lindström, M., Ståhl, A., Höök, K., Sundström, P., Laaksolathi, J., Combetto, M., ... & Bresin, R.
(2006, April). Affective diary: designing for bodily expressiveness and self-reflection. In CHI'06 extended abstracts on Human
factors in computing systems (pp. 1037-1042). ACM.
• (Ståhl, et al. 2009) Ståhl, A., Höök, K., Svensson, M., Taylor, A. S., & Combetto, M. (2009). Experiencing the affective diary.
Personal and Ubiquitous Computing, 13(5), 365-378.
• (Glahn et al., 2009) Glahn, C., Specht, M., & Koper, R. (2009). Visualisation of interaction footprints for engagement in online
communities. Educational Technology & Society, 12 (3), 44–57.
• (Malacria et al., 2013) Malacria, S., Scarr, J., Cockburn, A., Gutwin, C., & Grossman, T. (2013, October). Skillometers:
reflective widgets that motivate and help users to improve performance. In Proceedings of the 26th annual ACM symposium
on User interface software and technology (pp. 321-330). ACM.
• (Prilla et al., 2013) Prilla, M., Nolte, A., Blunk, O., Liedtke, D., & Renner, B. (2015). Analyzing collaborative reflection support:
a content analysis approach. In ECSCW 2015: Proceedings of the 14th European Conference on Computer Supported
Cooperative Work, 19-23 September 2015, Oslo, Norway (pp. 123-142). Springer, Cham.
23

24. © Know-Center GmbH
Know-Center GmbH
Research Center for Data-Driven
Business and Big Data Analytics
Inffeldgasse 13/6
8010 Graz, Austria
Firmenbuchgericht Graz
FN 199 685 f
UID: ATU 50367703
gefördert durch das Programm COMET (Competence Centers for Excellent Technologies), wir danken unseren Fördergebern:
Researcher
afessl@know-center.at
Angela Fessl
Head of Data-Driven Busniess
vpammer@know-center.at
Viktoria Pammer-Schindler
24