In-app reflection guidance for workplace learning means motivating and guiding users to reflect on their working and learning, based on users’ activities captured by the app. In this paper, we present a generic concept for such in-app reflection guidance for workplace learning, its implementation in three different applications, and its evaluation in three different settings (one setting per app). From this experience, we draw the following lessons learned: First, the implemented in-app reflection guidance components are perceived as useful tools for reflective learning and their usefulness increases with higher usage rates. Second, smart technological support is sufficient to trigger reflection, however with different implemented components also reflective learning takes place on different stages. A sophisticated, unobtrusive integration in the working environment is not trivial at all. Automatically created prompts need a sensible timing in order to be perceived as useful and must not disrupt the current working processes.

1. b
b
© Know-Center GmbH, www.know-center.at
In-App Reflection Guidance for
Workplace Learning
Angela Fessl, Gudrun Wesiak, Verónica Rivera-Pelayo,
Sandra Feyertag, Viktoria Pammer
EC-TEL 2015, Toledo, 17.09.2015

2. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
Introduction
• In-App reflection guidance concept to trigger reflective
learning
• … at the workplace
2
Purpose of this work

3. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
Introduction
• Conscious re-evaluation of past situations/experiences
with the goal to learn from them & use the gained
outcomes to guide future behaviour
• Workplace learning: core process to get new insights,
derive better practices -> improve own work
• Cognitive process based on intrinsic & extrinsic
motivation of an individual
3
Reflective Learning

4. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
Background – Reflection Guidance
• RL Tools: Prompts, Reflection Amplifiers, Diaries, Journals, e-
Portfolios
• Prompting approaches successful in formal learning
environments
• Journals/Diaries in educational settings e.g. nursing, athletics
• Little research yet in work-related settings
• Challenges:
• Work tasks not always known beforehand/vaguely known
• Right time to display prompts?
• Working environment is often stressful & work-intensive
4

5. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
Reflection Guidance Concept
• Reflection interventions
• Prompts to motivate users to utilise an application in general
• At beginning/end of day, at the start of an event or when application
was not used for a longer time
• Reflection amplifiers
• Well-considered prompts  initiate reflection & support planning
and reflection
• Make users aware of unusual behaviour/working patterns or other
significant situations
• Motivate users to insert their thoughts into corresponding
application
5
Reflection-in-Action Components
(Verpoorten et al., 2010)

6. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
Reflection Guidance Concept
• Reports
• Automatic summary of experiences/events captured during work
• Hourly, daily, weekly
• Compare them with own or other‘s working behaviour  detect
patterns & insights  learning outcomes
• Reflection Diary
• Store insights, thoughts, artefacts, learning outcomes in
structured way
• Re-evaluation of entries  trigger reflective learning & lead to
insights guiding future behaviour
6
Reflection-on-Action Components

7. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
Reflection Guidance Concept
• Contextualisation
• Manually or automatically tracked by application
• Automatic tracking: extensive amounts of data
• Reflective learning support:
• User has to think about context information when entering it
• Context information when revisiting captured data helps
remembering the respective working situation better
7
Interwined Component

8. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
Methodology
User studies in real work settings (3 companies, 3 Apps
with reflection guidance components)
1. How were the implemented reflection guidance
components perceived with regard to their usefulness at
work?
2. To what extent (at which stage) did reflective learning
occur?
8
Research Questions

9. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
Methodology
• Log-Data  objective usage rates
• Pre-Questionnaire: demographic data
• Post-Questionnaire
• Support of apps with respect to RL
• Perceived usefulness of reflection guidance components
• Interviews & workshop: deeper information
9
Analysis Data

10. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
Methodology
• Analysis of written content: qualitative analysis schema
(independent analysis & categorisation by 3 researchers)
• 1. stage: descriptions of experiences & emotions
• 2. stage: interpretation & justification of actions and working on
solutions
• 3. stage: insights, learning & conclusions from reflection
10
Evaluations
(Prilla et al., 2014)

11. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
User Studies
• MoodMap App (Call Center)
• Insights on how mood influences work performance &
collaboration within a team
• Medical Quiz (Stroke Unit)
• Refresh existing knowledge & gain new knowledge
• Connect theoretical knowledge with working practices &
reflection
• KnowSelf (IT Company)
• Support of individual reflective learning regarding time
management & self-organisation of knowledge workers
11
Overview and Learning Goals

12. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
User Study: MoodMap App at Call Center
• Maps mood on a coloured map along 2 dimensions:
• Valence (feeling good – feeling bad)
• Arousal (high energy – low energy)
• Reflection Guidance:
• Contextualisation: For each mood choose pre-defined context &
add note (supported by reflection amplifier question, e.g. What
currently influences your mood?)
• Reports (mood development over day) & reflection diary
12
MoodMap App

13. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
User Study: MoodMap App at Call Center
13
MoodMap App

14. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
User Study: MoodMap App at Call Center
• Call Center:
• 4 teams of 2 call centers of a large British telecommunication
company
• Each team:
• Call takers (take calls & solve issues directly with customers)
• Coaches (support & coach call takers for their work)
• Managers (supervice training sessions, review call takers‘ performance
& ensure their performance against targets)
• 4 weeks
• All participants were asked to integrate MoodMap App in their
daily work routines
14
MoodMap App – Setting

15. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
User Study: Medical Quiz at Hospital
• Quiz especially developed for nurses at a Stroke Unit in
German hospitals
• 4 different quiz types
• Randomised content-based questions
• Reflection guidance: reflection amplifiers
• At the beginning:
• address current knowledge status and play frequency
• In-between:
• put user‘s focus on content-based questions
• At the end:
• ask for gained insights or new knowledge w.r.t. the played quiz
15
Medical Quiz

16. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
User Study: Medical Quiz at Hospital
16
Medical Quiz

17. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
User Study: Medical Quiz at Hospital
• Qualification program for becoming nurses working for a
Stroke Unit
• 5 months
• Education: 1 week per month
• Nurses were asked to play quiz consequently
• Strengthen newly gained knowledge
• Reflect about content & draw connections between new
knowledge and daily work practices
17
Medical Quiz – Setting & Evaluation Procedure

18. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
User Study: KnowSelf at IT Company
• Automatic tracking of resources & apps used on PC
• Manual project & task recording
• Reflection Guidance:
• Reflection diary for notes, thoughts & insights
• Time-triggered prompts
• Remind user of reflecting by offering feedback on different aspects
of user‘s behaviour
• Event-triggered prompts
• After detection of a significant change (e.g. unusually high number
of switches)
18
KnowSelf

19. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
User Study: KnowSelf at IT Company
19
KnowSelf

20. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
User Study: KnowSelf at IT Company
• Consulting Company
• Consults, sells & personalises CRM (Customer Relationship
Management) Software
• Reflection regarding interactions with customers
• Installation on business PCs
• Reflect on collected data on daily basis over 6 weeks
• Weekly reminder
• Reflect on captured data
• Enter observations
• Fill in short questionnaire
20
KnowSelf - Setting

21. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
Results
• MoodMap App: 33 participants
• Interaction: M=42.67 (SD=72.2) from max. 338 to min 1
• Reflective Learning: M=3.25 (SD = .92)
• Medical Quiz: 18 participants
• Questions answered: M = 461.9 (SD=341), from max 1358 to min. 25
• Reflective Learning: M = 3.51 (SD=.42)
• KnowSelf: 10 participants
• Usages: M=38.5 min (SD 23.8), from max. 53.72min to min. 7.76min
• Reflective Learning: M = 3.28 (SD =.66)
• Sufficient app usage to evaluate reflection guidance components
• Components perceived as useful and reflective learning has taken place
• Positive correlations between extent of app usage & users‘ perceived support
for reflective learning (significant for Medical Quiz & KnowSelf)
21
Did participants use apps, and did they actually reflect?

22. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
Results: Usage and Usefulness of Reflection
Guidance
• Context distribution:
• After a call: 31%
• After a break: 6%
• After a coaching session: 2%
• Other: 61%
• How helpful was the contextualisation for
• Associating inserted moods to certain situations: M=3.39 (SD=.86)
• Recalling past experiences: M=3.21 (SD=.74)
• Helping to better reflect about past situations: M=3.24 (SD=.94)
• Context should be adaptable to individual preferences
22
Contextualisation – MoodMap App

23. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
Results: Usage and Usefulness of RGC
• Time-triggered prompts
• Reminder to use app: M=3.22 (SD=1.09)
• Reminder of project recording: M=2.75 (SD=1.28)
23
Reflection Interventions - KnowSelf

24. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
Results: Usage and Usefulness of RGC
• MoodMap App: Add note to moods
• 475 notes inserted
• Medical Quiz: reflective questions
• 1205 posed, 603 (52%) meaningful answered
• 59.3% at beginning, 51.5% at end, 42.4% in-between
• Willingness to reflect is higher at beginning and end of the quiz than inbetween (disruptive)
• KnowSelf: event-triggered prompts
• Notification about most used resources: M=3.30 (SD=1.25)
• Notification about unusual amount of idle time: M=2.57 (SD=0.79)
• Notification about specific amount of switches: M=2.30 (SD=1.34)
• Asking the right questions at the right time can trigger reflective learning
24
Reflection Amplifiers: MoodMap App, KnowSelf, Medical Quiz

25. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
Results: Content Analysis of inserted Notes
• MMA: 457 notes to 548 inserted moods: 283 individual reflective
items
• 95% 1.stage (descriptions of experiences & emotions)
• 5% 2.stage (interpretations/justifications of actions & working on solutions)
• KnowSelf: 103 statements  33 non-reflective, 11 no inter-coder
agreement, 59 reflective items
• 22% 1.stage (descriptions of experiences & emotions)
• 47% 2.stage (interpretations/justifications of actions & working on solutions)
• 27% 3.stage (insights, learning outcomes, conclusions from reflection)
• Medical Quiz: Answers to open questions were too short for analysis
(often only a single word)
25
Stages of reflection

26. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
Conclusion
1. How were the implemented reflection guidance components
perceived with regard to their usefulness at work?
• Components perceived as useful
• In-app reflection guidance facilitates technology-supported reflective learning
at work
2. To what extent (at which stage) did reflective learning occur?
• Achieved reflection stages differed between the apps and setting
• MoodMap App:
• 1. stage (experience) & 2. stage (interpretation)
• KnowSelf
• 1. stage (experience) & 2. stage (interpretation) & 3. stage (insights)
26
Research Questions

27. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
Conclusion
• Correct timing of reflection interventions & amplifiers is crucial ->
avoid disturbing interruptions
• Interruptibility: ongoing research challenge
• Give users control over notification
27
Lessons Learned

28. © Know-Center GmbH
gefördert durch das Programm COMET (Competence Centers for Excellent Technologies), wir danken unseren Fördergebern:
bQUESTIONS
Angela Fessl
afessl@know-center.at
Know-Center GmbH
Inffeldgasse 13
8010 Graz
28

29. © Know-Center GmbH • Research Center for Data-Driven Business and Big Data Analytics
References
• Prilla, M., Renner, B.: Supporting collaborative reflection at work: a
comparative case analysis. In: Proceedings of ACM Conference on Group
Work (GROUP 2014). ACM (2014)
• Schön, D.A.: The Reflective Practitioner: How Professionals Think In Action,
1st edn. Basic Books, New York (1984)
• Verpoorten, D., Westera, W., Specht, M.: Reflection amplifiers in online
courses: a classification framework. J. Interact. Learn. Res. 21(4), 654–666
(2010).
29