AERA talk on Nudge, a Computer support for studying at the right times. Goes through design process and system evaluation. Some slides with animation will be confusing or missing pieces because of Slideshare's importer.

1. +
1
Nudge
Computer support for
studying at the right times
Aleahmad, Koedinger and
Zimmerman

2. + 2
Usability
 Research often difficult to translate into practice
 Need usable knowledge (Lagemann, 2002)
 Tension between internal and ecological validity
 Modern computing lets us do controlled lab-style
research in real-world settings
 Lower the costs of deploying research in broad contexts
to understand contextual factors
 Usable systems
AERA 2012 — Aleahmad, Koedinger and Zimmerman

3. + 5
Process
Evaluation: Evaluation:
Informed
Enactment Local Broader
Exploration
Impact Impact
Integrative Learning Design Framework (Bannan-Ritland 2003)
AERA 2012 — Aleahmad, Koedinger and Zimmerman

4. + 6
Informed Exploration
Used HCI methods for usability:
 Focus setting  College lecture courses
 Contextual inquiry  Interviews with stakeholders
 Ideation  ~60 ideas for systems
 Scenario development  17 that express needs
 Needs validation  Interview stakeholders to check
 Example observation for Nudge need:
“by second semester freshman year I was trying to learn how to
study, pretty much teaching myself.”
AERA 2012 — Aleahmad, Koedinger and Zimmerman

5. + 7
Nudge Scenario
AERA 2012 — Aleahmad, Koedinger and Zimmerman

6. + 8
Needs Validation
 IES practice guides to see what practice
recommendations could be operationalized usably in
college lecture courses
 E.g. “help students allocate study time efficiently” from
Organizing Instruction and Study to Improve Student Learning
(Pashler et al., 2007) principle #6
 Ruled out all but a few systems because either
1. not interesting theoretically
2. required too much change by stakeholders
3. rejected by the culture
 e.g. study leaderboard. students didn't want competition.
AERA 2012 — Aleahmad, Koedinger and Zimmerman

7. + 9
Principles of Nudge Design
Feature Claim Warrant
Course task assigned Explicit and salient External deadlines boost task performance
to dates and dates more likely to more than self-determined deadlines (Ariely &
organized centrally be met Wertenbroch, 2002);
Students generally do whatever’s due soonest
(Kornell & Bjork, 2007)
Break-down of study Decomposition of Smaller tasks abate the planning fallacy (Forsyth
activity into smaller tasks improves time & Burt, 2008; Kruger & Evans, 2003);
actions allocation and Students procrastinate largely due to fear of
decreases failure (Solomon & Rothblum, 1984);
aversiveness In shared task lists, vague information preferred
(Blandford, 2001)
Maintaining and Recording task Self-monitoring of study behaviors improves
tracking status increases learning (Richards, 1975)
assignments, study awareness and
time and progress inclination
AERA 2012 — Aleahmad, Koedinger and Zimmerman

8. + 10
Iterative Development: Mockup
C✓ | Home
http:/ / coursecheck.org/ Google
Joe Student Dashboard Settings Log out
CourseCheck Thu Jan 13
Introduction to Modern Chemistry Calculus in 3D English 101 Pottery
Prof. Paul Karol 01-234
Full course
Past due
Mastery Exam 1 Ch2: exercises 21, 23, 30, 31, 37, 38 Jan 11
Mastery Exam 1 Ch2: exercises 55, 63 Jan 13
Coming week
Minutes spent Save
Notes to self
Notes to instructor
Ch3: exercises 23, 25, 27, 31, 37, 41 Jan 13
for Quiz 1
Ch3: exercises 47, 55 Jan 18
Ch4: exercises 17, 19 Jan 18
AERA 2012 — Aleahmad, Koedinger and Zimmerman

9. +

10. + 12
Final Iteration for Evaluation
AERA 2012 — Aleahmad, Koedinger and Zimmerman

11. + 13
Study
 Large lecture-based intro chemistry class
 15 week fall semester
 60 tasks defined for the semester
 (14 required, 43 advised, 3 suppl.)
 Do and submit HW-n (required)
 Take a practice text for Exam-n (advised)
 Review notes for Exam IV (advised)
AERA 2012 — Aleahmad, Koedinger and Zimmerman

12. + 14
Conditions
Introductory Chemistry Course
N=136
All nudges polled on tasks before due (n=45)
Nudge?
No nudges, tasks batched after exam to poll (n=48)
Everyone else gets message before due, at a
schedule they choose. (n=42)
AERA 2012 — Aleahmad, Koedinger and Zimmerman

13. + 15
Hypotheses
H-grades: Student sent all Nudge messages perform better
on exams than students sent no Nudge messages.
H-disposition: Students with poor study time use benefit
more from Nudge messages.
 Time/Environment scale from Motivated Strategies for
Learning Questionnaire (Pintrich et al. 2001)
AERA 2012 — Aleahmad, Koedinger and Zimmerman

14. + 16
Evaluation of Operation
H-grade: No main effects…
H-disposition: Better time
management led to better exam
scores (F(1,76.9)=6.4, p=.014) but
Nudge interacts to help students
with poor management
(F(1,76.9)=4.6, p=.036).
Nudge helps students with poor management
…and hinders with good management?
AERA 2012 — Aleahmad, Koedinger and Zimmerman

15. + 17
Mechanism
The worse a student’s time use the greater
the benefit of opening each Nudge
message.
AERA 2012 — Aleahmad, Koedinger and Zimmerman

16. + 18
Broader Impact
 75% of nudged respondents (n=28) rated “Email reminders
about course work” as “Good” or “Great”
 40% agreed, “The reminder emails helped me in the class”
 46% agreed, “I wish I could have email reminders for all my
classes”
 44% agreed, “Without the reminders I would have forgotten
to do something.”
 80% of students who could stop the message didn’t
Group Replied to task polls Agreement with “What I enter is accurate”.
ever (7pt Likert)
No nudges 83% (40/48) 5.5 (n=26, sd=1.4)
All nudges 87% (39/45) 5.8 (n=29, sd=1.3)
AERA 2012 — Aleahmad, Koedinger and Zimmerman

17. + 19
Discussion
 Need validation ruled out systems I thought would work;
saved resources
 Indications that it is easy to adopt and would be used
voluntarily by students and teachers
 Need to test adoption empirically
 Usability focus needs more methods for validation
 Examplify
 Future work
AERA 2012 — Aleahmad, Koedinger and Zimmerman

18. + 20
Thanks for your attention!
@tfunq
Turadg Aleahmad
http://coursecheck.org
http://turadg.aleahmad.net
http://openeducationresearch.org
AERA 2012 — Aleahmad, Koedinger and Zimmerman