1) Joseph Jay Williams conducts research using experiments and cognitive science to improve the design of online learning resources. 2) His work focuses on explaining concepts, teaching learning strategies, adding motivational messages, developing experimental paradigms, and designing experiments into online courses. 3) Some findings include that explaining "why" helps learning more than simply stating facts, teaching self-questioning strategies like "What? Why? How?" improves understanding, and embedding growth mindset messages increases student motivation more than positive messages alone.

1. Using Experiments and Cognitive Science
Research to Improve the Design of Online
Resources for Learning
Joseph Jay Williams
josephjaywilliams@stanford.edu
www.josephjaywilliams.com/researchoverview
1

2. Online Education & Learning Online
• New research area?
• Convergence of
computational &
behavioral science
NIPS “Data-Driven
Education”
New ACM conference
“Learning at Scale”
CHI

3. Novel Research Opportunity: Real-World +
Laboratory

4. Integrate Research & Practice
• Randomized assignment
• Experimental Control
• Rich data
• Generalizable theories
• “in vivo” experiments
• Diverse populations
• Real-world
environment
• Authentic activities
• Practical Challenges
• Practical improvements
• Disseminate research
• Generate Funding

5. Overview
•
•
•
•
•
Explanation & Learning
Teaching Learning Strategies
Motivational Messages
Experimental Paradigm
Experiment-focused Design
5

6. Why does explaining “why?” help learning?
• General boost to Learning Engagement vs.
• The Subsumptive Constraints Account:
Interpret target of why-explanation in terms of a
broader generalization (Williams & Lombrozo, 2010)
•
•
2x3=6
Why?
6

7. Explanation and Learning: Lab  Online
• Discovery & transfer
GLORP
(Williams & Lombrozo, 2010, Cognitive
Science)
• Use of prior knowledge
(Williams & Lombrozo, 2013,
Cog. Psych.)
• Erroneously overgeneralize at
expense of exceptions
• Promotes belief revision – given
sufficient anomalies
(Williams et al, 2013, JEP: General)
(Williams, Walker, Maldonado &
Lombrozo, 2012; 2013, Cog Sci Conference; in prep)
• Prompts in online (math) exercises
(Williams, Paunesku, Haley, Sohl-Dickstein, 2013, AIED Moocshop; ongoing)
DRENT

8. Learning Task & Experimental Paradigm
• Online (Math) Exercise
1. Number of
Problems
Completed
2. Percent
Correct
8

9. Logic of my Previous Research
Explain why this is
correct.
Elaborate on
what you are
thinking now.
•
•
•
•
Post-Study test questions
Transfer/Generalization
questions
Questions about key principle
Memory for details
9

10. Future: Generate, Receive, Compare
Generate
E.g. Explain
why this is
correct.
Receive
Compare
Geza Kovacs
Simultaneously Learning AND Crowdsourcing
Improvement of Learning Resources
Williams, Thille, Siemens, Trumbore, Stigler. How online resources can
facilitate interdisciplinary collaboration. Invited talk to be presented
at SIG on Computer and Internet Applications in Education, AERA
10
2014.

11. Teaching Learning Strategies
• Spend time teaching specific content, or
general strategies?
• Online: Collect data that would be extremely
difficult to get in the real world
• Online: Repeatedly reinforce habits &
educational behaviors
• Teach “What? Why? How?” selfquestioning/explanation strategies (Palinscar &
Brown, 1984, Cognition and instruction; McNamara, 2004, Discourse Processes; Williams &
Lombrozo, 2010, Cognitive Science)
• Understanding vs. Problem-solving
• vs. Interpreting vs. Practice-as-Usual
11

12. Experimentally manipulate additional prompts
Clickable link. + Prompts embedded into hints
[Click here to learn about the “What? Why? How?” strategy]

13. Self-questioning strategy: What? Why? How?

14. Embedded Prompts between Hint/Solution
Steps

15. Prompts
Prompt
type
Promptedunderstanding
What?
What does this step
mean to you?
Why?
Why is what you are
currently doing
Why is it helpful to
helpful? Why is it
take this step?
useful for achieving
your goal?
How?
How well is your
current approach to
this problem
working?
How do you know
this step is right?
Promptedproblem-solving
Promptedinterpretation
What are you doing What is this step
or thinking right saying? Restate it in
now?
your own words.

16. Explanation study
Practice-asusual
Control
Self-Regulating
Thinking
Reflecting on
Meaning
Problems
Attempted
Problems
Correct
Practice Tasks
Completed
Mastery Tasks
Completed
Problem
Accuracy
16

17. Explanation study
Practice-as-usual
No Textbox
Textbox
Problems Attempted
Problems Correct
Practice Tasks
Completed
Mastery Tasks
Completed
Problem Accuracy
17

18. Learning Behavior Support
• Clickable link to Drop-Down text with
suite of strategies:
Are you stuck?
Click here for some tips.
• Provide previously examined prompts.
• Use mouseover and drop-down text
to reveal information “as requested”,
rich traversal of options, guided by
student

19. Natural Link to Learning Strategy Training
• If you want to learn more about
strategies to keep motivated and
learn well, go to
tiny.cc/learningassistant or XX or YY

20. 3. Add motivational messages
Practice-as-usual
Growth Mindset Message
Remember, the more you practice the smarter you become!
20

21. 3. Embedded in vivo Experiment
• Benefit of Growth Mindset Message?
• Practice-as-usual
Jascha Sohl-Dickstein
• Growth Mindset
Message
•
•
"Remember, the more you practice
the smarter you become.”,
"Mistakes help you learn. Think hard
to learn from them.”
21

22. Results: More motivated?
• Growth Mindset Message > Practice-asUsual
• extra problems attempted
• more problems correct
• Percent Correct: Problems
correct/Problems attempted
• increase in Percent Correct
22

23. 3. Add motivational messages
Practice-as-usual Message
Growth Mindset Message
Positive
Remember, the more you practice the smarter you become!
Some of these problems are hard. Do your
best!
23

24. Does any positive message work?
• Practice-as-usual
• Growth Mindset
Message
•
•
"Remember, the more you practice
the smarter you become.”,
"Mistakes help you learn. Think hard
to learn from them.”
• Positive Message
•
•
"Some of these problems are
hard. Just do your best."
"This might be a tough problem,
but we know you can do it.”
24

25. Effects of Positive Messages?
• Positive Messages ~= Practice-asUsual
• Growth Mindset > Positive
• extra problems attempted
• more problems correct
• increase in Percent Correct
25

26. Computational Modeling
•
•
•
•
•
•
Williams, Mitchell, Heffernan. MOOC Research Initiative grant from
Gates Foundation & Athabasca. Investigating the benefits of
embedding motivational messages in online exercises.
2 million users on 12 kinds of fractions exercises, ~100 problems each
Moderators & Mediators
Item Response Theory
Non-parametric Bayesian clustering of Users (CrossCat, JMLR)
Model latent knowledge states
26

27. Synthesize Scientific Findings
•
Williams, J.J. (2013)Improving Learning in MOOCs
by Applying Cognitive Science. Paper presented
at the MOOCshop Workshop, International
Conference on Artificial Intelligence in Education,
Memphis, TN.
• www.josephjaywilliams.com/education
27

28. Experimental Paradigm: R.E.P.E.A.T.
•
•
Williams, J. J. (2013). Finding connections between
basic experimental research and realistic online
education contexts. In J. J. Williams (chair), Online
Learning and Psychological Science: Opportunities
to integrate research and practice. Symposium
conducted at the annual convention of the
Association for Psychological Science.
Williams, J. J., Renkl, A., Koedinger, K., Stamper, J.
(2013). Online Education: A Unique Opportunity for
Cognitive Scientists to Integrate Research and
Practice. In M. Knauff, M. Pauen, N. Sebanz, & I.
Wachsmuth (Eds.), Proceedings of the 35th Annual
Conference of the Cognitive Science Society.
28
Austin, TX: Cognitive Science Society. (pdf)

29. Experiments
•
•
Williams, J.J. & Williams, B.A. (under review). Online
A/B Tests & Experiments: A Practical But
Scientifically Informed Introduction. Course
proposal submitted to ACM CHI Conference on
Human Factors in Computing Systems. Toronto,
Canada. (pdf)
Williams, J.J., Heffernan, N., & Koedinger, K.
Experiments at Scale: Instrumenting MOOCs for
experimentation and course-improving data
analysis. Tutorial proposal submitted to the First
Annual ACM Conference on Learning at Scale.
29

30. Experiment-Focused Design
• Williams, J.J. & Williams, B. A. (2013).
Using Interventions to Improve Online
Learning. Paper to be presented at
the NIPS 2013 Workshop on Data
Driven Education.
30

31. Review
•
•
•
•
•
•
Explanation & Learning
Teaching Learning Strategies
Motivational Messages
Experimental Paradigm
Experiment-focused Design
Williams, J.J., Klemmer, S., Kizilcec, R., &
Russel, D. (under review). Learning
Innovations at Scale. Workshop proposal
submitted to ACM CHI Conference on
Human Factors in Computing
Systems. Toronto, Canada.
31

32. Acknowledgements
•
•
•
•
•
Jascha Sohl-Dickstein
Jace Kohlmeier & Khan Academy
Sam Maldonado
Lytics Lab (lytics.stanford.edu)
VPOL (Vice Provost of Online Learning,
online.stanford.edu)
32