1) Joseph Jay Williams conducts research using experiments and cognitive science to improve the design of online learning resources.
2) His work focuses on explaining concepts, teaching learning strategies, adding motivational messages, developing experimental paradigms, and designing experiments into online courses.
3) Some findings include that explaining "why" helps learning more than simply stating facts, teaching self-questioning strategies like "What? Why? How?" improves understanding, and embedding growth mindset messages increases student motivation more than positive messages alone.
MS4 level being good citizen -imperative- (1) (1).pdf
Using Experiments and Cognitive Science to Improve Online Learning Design
1. Using Experiments and Cognitive Science
Research to Improve the Design of Online
Resources for Learning
Joseph Jay Williams
josephjaywilliams@stanford.edu
www.josephjaywilliams.com/researchoverview
1
2. Online Education & Learning Online
• New research area?
• Convergence of
computational &
behavioral science
NIPS “Data-Driven
Education”
New ACM conference
“Learning at Scale”
CHI
6. Why does explaining “why?” help learning?
• General boost to Learning Engagement vs.
• The Subsumptive Constraints Account:
Interpret target of why-explanation in terms of a
broader generalization (Williams & Lombrozo, 2010)
•
•
2x3=6
Why?
6
7. Explanation and Learning: Lab Online
• Discovery & transfer
GLORP
(Williams & Lombrozo, 2010, Cognitive
Science)
• Use of prior knowledge
(Williams & Lombrozo, 2013,
Cog. Psych.)
• Erroneously overgeneralize at
expense of exceptions
• Promotes belief revision – given
sufficient anomalies
(Williams et al, 2013, JEP: General)
(Williams, Walker, Maldonado &
Lombrozo, 2012; 2013, Cog Sci Conference; in prep)
• Prompts in online (math) exercises
(Williams, Paunesku, Haley, Sohl-Dickstein, 2013, AIED Moocshop; ongoing)
DRENT
8. Learning Task & Experimental Paradigm
• Online (Math) Exercise
1. Number of
Problems
Completed
2. Percent
Correct
8
9. Logic of my Previous Research
Explain why this is
correct.
Elaborate on
what you are
thinking now.
•
•
•
•
Post-Study test questions
Transfer/Generalization
questions
Questions about key principle
Memory for details
9
10. Future: Generate, Receive, Compare
Generate
E.g. Explain
why this is
correct.
Receive
Compare
Geza Kovacs
Simultaneously Learning AND Crowdsourcing
Improvement of Learning Resources
Williams, Thille, Siemens, Trumbore, Stigler. How online resources can
facilitate interdisciplinary collaboration. Invited talk to be presented
at SIG on Computer and Internet Applications in Education, AERA
10
2014.
11. Teaching Learning Strategies
• Spend time teaching specific content, or
general strategies?
• Online: Collect data that would be extremely
difficult to get in the real world
• Online: Repeatedly reinforce habits &
educational behaviors
• Teach “What? Why? How?” selfquestioning/explanation strategies (Palinscar &
Brown, 1984, Cognition and instruction; McNamara, 2004, Discourse Processes; Williams &
Lombrozo, 2010, Cognitive Science)
• Understanding vs. Problem-solving
• vs. Interpreting vs. Practice-as-Usual
11
12. Experimentally manipulate additional prompts
Clickable link. + Prompts embedded into hints
[Click here to learn about the “What? Why? How?” strategy]
15. Prompts
Prompt
type
Promptedunderstanding
What?
What does this step
mean to you?
Why?
Why is what you are
currently doing
Why is it helpful to
helpful? Why is it
take this step?
useful for achieving
your goal?
How?
How well is your
current approach to
this problem
working?
How do you know
this step is right?
Promptedproblem-solving
Promptedinterpretation
What are you doing What is this step
or thinking right saying? Restate it in
now?
your own words.
18. Learning Behavior Support
• Clickable link to Drop-Down text with
suite of strategies:
Are you stuck?
Click here for some tips.
• Provide previously examined prompts.
• Use mouseover and drop-down text
to reveal information “as requested”,
rich traversal of options, guided by
student
19. Natural Link to Learning Strategy Training
• If you want to learn more about
strategies to keep motivated and
learn well, go to
tiny.cc/learningassistant or XX or YY
20. 3. Add motivational messages
Practice-as-usual
Growth Mindset Message
Remember, the more you practice the smarter you become!
20
21. 3. Embedded in vivo Experiment
• Benefit of Growth Mindset Message?
• Practice-as-usual
Jascha Sohl-Dickstein
• Growth Mindset
Message
•
•
"Remember, the more you practice
the smarter you become.”,
"Mistakes help you learn. Think hard
to learn from them.”
21
22. Results: More motivated?
• Growth Mindset Message > Practice-asUsual
• extra problems attempted
• more problems correct
• Percent Correct: Problems
correct/Problems attempted
• increase in Percent Correct
22
23. 3. Add motivational messages
Practice-as-usual Message
Growth Mindset Message
Positive
Remember, the more you practice the smarter you become!
Some of these problems are hard. Do your
best!
23
24. Does any positive message work?
• Practice-as-usual
• Growth Mindset
Message
•
•
"Remember, the more you practice
the smarter you become.”,
"Mistakes help you learn. Think hard
to learn from them.”
• Positive Message
•
•
"Some of these problems are
hard. Just do your best."
"This might be a tough problem,
but we know you can do it.”
24
25. Effects of Positive Messages?
• Positive Messages ~= Practice-asUsual
• Growth Mindset > Positive
• extra problems attempted
• more problems correct
• increase in Percent Correct
25
26. Computational Modeling
•
•
•
•
•
•
Williams, Mitchell, Heffernan. MOOC Research Initiative grant from
Gates Foundation & Athabasca. Investigating the benefits of
embedding motivational messages in online exercises.
2 million users on 12 kinds of fractions exercises, ~100 problems each
Moderators & Mediators
Item Response Theory
Non-parametric Bayesian clustering of Users (CrossCat, JMLR)
Model latent knowledge states
26
27. Synthesize Scientific Findings
•
Williams, J.J. (2013)Improving Learning in MOOCs
by Applying Cognitive Science. Paper presented
at the MOOCshop Workshop, International
Conference on Artificial Intelligence in Education,
Memphis, TN.
• www.josephjaywilliams.com/education
27
28. Experimental Paradigm: R.E.P.E.A.T.
•
•
Williams, J. J. (2013). Finding connections between
basic experimental research and realistic online
education contexts. In J. J. Williams (chair), Online
Learning and Psychological Science: Opportunities
to integrate research and practice. Symposium
conducted at the annual convention of the
Association for Psychological Science.
Williams, J. J., Renkl, A., Koedinger, K., Stamper, J.
(2013). Online Education: A Unique Opportunity for
Cognitive Scientists to Integrate Research and
Practice. In M. Knauff, M. Pauen, N. Sebanz, & I.
Wachsmuth (Eds.), Proceedings of the 35th Annual
Conference of the Cognitive Science Society.
28
Austin, TX: Cognitive Science Society. (pdf)
29. Experiments
•
•
Williams, J.J. & Williams, B.A. (under review). Online
A/B Tests & Experiments: A Practical But
Scientifically Informed Introduction. Course
proposal submitted to ACM CHI Conference on
Human Factors in Computing Systems. Toronto,
Canada. (pdf)
Williams, J.J., Heffernan, N., & Koedinger, K.
Experiments at Scale: Instrumenting MOOCs for
experimentation and course-improving data
analysis. Tutorial proposal submitted to the First
Annual ACM Conference on Learning at Scale.
29
30. Experiment-Focused Design
• Williams, J.J. & Williams, B. A. (2013).
Using Interventions to Improve Online
Learning. Paper to be presented at
the NIPS 2013 Workshop on Data
Driven Education.
30
31. Review
•
•
•
•
•
•
Explanation & Learning
Teaching Learning Strategies
Motivational Messages
Experimental Paradigm
Experiment-focused Design
Williams, J.J., Klemmer, S., Kizilcec, R., &
Russel, D. (under review). Learning
Innovations at Scale. Workshop proposal
submitted to ACM CHI Conference on
Human Factors in Computing
Systems. Toronto, Canada.
31
Originally from Trinidad. I’m a Research Fellow in theLytics Lab & Office of Online Learning. Cognitive Science background – experimental psychology, statistical modeling and machine learning.Core interest is being a knowledge broker – reviewing and synthesizing the thousands of published studies in cognitive science, education research, the learning sciences to see which ones are relevant to real-world outcomes, and building products or conducting experiments that improve practically ad financially valuable outcomes. Illustrate this approach with an experiment that shows how to improve students’ motivation while learning from mathematics problems on Khan Academy.