Similar a Learning Engineering of MOOClets: Simultaneously benefiting Professional Learning, Financial Success, and Cognitive & Learning Sciences Research
2019 New Trends in Education -Teaching Innovation Timothy Wooi
Similar a Learning Engineering of MOOClets: Simultaneously benefiting Professional Learning, Financial Success, and Cognitive & Learning Sciences Research (20)
Learning Engineering of MOOClets: Simultaneously benefiting Professional Learning, Financial Success, and Cognitive & Learning Sciences Research
1. Learning Engineering of MOOClets: Simultaneously
benefiting Professional Learning, Financial Success,
and Cognitive & Learning Sciences Research
Joseph Jay Williams
Lytics Lab, Graduate School of Education
Office of the Vice Provost for Online Learning, Stanford
josephjaywilliams@stanford.edu
www.josephjaywilliams.com
1
[Originally from Trinidad & Tobago – apologies for difficulty in following accent.]
2. Topics
• I. Motivation
• II. Learning through
Questions/Assessments
• III. General Learning & Problem-
Solving Skills
• IV. Performance Support Tools &
Checklists
2
3. Guiding Principles
• Multiple Constraints: Practical, Scientific,
Financial
• Modularity in Product Design:
“MOOClets”
• Future-thinking but Actionable
• Empirical and Integrative Work
• Experiment-Focused Instructional Design
• Matchmaking Relevant Researchers
• Iterative Improvement
• Collaborative Development
3
4. I. Motivation
• Beliefs about Intelligence
• To what extent do you agree?
How smart you are can be significantly
changed by your effort and hard work.
You have a certain amount of intelligence,
and what you do doesn’t really change
that much.
• Growth Mindset vs. Fixed Mindset
• Dweck, 2008; Yeager & Walton, 2012
• Williams, Paunesku, Haley & Sohl-Dickstein, 2013
4
5. MOOClet 1: Exercise (Khan Academy)
5
1. Number of
Problems
Practiced
2. Accuracy
[TEXT SHOWN HERE – Randomize]
7. Growth Mindset Message helpful?
• Growth Mindset
Message
• "Remember, the more you practice
the smarter you become.”,
• "Mistakes help you learn. Think hard
to learn from them.”
• Practice-as-usual
Jascha Sohl-Dickstein
7
8. Results: More motivated?
• Growth Mindset Message > Practice-as-
Usual
• reduction in dropout rate
• increase in Number of problems
practiced
• increase in Number Correct
• Accuracy: Problems correct/Problems
attempted
• Accuracy improvement
8
9. Interpreting results
• Obvious?
• Skeptical? Anything’s statistically
significant with large N…
• Second experiment (run simultaneously)
9
10. Practice-as-usualGrowth Mindset MessagePositive Message
Different motivational messages
Some of these problems are hard. Do your
best!
Remember, the more you practice the smarter you become!
10
11. Does any positive message work?
• Growth Mindset
Message
• "Remember, the more you practice
the smarter you become.”,
• "Mistakes help you learn. Think hard
to learn from them.”
• Positive Message
• "Some of these problems are
hard. Just do your best."
• "This might be a tough problem,
but we know you can do it.”
• Practice-as-usual
11
12. Effects of Positive Messages?
• Positive Messages not better than Practice-as-
Usual!
• Growth Mindset > Positive Message
• How intuitive is this result?
• Teachers?
• Ongoing: After incorrect answer
• www.assistments.org Neil Heffernan
• Gates Grant for Modeling Research
12Rapid Experimentation: tiny.cc/mechanicalturk
13. Guiding Principles
• Multiple Constraints: Practical, Scientific,
Financial
• Modularity in Product Design:
“MOOClets”
• Future-thinking but Actionable
• Empirical and Integrative Work
• Experiment-Focused Instructional Design
• Matchmaking Relevant Researchers
• Iterative Improvement
• Collaborative Development
13
15. “MOOClets” or Modular Online Resources
• E.g. MOOClet: Math exercise, 5 minute
interactive video, 30 minute lesson, Study
Strategy coach, Quiz/Assessment
• www.josephjaywilliams.com/mooclets
• MOOClets Group Tue 12-1 PST (remote)
• tiny.cc/joinmooclets mailing list & updates
• Williams, J.J. & Williams, B. A. (2013). Randomized Experiments
as a Methodological and Conceptual Tool for Improving
Design of Online Learning. Data Driven Education, NIPS
Workshop. tiny.cc/experimentsonlinelearning
15
17. Where is relevant research, best practice, technology, funding,
market? tiny.cc/mapofspace, tiny.cc/stanfordmap
MOOCs
(Coursera, EdX, Udacity)
www.KhanAcademy.org
www.Assistments.org
K-12 Ed-Tech space
(Bay Area, Boston...)
(www.edsurge.com)
(www.edudemic.com )
Higher-Ed Technology/LMS
E-Learning & Training for
Corporate, State, Federal, Army
Psychological Scientists
Computer Scientists
Education
Corporate Training
Funding Agencies:
IES, Dept. of Education, Dept. of Labor,
NIH, NSF
Practical Reports:
What Works Clearinghouse
IES Practice Guides
University Teaching & Learning Centers
Department of Education reports
Think Tanks:
SRI International
WestEd
Pittsburgh Science of Learning Center
Learning Sciences
Mooc Research Initiative
Open Learning Initiative
Distance Learning
Athabasca, Western Governor’s University
Economics of Education
Habit & Behavior Change
Medical Education
Centers/Scholarship of
Teaching & Learning
Assessment
Pearson
ETS
Open Education Resources
Creative Commons
Foundations:
Gates, Spencer, Hewlett, Kresge
Venture Capital
Private VCs
New Schools
Impact Investing
Accelerators:
Imagine K-12,
HCI, Design Thinking
Workforce Development
17
18. Relevant Resources
• www.josephjaywilliams.com/home/useful-
software
• www.josephjaywilliams.com/education
• www.josephjaywilliams.com/online-education-
experiments
• tiny.cc/mapofspace
• Applying Cognitive Science to Online Learning.
Paper presented at NIPS Workshop on Data
Driven Education
(tiny.cc/cognitivescienceinmoocs)
• Improving Outcome of Psychosocial Treatments
by Enhancing Memory and Learning.
Perspectives in Psychological Science.
18
19. Guiding Principles
• Multiple Constraints: Practical, Scientific,
Financial
• Modularity in Product Design:
“MOOClets”
• Future-thinking but Actionable
• Empirical and Integrative Work
• Experiment-Focused Instructional Design
• Matchmaking Relevant Researchers
• Iterative Improvement
• Collaborative Development
19
20. Messages in Coursera?
20
Week 6 Quiz
Whether or not you’re correct on the first try, each attempt at a tough problem can
make you better at statistics.
22. Instructor Guide to foster Growth Mindset
• tiny.cc/idstrategymindset
22
Where do you think you will apply this?
Do you have any questions or want
clarification?
Click to see any of the Case Studies below
Would you like to?
23. Guiding Principles
• Multiple Constraints: Practical, Scientific,
Financial
• Modularity in Product Design:
“MOOClets”
• Future-thinking but Actionable
• Empirical and Integrative Work
• Experiment-Focused Instructional Design
• Matchmaking Relevant Researchers
• Iterative Improvement
• Collaborative Development
23
24. Topics
• I. Motivation
• II. Learning through
Questions/Assessments
• III. General Learning & Problem-
Solving Skills
• IV. Performance Support Tools &
Checklists
24
25. II. Learning through Questions/Assessments
• Discovery of patterns
(Williams & Lombrozo, 2010, Cognitive Science)
• Use of prior knowledge
(Williams & Lombrozo, 2013, Cognitive Psychology)
• Children’s causal learning
(Walker, Williams, Lombrozo, Gopnik, 2012)
• Overgeneralize at expense of specific facts
(Williams et al, 2013, Journal of Experimental Psychology: General)
• Contradictions & belief revision
(Williams, et al 2013; 2014; Cog Sci, CHI WiP)
• Comparison of examples
(Edwards, Williams, Lombrozo, Gentner, 2013)
GLORP
DRENT
Online/Blended
Educational
Resources
35. Guiding Principles
• Multiple Constraints: Practical, Scientific,
Financial
• Modularity in Product Design:
“MOOClets”
• Future-thinking but Actionable
• Empirical and Integrative Work
• Experiment-Focused Instructional Design
• Matchmaking Relevant Researchers
• Iterative Improvement
• Collaborative Development
35
36. Topics
• I. Motivation
• II. Learning through
Questions/Assessments
• III. General Learning & Problem-
Solving Skills
• IV. Performance Support Tools &
Checklists
36
41. Topics
• I. Motivation
• II. Learning through
Questions/Assessments
• III. General Learning & Problem-
Solving Skills
• IV. Performance Support Tools &
Checklists
41
42. Guiding Principles
• Multiple Constraints: Practical, Scientific,
Financial
• Modularity in Product Design:
“MOOClets”
• Future-thinking but Actionable
• Empirical and Integrative Work
• Experiment-Focused Instructional Design
• Matchmaking Relevant Researchers
• Iterative Improvement
• Collaborative Development
42
43. Acknowledgements
• Tania Lombrozo & Tom Griffiths
• John Mitchell & Candace Thille
• Jascha Sohl-Dickstein
• Jace Kohlmeier & Khan Academy
• Sam Maldonado
• Lytics Lab (lytics.stanford.edu)
• VPOL (Vice Provost for Online
Learning, online.stanford.edu)
43
Notas del editor
Originally from Trinidad. Thank you for having me here, it’s a pleasure because of how this entire movement has impacted me. Two years ago was finishing my dissertation work in cognitive science – psychology experiments on learning, & computational psychology, or using Bayesian statistics & machine learning to model human thinking, but I had been doing work in the education school, and wanted to have a real-world impact, and actually build products. Seemed impossible to do both, but all of a sudden – online education exploded!
Tremendously excited, started research in collaboration with Khan Academy, and have been trying to add value by bringing to bear scientific knowledge about learning in a way that’s actually practically useful, and linking researchers to these practical platforms, like writing grants with Khan Academy & Stanford Online. So I will present a subset of that work.
But it will be different from a typical research talk, because I took such an amazing group as a chance to share explicitly on the approach I take to developing online educational materials, point to resources people here might also find useful, and emphasize some broad themes in online learning where I think real-world impact and science converge.
Interested in discussion & feedback from you on these – you have thought about it a tremendous deal here at Coursera, and much more familiar with your and others’ platforms. Don’t hesitate to make suggestions or state different opinion, I’d like to learn more about your thinking on these issues.
Start with practical or “product-first”. Not necessarily a typical focus for researchers, except Candace Thille, Ken Koedinger, ?
50% is primary research, which I’ll present. 50% synthesizing existing work, having broad view of technology, entrepreneurship. I like collecting these list, and think there’s a lot of value to be tapped.
Take a business perspective – where is there unrealized value, because of lack of information?
So I’m going to discuss ongoing work with Khan Academy & MOOCs and residential education.
Atypical Trajectory, which means the talk has many moving pieces: Use digital online educational resources to do use-inspired basic research.
Extremely tough to link research and practice, so the last few years I’ve taken positions that allowed me to invest a lot of time into having a broad view of the emerging trends – understanding technology and goals of the MOOC platforms, Khan Academy, hundreds of Startups in Bay Area, as well as e-learning in the corporate training & workforce development world.
Also been trying to get a broad view of which research topics and themes are particularly well suited to these contexts, from cognitive psychology to educational topics, social & clinical, health behavior change. Methodologies from HCI, education policy.
Collaborative Development
Guided by Practical and Research Goals
ASSESSMENT/MEASUREMENT/COMPETENCY BASED?
o.32
0.27
Collaborative Development
Guided by Practical and Research Goals
ASSESSMENT/MEASUREMENT/COMPETENCY BASED?
Have an unusual background, so I thought it might help to provide some context.
Approach: Use digital educational resources to do inspired basic research (e.g. math exercises), understand how cognitive processes like explaining can guide learning & how people’s beliefs influence motivation. (Pasteur’s Quadrant).
Principles & Demonstrations for doing this kind of work laid by PSLC.
ust to provide some context about Use digital online An interdisciplinary approach to linking research and practice,PhD training is a mixture of Experimental Psychology, Computational Psychology (using methods from machine learning & Bayesian statistics to model cognition),
Exhausting… but exciting!
Criticized for not using literature… but don’t have time to.
Researchers haven’t looked at & thought enough about platforms, to synthesize the key ideas. Also rare to have the time to take a broad view.
Collaborative Development
Guided by Practical and Research Goals
ASSESSMENT/MEASUREMENT/COMPETENCY BASED?
Add textbox below first two questions for people to type into.
Add some hyperlink looking things below “Click … case studies” with something like these fields:
Encouraging students to see literary skills as malleable through class emails
Improving persistence in Database Course programming assignments
Enhancing receptiveness to peer feedback in management courses
Under would you like to, same kind of thing, but with:
Learn more about the research behind this concept
Ask a question or share thoughts with the Teach-Net mailing list
Request a future reminder for applying this guide
Collaborative Development
Guided by Practical and Research Goals
ASSESSMENT/MEASUREMENT/COMPETENCY BASED?
Not traditional, but ubiquitous
A challenging choice… Take the next theoretical step in this line of research, with a new set of materials.
On the other hand, this is venturing into new territory. PSLC & others been successful, they have full control.
Also, since there’s little direct past research in many of these contexts, don’t even know yet what form the phenomenon takes.
Start with a practical challenge first, seek to improve it by bringing to bear research. Establishing whether there is an effect, sets the stage for using this online context to probe its underlying mechanisms. Jump to the other side, use-inspired basic research.
Collaborative Development
Guided by Practical and Research Goals
ASSESSMENT/MEASUREMENT/COMPETENCY BASED?
Collaborative Development
Guided by Practical and Research Goals
ASSESSMENT/MEASUREMENT/COMPETENCY BASED?