SlideShare una empresa de Scribd logo
1 de 29
Descargar para leer sin conexión
How online educational resources provide novel
affordances for conducting practical interventions
and doing psychology experiments
www.josephjaywilliams.com/experiments-online-education

Joseph Jay Williams, Stanford University
Lytics Lab (lytics.stanford.edu), Graduate School of Education
Office of the Vice Provost of Online Learning (vpol.stanford.edu)
josephjaywilliams@stanford.edu

www.josephjaywilliams.com

Originally from Trinidad – apologies for thick accent!
1
Overview
•
•
•
•
•
•

MOOCs VS Digital Online Resources
Experiments: Link Real World & Laboratory
Motivational Messages Experiment
Simultaneous Basic & Applied Research
Ed-Surge Meetup to link Research & Practice
(Wed Feb 19 @ 7)
Current Directions
Teaching Learning Strategies
Learning Coach
MOOC Experiment

•

MOOClets Research & Practice Group
2
Online Education
Moving from Physical to Digital World
Experiments: Link Real World & Laboratory

• Randomized assignment
• Experimental Control
• Rich data

• Real-world
environment
• Authentic activities
• Practical Challenges
Experiment
• Practical goal on Khan Academy
• Extend motivation research

6
Motivate students to practice math
• Online (Math) Exercise
Motivational Message: Keep up the good work!

1. Number of
Problems
Practiced
2. Improve
Accuracy

7
Add motivational messages
Practice-as-usual
Growth Mindset Message
Remember, the more you practice the smarter you become!

8
Which messages to add?
•
•
•
•
•
•

"This might be a tough problem, but we know
you can do it!”
Remember, the more you practice the smarter
you become!
Some of these problems are hard. Just do your
best.”
Encourage “Growth Mindset” beliefs about
intelligence (Dweck, 2008)
Fixed Mindset
Growth Mindset
9
Embedded in vivo Experiment
Jascha Sohl-Dickstein

• Practice-as-usual
• Growth Mindset
Message
•
•

"Remember, the more you practice
the smarter you become.”,
"Mistakes help you learn. Think hard
to learn from them.”

10
Results: More motivated?
• Growth Mindset Message > Practice-asUsual
• reduction in dropout rate
• increase in Number of problems
practiced
• increase in Number Correct
• Accuracy: Problems correct/Problems
attempted
• Accuracy improvement
11
Interpreting results
• Obvious?
• Skepticism: Anything’s statistically
signficant in a MOOC

12
Different motivational messages
Practice-as-usual Message
Growth Mindset Message
Positive
Remember, the more you practice the smarter you become!
Some of these problems are hard. Do your
best!

13
Does any positive message work?

• Practice-as-usual
• Growth Mindset
Message
•
•

"Remember, the more you practice
the smarter you become.”,
"Mistakes help you learn. Think hard
to learn from them.”

• Positive Message
•
•

"Some of these problems are
hard. Just do your best."
"This might be a tough problem,
but we know you can do it.”

14
Effects of Positive Messages?
• Positive Messages not better than
Practice-as-Usual!
• Growth Mindset better than Positive
• reduction in dropout rate
• increase in Number of problems
practiced
• increase in Number Correct

• Accuracy improvement
15
Next: Situationally appropriate messages

• Message blast, can be ignored
• Providing motivational message Justin-Time: after incorrect answer
• Laboratory experiments
• www.assistments.org Neil Heffernan

16
Simultaneous Applied & Basic Research
• Start with “Product”
• Constraints

Randomization (Technical, use Qualtrics)
Behavioral/Learning Measures

Williams & Williams (2013). Using Interventions to improve Online Learning. NIPS
Data Driven Education workshop. tiny.cc/experimentsonlinelearning

• Consider Synthesis

Williams, J.J. (2013) Improving Learning in MOOCs by Applying Cognitive
Science. AIED MOOCshop & NIPS Data-Driven Workshop.
tiny.cc/cognitivescienceinmoocs

Reading List: www.josephjaywilliams.com/education

Improving outcome of psychosocial treatments by enhancing memory and
learning.(in press) Perspectives in Psychological Science.

• High Investment but High Return

17
Multidisciplinary – Relevant Communities
•

•
•
•

•

Association for Psychological Science 2013. Online Learning and Psychological
Science: Opportunities to integrate research and practice. Williams, Saxberg,
Means, Mitros.
Cognitive Science Society 2013. Online Education: A Unique Opportunity for
Cognitive Scientists to Integrate Research and Practice. Williams, Renkl,
Koedinger, Stamper.
AERA 2014– American Education Research Association. How online resources can
facilitate interdisciplinary collaboration. Williams, Thille, Trumbore, Stigler, Siemens.
ICLS (under review) – International Conference for the Learning Sciences. Online &
Blended Education: Opportunities and Obstacles for the Learning Sciences.
Williams, Linn, Slotta, Rose, Dillenbourg.
CHI 2014 – Computer Human Interaction. Learning Innovations at Scale. Williams,
Kizilcec, Russell, Klemmer.

•
•
•

E-learning guild’s DevLearn
Educational Data Mining
Artificial Intelligence in Education

•
•
•

Computer Supported Collaborative Learning
Professional & Organizational Development Network
American Society for Training & Development
Laboratory Paradigms & Online Products
• A matching problem
• www.edsurge.com,
www.edudemic.com
• josephjaywilliams@stanford.edu
•

Williams, J.J., Teachman, B.A., Richland, L., Brady, S.T, & Aleahmad,
T. (2014). Leveraging the Internet to do Laboratory Research in the
Real World. Symposium to be conducted at the annual convention
of the Association for Psychological Science. San Francisco, CA.

19
Example: Explanation & Learning
•

Williams & Lombrozo (2010, Cognitive Science; 2013,
Cognitive Psychology; w Rehder, JEP: General)
Glorp

Right

Drent

Wrong

20
EdSurge Meetup Wed Feb 19 7 pm CERAS

• “Improving Ed-Tech using Research on
Learning & Advancing Research on
Learning using Ed-Tech”
• Ed-Tech Startup Survey
• Researcher Survey
• Featured Company & Researcher
• Panelist & Question suggestions?

21
Completely New Horizons for Experimentation

• “Bridging Studies”
• www.josephjaywilliams.com/experime
nts-online-education
•

Williams, J. J. (2013). Enhancing Educational Research &
Practice using Experiments on Online Educational
Resources. Talk at Pittsburgh Science of Learning Center
LearnLab Summer School, Pittsburgh, PA.
tiny.cc/onlineexperiments

22
Next: Teaching Learning Strategies
• Limited teaching time: Specific content
or general strategies?
• Online Advantage:
Easier to run RCTs & collect longitudinal data
Reinforce new educational habits & behaviors

• Teach metacognitive self-questioning
strategies
(Palinscar & Brown, 1984, Cognition and instruction; McNamara, 2004, Discourse Processes; Williams &
Lombrozo, 2010, Cognitive Science)

23
Drop-down text about strategies
Clickable link.
[Click here to learn about the “What? Why? How?” strategy]
Learning Coach
• Tiny.cc/kalearningcoach

25
MOOC experiment
• Motivational Video
Control
Energetic Thank you & Encouragement
Growth Mindset (by Instructor)
Growth Mindset (outside Instructor)
“Belonging”

• Questions for reflection after video
None VS Just Think VS Write response

26
MOOClets Group – Develop Digital Modules
•

•
•
•
•
•
•
•
•

E.g. MOOClet: Math exercise, 5 minute interactive video, 30
minute lesson, Study Strategy coach, Quiz/Assessment
Instructors can use for blended classroom education (High
School, Community College, University)
Continuously in use and being improved
Easy to build, prototype, test & iterate
Researchers directly implement & collect data
Small scale & online access facilitates multidisciplinary
collaboration & research-practitioner development (see
AERA 2014 invited panel)
Illustrate/Operationalize instructional design principles
MOOClet Reading & Discussion Group meets Tue 11-12 PST @
Barnum (Stanford) or tiny.cc/lyticshangout
Multidisciplinary group of researchers & practitioners
(tiny.cc/mooclet)
27
Review
•
•
•
•
•

•

•

MOOCs VS Digital Online Resources
Experiments: Link Real World & Laboratory
Motivational Messages Experiment
Simultaneous Basic & Applied Research
Ed-Surge Meetup to link Research & Practice (Wed
Feb 19 @ 7)
Current Directions
Teaching Learning Strategies
Learning Coach
MOOC Experiment

MOOClets Research & Practice Group Meets
Tuesdays 11-12 PST (tiny.cc/lyticshangout,
tiny.cc/mooclet)
28
Acknowledgements
•
•
•
•
•
•

Jascha Sohl-Dickstein
Jace Kohlmeier & Khan Academy
PERTS
Sam Maldonado
Lytics Lab (lytics.stanford.edu)
VPOL (Vice Provost of Online Learning,
online.stanford.edu)
29

Más contenido relacionado

La actualidad más candente

Keynote address at Western University Open Education Day
Keynote address at Western University Open Education DayKeynote address at Western University Open Education Day
Keynote address at Western University Open Education DayRajiv Jhangiani
 
21st Century Leadership and Learning
21st Century Leadership and Learning 21st Century Leadership and Learning
21st Century Leadership and Learning Vicki Davis
 
Tips For the Virtual Shift of F2f Math Classes
Tips For the Virtual Shift of F2f Math ClassesTips For the Virtual Shift of F2f Math Classes
Tips For the Virtual Shift of F2f Math ClassesFred Feldon
 
Total learning: The Intersection of Formal, Social and Experiential Learning
Total learning:  The Intersection of Formal, Social and Experiential LearningTotal learning:  The Intersection of Formal, Social and Experiential Learning
Total learning: The Intersection of Formal, Social and Experiential LearningJohn Leh
 
OpenEd - The Right Resource for Every Standard and Every Student
OpenEd - The Right Resource for Every Standard and Every StudentOpenEd - The Right Resource for Every Standard and Every Student
OpenEd - The Right Resource for Every Standard and Every StudentAdam Blum
 
50 (+ plus!) Ways to Improve your Classroom with Technology
50 (+ plus!) Ways to Improve your Classroom with Technology50 (+ plus!) Ways to Improve your Classroom with Technology
50 (+ plus!) Ways to Improve your Classroom with TechnologyVicki Davis
 
Community Colleges of Spokane: Open Ed Landscape, Open Course Design, and I...
Community Colleges of Spokane: Open Ed Landscape, Open Course Design, and I...Community Colleges of Spokane: Open Ed Landscape, Open Course Design, and I...
Community Colleges of Spokane: Open Ed Landscape, Open Course Design, and I...Ronda Dorsey
 
Use Technology to Augment your Already Awesome Classroom: Using a Home Base (...
Use Technology to Augment your Already Awesome Classroom: Using a Home Base (...Use Technology to Augment your Already Awesome Classroom: Using a Home Base (...
Use Technology to Augment your Already Awesome Classroom: Using a Home Base (...Jason Neiffer
 
OpenEd - The Right Resource for Each Standard and Student
OpenEd - The Right Resource for Each Standard and StudentOpenEd - The Right Resource for Each Standard and Student
OpenEd - The Right Resource for Each Standard and StudentAdam Blum
 
Bring your own idea - Visual learning analytics
Bring your own idea - Visual learning analyticsBring your own idea - Visual learning analytics
Bring your own idea - Visual learning analyticsJoris Klerkx
 
Coping with COVID-19 Webinar
Coping with COVID-19 WebinarCoping with COVID-19 Webinar
Coping with COVID-19 WebinarJohn Allan
 
Tips and Tricks for Teaching Math Online
Tips and Tricks for Teaching Math OnlineTips and Tricks for Teaching Math Online
Tips and Tricks for Teaching Math OnlineFred Feldon
 
Authenticity Online-Teaching, Learning and Assessment
Authenticity Online-Teaching, Learning and AssessmentAuthenticity Online-Teaching, Learning and Assessment
Authenticity Online-Teaching, Learning and AssessmentFred Feldon
 
Nudging students towards effective study behaviours using Brightspace data
Nudging students towards effective study behaviours using Brightspace dataNudging students towards effective study behaviours using Brightspace data
Nudging students towards effective study behaviours using Brightspace dataD2L Barry
 
Increase Engagement and Authentic Assessment in Online Classes
Increase Engagement and Authentic Assessment in Online ClassesIncrease Engagement and Authentic Assessment in Online Classes
Increase Engagement and Authentic Assessment in Online ClassesFred Feldon
 
Learning design and learning analytics
Learning design and learning analyticsLearning design and learning analytics
Learning design and learning analyticsRebecca Ferguson
 
Dollar-Fifty-a-Pair Socks Come in a 3-Pack
Dollar-Fifty-a-Pair Socks Come in a 3-PackDollar-Fifty-a-Pair Socks Come in a 3-Pack
Dollar-Fifty-a-Pair Socks Come in a 3-PackFred Feldon
 

La actualidad más candente (20)

Keynote address at Western University Open Education Day
Keynote address at Western University Open Education DayKeynote address at Western University Open Education Day
Keynote address at Western University Open Education Day
 
Denver Chadron
Denver ChadronDenver Chadron
Denver Chadron
 
Behind Their Eyes v1
Behind Their Eyes v1Behind Their Eyes v1
Behind Their Eyes v1
 
21st Century Leadership and Learning
21st Century Leadership and Learning 21st Century Leadership and Learning
21st Century Leadership and Learning
 
Tips For the Virtual Shift of F2f Math Classes
Tips For the Virtual Shift of F2f Math ClassesTips For the Virtual Shift of F2f Math Classes
Tips For the Virtual Shift of F2f Math Classes
 
Total learning: The Intersection of Formal, Social and Experiential Learning
Total learning:  The Intersection of Formal, Social and Experiential LearningTotal learning:  The Intersection of Formal, Social and Experiential Learning
Total learning: The Intersection of Formal, Social and Experiential Learning
 
OpenEd - The Right Resource for Every Standard and Every Student
OpenEd - The Right Resource for Every Standard and Every StudentOpenEd - The Right Resource for Every Standard and Every Student
OpenEd - The Right Resource for Every Standard and Every Student
 
50 (+ plus!) Ways to Improve your Classroom with Technology
50 (+ plus!) Ways to Improve your Classroom with Technology50 (+ plus!) Ways to Improve your Classroom with Technology
50 (+ plus!) Ways to Improve your Classroom with Technology
 
Community Colleges of Spokane: Open Ed Landscape, Open Course Design, and I...
Community Colleges of Spokane: Open Ed Landscape, Open Course Design, and I...Community Colleges of Spokane: Open Ed Landscape, Open Course Design, and I...
Community Colleges of Spokane: Open Ed Landscape, Open Course Design, and I...
 
Use Technology to Augment your Already Awesome Classroom: Using a Home Base (...
Use Technology to Augment your Already Awesome Classroom: Using a Home Base (...Use Technology to Augment your Already Awesome Classroom: Using a Home Base (...
Use Technology to Augment your Already Awesome Classroom: Using a Home Base (...
 
OpenEd - The Right Resource for Each Standard and Student
OpenEd - The Right Resource for Each Standard and StudentOpenEd - The Right Resource for Each Standard and Student
OpenEd - The Right Resource for Each Standard and Student
 
Bring your own idea - Visual learning analytics
Bring your own idea - Visual learning analyticsBring your own idea - Visual learning analytics
Bring your own idea - Visual learning analytics
 
Coping with COVID-19 Webinar
Coping with COVID-19 WebinarCoping with COVID-19 Webinar
Coping with COVID-19 Webinar
 
Web quest clip 1
Web quest clip 1Web quest clip 1
Web quest clip 1
 
Tips and Tricks for Teaching Math Online
Tips and Tricks for Teaching Math OnlineTips and Tricks for Teaching Math Online
Tips and Tricks for Teaching Math Online
 
Authenticity Online-Teaching, Learning and Assessment
Authenticity Online-Teaching, Learning and AssessmentAuthenticity Online-Teaching, Learning and Assessment
Authenticity Online-Teaching, Learning and Assessment
 
Nudging students towards effective study behaviours using Brightspace data
Nudging students towards effective study behaviours using Brightspace dataNudging students towards effective study behaviours using Brightspace data
Nudging students towards effective study behaviours using Brightspace data
 
Increase Engagement and Authentic Assessment in Online Classes
Increase Engagement and Authentic Assessment in Online ClassesIncrease Engagement and Authentic Assessment in Online Classes
Increase Engagement and Authentic Assessment in Online Classes
 
Learning design and learning analytics
Learning design and learning analyticsLearning design and learning analytics
Learning design and learning analytics
 
Dollar-Fifty-a-Pair Socks Come in a 3-Pack
Dollar-Fifty-a-Pair Socks Come in a 3-PackDollar-Fifty-a-Pair Socks Come in a 3-Pack
Dollar-Fifty-a-Pair Socks Come in a 3-Pack
 

Destacado

Experiments
ExperimentsExperiments
Experimentsvautoud
 
Intro to Psych
Intro to PsychIntro to Psych
Intro to PsychMr.J
 
Psychology experiments
Psychology experimentsPsychology experiments
Psychology experimentsDave Appleby
 
Bradford 213 lecture 3 four famous social psychology experiments
Bradford 213 lecture 3 four famous social psychology experimentsBradford 213 lecture 3 four famous social psychology experiments
Bradford 213 lecture 3 four famous social psychology experimentsJohn Bradford
 
Behavioural psychology
Behavioural psychologyBehavioural psychology
Behavioural psychologySLewis31
 
Psych 101 - Introduction to Psychology - Lecture 1
Psych 101 - Introduction to Psychology - Lecture 1Psych 101 - Introduction to Psychology - Lecture 1
Psych 101 - Introduction to Psychology - Lecture 1WhatisPsychology
 
Introduction to general psychology
Introduction to general psychologyIntroduction to general psychology
Introduction to general psychologyShaikh Mosin
 

Destacado (7)

Experiments
ExperimentsExperiments
Experiments
 
Intro to Psych
Intro to PsychIntro to Psych
Intro to Psych
 
Psychology experiments
Psychology experimentsPsychology experiments
Psychology experiments
 
Bradford 213 lecture 3 four famous social psychology experiments
Bradford 213 lecture 3 four famous social psychology experimentsBradford 213 lecture 3 four famous social psychology experiments
Bradford 213 lecture 3 four famous social psychology experiments
 
Behavioural psychology
Behavioural psychologyBehavioural psychology
Behavioural psychology
 
Psych 101 - Introduction to Psychology - Lecture 1
Psych 101 - Introduction to Psychology - Lecture 1Psych 101 - Introduction to Psychology - Lecture 1
Psych 101 - Introduction to Psychology - Lecture 1
 
Introduction to general psychology
Introduction to general psychologyIntroduction to general psychology
Introduction to general psychology
 

Similar a How online educational resources provide novel affordances for conducting practical interventions and doing psychology experiments

Learning Engineering of MOOClets: Simultaneously benefiting Professional Lear...
Learning Engineering of MOOClets: Simultaneously benefiting Professional Lear...Learning Engineering of MOOClets: Simultaneously benefiting Professional Lear...
Learning Engineering of MOOClets: Simultaneously benefiting Professional Lear...Joseph Jay Williams
 
Introduction to Personal Digital Inquiry in Grades K-8
Introduction to Personal Digital Inquiry in Grades K-8Introduction to Personal Digital Inquiry in Grades K-8
Introduction to Personal Digital Inquiry in Grades K-8Julie Coiro
 
Where Does It Live: Building Systems and Structures Around What You Believe
Where Does It Live: Building Systems and Structures Around What You BelieveWhere Does It Live: Building Systems and Structures Around What You Believe
Where Does It Live: Building Systems and Structures Around What You BelieveChris Lehmann
 
Winston Salem Administrators Workshop - 7.28.09
Winston Salem Administrators Workshop - 7.28.09Winston Salem Administrators Workshop - 7.28.09
Winston Salem Administrators Workshop - 7.28.09Chris Lehmann
 
Supporting Instructors in MOOCs: Using cognitive science research to guide pe...
Supporting Instructors in MOOCs: Using cognitive science research to guide pe...Supporting Instructors in MOOCs: Using cognitive science research to guide pe...
Supporting Instructors in MOOCs: Using cognitive science research to guide pe...Joseph Jay Williams
 
Umkc presentation
Umkc presentationUmkc presentation
Umkc presentationJohn Bourne
 
How can Cognitive Science improve Online Learning & Education?
How can Cognitive Science improve Online Learning & Education?How can Cognitive Science improve Online Learning & Education?
How can Cognitive Science improve Online Learning & Education?Joseph Jay Williams
 
Tony Wagner: Creating Innovators
Tony Wagner: Creating InnovatorsTony Wagner: Creating Innovators
Tony Wagner: Creating InnovatorsLee Carlton
 
Personal Digital Inquiry Summer Institute in Digital Literacy 2019
Personal Digital Inquiry Summer Institute in Digital Literacy 2019Personal Digital Inquiry Summer Institute in Digital Literacy 2019
Personal Digital Inquiry Summer Institute in Digital Literacy 2019Julie Coiro
 
Personal Digital Inquiry: Connecting Learners in Ways That Matter
Personal Digital Inquiry: Connecting Learners in Ways That MatterPersonal Digital Inquiry: Connecting Learners in Ways That Matter
Personal Digital Inquiry: Connecting Learners in Ways That MatterJulie Coiro
 
Next Generation Open Courseware
Next Generation Open CoursewareNext Generation Open Courseware
Next Generation Open CoursewareLumen Learning
 
Workshop: Building the Future Learning Organization
Workshop: Building the Future Learning OrganizationWorkshop: Building the Future Learning Organization
Workshop: Building the Future Learning OrganizationJohan Skoglöf
 
Putting the Pedagogic Horse before the Technology Cart
Putting the Pedagogic Horse before the Technology CartPutting the Pedagogic Horse before the Technology Cart
Putting the Pedagogic Horse before the Technology CartCharles Darwin University
 
Intelligent Adaptive Learning: A Powerful Element for 21st Century Learning &...
Intelligent Adaptive Learning: A Powerful Element for 21st Century Learning &...Intelligent Adaptive Learning: A Powerful Element for 21st Century Learning &...
Intelligent Adaptive Learning: A Powerful Element for 21st Century Learning &...DreamBox Learning
 
Bottom-up growth of learning analytics at two Australian universities: Empow...
Bottom-up growth of learning analytics at two Australian universities: Empow...Bottom-up growth of learning analytics at two Australian universities: Empow...
Bottom-up growth of learning analytics at two Australian universities: Empow...Danny Liu
 
Echo presentation hierarhical process modelling case study
Echo presentation  hierarhical process modelling case studyEcho presentation  hierarhical process modelling case study
Echo presentation hierarhical process modelling case studyRuth Deakin Crick
 
Many Chances to Fail: Scholarly Teaching in Physics - CO/WY AAPT - April 2014
Many Chances to Fail: Scholarly Teaching in Physics - CO/WY AAPT - April 2014Many Chances to Fail: Scholarly Teaching in Physics - CO/WY AAPT - April 2014
Many Chances to Fail: Scholarly Teaching in Physics - CO/WY AAPT - April 2014Jeff Loats
 
Learning in the digital age 2014
Learning in the digital age 2014Learning in the digital age 2014
Learning in the digital age 2014Cameron Furnival
 
Actively engaging learners by using a technology enhanced approach that enric...
Actively engaging learners by using a technology enhanced approach that enric...Actively engaging learners by using a technology enhanced approach that enric...
Actively engaging learners by using a technology enhanced approach that enric...Andy Jaffrey
 

Similar a How online educational resources provide novel affordances for conducting practical interventions and doing psychology experiments (20)

Learning Engineering of MOOClets: Simultaneously benefiting Professional Lear...
Learning Engineering of MOOClets: Simultaneously benefiting Professional Lear...Learning Engineering of MOOClets: Simultaneously benefiting Professional Lear...
Learning Engineering of MOOClets: Simultaneously benefiting Professional Lear...
 
Introduction to Personal Digital Inquiry in Grades K-8
Introduction to Personal Digital Inquiry in Grades K-8Introduction to Personal Digital Inquiry in Grades K-8
Introduction to Personal Digital Inquiry in Grades K-8
 
Where Does It Live: Building Systems and Structures Around What You Believe
Where Does It Live: Building Systems and Structures Around What You BelieveWhere Does It Live: Building Systems and Structures Around What You Believe
Where Does It Live: Building Systems and Structures Around What You Believe
 
Winston Salem Administrators Workshop - 7.28.09
Winston Salem Administrators Workshop - 7.28.09Winston Salem Administrators Workshop - 7.28.09
Winston Salem Administrators Workshop - 7.28.09
 
Supporting Instructors in MOOCs: Using cognitive science research to guide pe...
Supporting Instructors in MOOCs: Using cognitive science research to guide pe...Supporting Instructors in MOOCs: Using cognitive science research to guide pe...
Supporting Instructors in MOOCs: Using cognitive science research to guide pe...
 
Passion CF
Passion CFPassion CF
Passion CF
 
Umkc presentation
Umkc presentationUmkc presentation
Umkc presentation
 
How can Cognitive Science improve Online Learning & Education?
How can Cognitive Science improve Online Learning & Education?How can Cognitive Science improve Online Learning & Education?
How can Cognitive Science improve Online Learning & Education?
 
Tony Wagner: Creating Innovators
Tony Wagner: Creating InnovatorsTony Wagner: Creating Innovators
Tony Wagner: Creating Innovators
 
Personal Digital Inquiry Summer Institute in Digital Literacy 2019
Personal Digital Inquiry Summer Institute in Digital Literacy 2019Personal Digital Inquiry Summer Institute in Digital Literacy 2019
Personal Digital Inquiry Summer Institute in Digital Literacy 2019
 
Personal Digital Inquiry: Connecting Learners in Ways That Matter
Personal Digital Inquiry: Connecting Learners in Ways That MatterPersonal Digital Inquiry: Connecting Learners in Ways That Matter
Personal Digital Inquiry: Connecting Learners in Ways That Matter
 
Next Generation Open Courseware
Next Generation Open CoursewareNext Generation Open Courseware
Next Generation Open Courseware
 
Workshop: Building the Future Learning Organization
Workshop: Building the Future Learning OrganizationWorkshop: Building the Future Learning Organization
Workshop: Building the Future Learning Organization
 
Putting the Pedagogic Horse before the Technology Cart
Putting the Pedagogic Horse before the Technology CartPutting the Pedagogic Horse before the Technology Cart
Putting the Pedagogic Horse before the Technology Cart
 
Intelligent Adaptive Learning: A Powerful Element for 21st Century Learning &...
Intelligent Adaptive Learning: A Powerful Element for 21st Century Learning &...Intelligent Adaptive Learning: A Powerful Element for 21st Century Learning &...
Intelligent Adaptive Learning: A Powerful Element for 21st Century Learning &...
 
Bottom-up growth of learning analytics at two Australian universities: Empow...
Bottom-up growth of learning analytics at two Australian universities: Empow...Bottom-up growth of learning analytics at two Australian universities: Empow...
Bottom-up growth of learning analytics at two Australian universities: Empow...
 
Echo presentation hierarhical process modelling case study
Echo presentation  hierarhical process modelling case studyEcho presentation  hierarhical process modelling case study
Echo presentation hierarhical process modelling case study
 
Many Chances to Fail: Scholarly Teaching in Physics - CO/WY AAPT - April 2014
Many Chances to Fail: Scholarly Teaching in Physics - CO/WY AAPT - April 2014Many Chances to Fail: Scholarly Teaching in Physics - CO/WY AAPT - April 2014
Many Chances to Fail: Scholarly Teaching in Physics - CO/WY AAPT - April 2014
 
Learning in the digital age 2014
Learning in the digital age 2014Learning in the digital age 2014
Learning in the digital age 2014
 
Actively engaging learners by using a technology enhanced approach that enric...
Actively engaging learners by using a technology enhanced approach that enric...Actively engaging learners by using a technology enhanced approach that enric...
Actively engaging learners by using a technology enhanced approach that enric...
 

Más de Joseph Jay Williams

CHI (Computer Human Interaction) 2019 enhancing online problems through instr...
CHI (Computer Human Interaction) 2019 enhancing online problems through instr...CHI (Computer Human Interaction) 2019 enhancing online problems through instr...
CHI (Computer Human Interaction) 2019 enhancing online problems through instr...Joseph Jay Williams
 
Experiments in Educational Research and Practice
Experiments in Educational Research and PracticeExperiments in Educational Research and Practice
Experiments in Educational Research and PracticeJoseph Jay Williams
 
Learning innovation at scale chi 2014 workshop extended abstract
Learning innovation at scale chi 2014 workshop extended abstractLearning innovation at scale chi 2014 workshop extended abstract
Learning innovation at scale chi 2014 workshop extended abstractJoseph Jay Williams
 
Experiments in Educational Research & Practice
Experiments in Educational Research & PracticeExperiments in Educational Research & Practice
Experiments in Educational Research & PracticeJoseph Jay Williams
 
Doing online learning research with both scientific and financial value
Doing online learning research with both scientific and financial valueDoing online learning research with both scientific and financial value
Doing online learning research with both scientific and financial valueJoseph Jay Williams
 
Joseph Williams – Bloomsburg Corporate Advisory Council Meeting
Joseph Williams – Bloomsburg Corporate Advisory Council MeetingJoseph Williams – Bloomsburg Corporate Advisory Council Meeting
Joseph Williams – Bloomsburg Corporate Advisory Council MeetingJoseph Jay Williams
 
Explanation & learning slides (talk @ pittsburgh science of learning center)
Explanation & learning slides (talk @ pittsburgh science of learning center)Explanation & learning slides (talk @ pittsburgh science of learning center)
Explanation & learning slides (talk @ pittsburgh science of learning center)Joseph Jay Williams
 

Más de Joseph Jay Williams (8)

CHI (Computer Human Interaction) 2019 enhancing online problems through instr...
CHI (Computer Human Interaction) 2019 enhancing online problems through instr...CHI (Computer Human Interaction) 2019 enhancing online problems through instr...
CHI (Computer Human Interaction) 2019 enhancing online problems through instr...
 
Experiments in Educational Research and Practice
Experiments in Educational Research and PracticeExperiments in Educational Research and Practice
Experiments in Educational Research and Practice
 
Learning innovation at scale chi 2014 workshop extended abstract
Learning innovation at scale chi 2014 workshop extended abstractLearning innovation at scale chi 2014 workshop extended abstract
Learning innovation at scale chi 2014 workshop extended abstract
 
Experiments in Educational Research & Practice
Experiments in Educational Research & PracticeExperiments in Educational Research & Practice
Experiments in Educational Research & Practice
 
Doing online learning research with both scientific and financial value
Doing online learning research with both scientific and financial valueDoing online learning research with both scientific and financial value
Doing online learning research with both scientific and financial value
 
Joseph Williams – Bloomsburg Corporate Advisory Council Meeting
Joseph Williams – Bloomsburg Corporate Advisory Council MeetingJoseph Williams – Bloomsburg Corporate Advisory Council Meeting
Joseph Williams – Bloomsburg Corporate Advisory Council Meeting
 
Williams lombrozo2010
Williams lombrozo2010Williams lombrozo2010
Williams lombrozo2010
 
Explanation & learning slides (talk @ pittsburgh science of learning center)
Explanation & learning slides (talk @ pittsburgh science of learning center)Explanation & learning slides (talk @ pittsburgh science of learning center)
Explanation & learning slides (talk @ pittsburgh science of learning center)
 

Último

Drug Information Services- DIC and Sources.
Drug Information Services- DIC and Sources.Drug Information Services- DIC and Sources.
Drug Information Services- DIC and Sources.raviapr7
 
The Stolen Bacillus by Herbert George Wells
The Stolen Bacillus by Herbert George WellsThe Stolen Bacillus by Herbert George Wells
The Stolen Bacillus by Herbert George WellsEugene Lysak
 
Presentation on the Basics of Writing. Writing a Paragraph
Presentation on the Basics of Writing. Writing a ParagraphPresentation on the Basics of Writing. Writing a Paragraph
Presentation on the Basics of Writing. Writing a ParagraphNetziValdelomar1
 
Quality Assurance_GOOD LABORATORY PRACTICE
Quality Assurance_GOOD LABORATORY PRACTICEQuality Assurance_GOOD LABORATORY PRACTICE
Quality Assurance_GOOD LABORATORY PRACTICESayali Powar
 
Ultra structure and life cycle of Plasmodium.pptx
Ultra structure and life cycle of Plasmodium.pptxUltra structure and life cycle of Plasmodium.pptx
Ultra structure and life cycle of Plasmodium.pptxDr. Asif Anas
 
What is the Future of QuickBooks DeskTop?
What is the Future of QuickBooks DeskTop?What is the Future of QuickBooks DeskTop?
What is the Future of QuickBooks DeskTop?TechSoup
 
UKCGE Parental Leave Discussion March 2024
UKCGE Parental Leave Discussion March 2024UKCGE Parental Leave Discussion March 2024
UKCGE Parental Leave Discussion March 2024UKCGE
 
Clinical Pharmacy Introduction to Clinical Pharmacy, Concept of clinical pptx
Clinical Pharmacy  Introduction to Clinical Pharmacy, Concept of clinical pptxClinical Pharmacy  Introduction to Clinical Pharmacy, Concept of clinical pptx
Clinical Pharmacy Introduction to Clinical Pharmacy, Concept of clinical pptxraviapr7
 
How to Solve Singleton Error in the Odoo 17
How to Solve Singleton Error in the  Odoo 17How to Solve Singleton Error in the  Odoo 17
How to Solve Singleton Error in the Odoo 17Celine George
 
5 charts on South Africa as a source country for international student recrui...
5 charts on South Africa as a source country for international student recrui...5 charts on South Africa as a source country for international student recrui...
5 charts on South Africa as a source country for international student recrui...CaraSkikne1
 
Practical Research 1 Lesson 9 Scope and delimitation.pptx
Practical Research 1 Lesson 9 Scope and delimitation.pptxPractical Research 1 Lesson 9 Scope and delimitation.pptx
Practical Research 1 Lesson 9 Scope and delimitation.pptxKatherine Villaluna
 
DUST OF SNOW_BY ROBERT FROST_EDITED BY_ TANMOY MISHRA
DUST OF SNOW_BY ROBERT FROST_EDITED BY_ TANMOY MISHRADUST OF SNOW_BY ROBERT FROST_EDITED BY_ TANMOY MISHRA
DUST OF SNOW_BY ROBERT FROST_EDITED BY_ TANMOY MISHRATanmoy Mishra
 
The basics of sentences session 10pptx.pptx
The basics of sentences session 10pptx.pptxThe basics of sentences session 10pptx.pptx
The basics of sentences session 10pptx.pptxheathfieldcps1
 
Prescribed medication order and communication skills.pptx
Prescribed medication order and communication skills.pptxPrescribed medication order and communication skills.pptx
Prescribed medication order and communication skills.pptxraviapr7
 
Diploma in Nursing Admission Test Question Solution 2023.pdf
Diploma in Nursing Admission Test Question Solution 2023.pdfDiploma in Nursing Admission Test Question Solution 2023.pdf
Diploma in Nursing Admission Test Question Solution 2023.pdfMohonDas
 
How to Make a Field read-only in Odoo 17
How to Make a Field read-only in Odoo 17How to Make a Field read-only in Odoo 17
How to Make a Field read-only in Odoo 17Celine George
 
Patient Counselling. Definition of patient counseling; steps involved in pati...
Patient Counselling. Definition of patient counseling; steps involved in pati...Patient Counselling. Definition of patient counseling; steps involved in pati...
Patient Counselling. Definition of patient counseling; steps involved in pati...raviapr7
 
How to Add a many2many Relational Field in Odoo 17
How to Add a many2many Relational Field in Odoo 17How to Add a many2many Relational Field in Odoo 17
How to Add a many2many Relational Field in Odoo 17Celine George
 

Último (20)

Drug Information Services- DIC and Sources.
Drug Information Services- DIC and Sources.Drug Information Services- DIC and Sources.
Drug Information Services- DIC and Sources.
 
The Stolen Bacillus by Herbert George Wells
The Stolen Bacillus by Herbert George WellsThe Stolen Bacillus by Herbert George Wells
The Stolen Bacillus by Herbert George Wells
 
Presentation on the Basics of Writing. Writing a Paragraph
Presentation on the Basics of Writing. Writing a ParagraphPresentation on the Basics of Writing. Writing a Paragraph
Presentation on the Basics of Writing. Writing a Paragraph
 
Quality Assurance_GOOD LABORATORY PRACTICE
Quality Assurance_GOOD LABORATORY PRACTICEQuality Assurance_GOOD LABORATORY PRACTICE
Quality Assurance_GOOD LABORATORY PRACTICE
 
Ultra structure and life cycle of Plasmodium.pptx
Ultra structure and life cycle of Plasmodium.pptxUltra structure and life cycle of Plasmodium.pptx
Ultra structure and life cycle of Plasmodium.pptx
 
Prelims of Kant get Marx 2.0: a general politics quiz
Prelims of Kant get Marx 2.0: a general politics quizPrelims of Kant get Marx 2.0: a general politics quiz
Prelims of Kant get Marx 2.0: a general politics quiz
 
What is the Future of QuickBooks DeskTop?
What is the Future of QuickBooks DeskTop?What is the Future of QuickBooks DeskTop?
What is the Future of QuickBooks DeskTop?
 
UKCGE Parental Leave Discussion March 2024
UKCGE Parental Leave Discussion March 2024UKCGE Parental Leave Discussion March 2024
UKCGE Parental Leave Discussion March 2024
 
Clinical Pharmacy Introduction to Clinical Pharmacy, Concept of clinical pptx
Clinical Pharmacy  Introduction to Clinical Pharmacy, Concept of clinical pptxClinical Pharmacy  Introduction to Clinical Pharmacy, Concept of clinical pptx
Clinical Pharmacy Introduction to Clinical Pharmacy, Concept of clinical pptx
 
How to Solve Singleton Error in the Odoo 17
How to Solve Singleton Error in the  Odoo 17How to Solve Singleton Error in the  Odoo 17
How to Solve Singleton Error in the Odoo 17
 
5 charts on South Africa as a source country for international student recrui...
5 charts on South Africa as a source country for international student recrui...5 charts on South Africa as a source country for international student recrui...
5 charts on South Africa as a source country for international student recrui...
 
Practical Research 1 Lesson 9 Scope and delimitation.pptx
Practical Research 1 Lesson 9 Scope and delimitation.pptxPractical Research 1 Lesson 9 Scope and delimitation.pptx
Practical Research 1 Lesson 9 Scope and delimitation.pptx
 
DUST OF SNOW_BY ROBERT FROST_EDITED BY_ TANMOY MISHRA
DUST OF SNOW_BY ROBERT FROST_EDITED BY_ TANMOY MISHRADUST OF SNOW_BY ROBERT FROST_EDITED BY_ TANMOY MISHRA
DUST OF SNOW_BY ROBERT FROST_EDITED BY_ TANMOY MISHRA
 
The basics of sentences session 10pptx.pptx
The basics of sentences session 10pptx.pptxThe basics of sentences session 10pptx.pptx
The basics of sentences session 10pptx.pptx
 
Prescribed medication order and communication skills.pptx
Prescribed medication order and communication skills.pptxPrescribed medication order and communication skills.pptx
Prescribed medication order and communication skills.pptx
 
Diploma in Nursing Admission Test Question Solution 2023.pdf
Diploma in Nursing Admission Test Question Solution 2023.pdfDiploma in Nursing Admission Test Question Solution 2023.pdf
Diploma in Nursing Admission Test Question Solution 2023.pdf
 
How to Make a Field read-only in Odoo 17
How to Make a Field read-only in Odoo 17How to Make a Field read-only in Odoo 17
How to Make a Field read-only in Odoo 17
 
Finals of Kant get Marx 2.0 : a general politics quiz
Finals of Kant get Marx 2.0 : a general politics quizFinals of Kant get Marx 2.0 : a general politics quiz
Finals of Kant get Marx 2.0 : a general politics quiz
 
Patient Counselling. Definition of patient counseling; steps involved in pati...
Patient Counselling. Definition of patient counseling; steps involved in pati...Patient Counselling. Definition of patient counseling; steps involved in pati...
Patient Counselling. Definition of patient counseling; steps involved in pati...
 
How to Add a many2many Relational Field in Odoo 17
How to Add a many2many Relational Field in Odoo 17How to Add a many2many Relational Field in Odoo 17
How to Add a many2many Relational Field in Odoo 17
 

How online educational resources provide novel affordances for conducting practical interventions and doing psychology experiments

  • 1. How online educational resources provide novel affordances for conducting practical interventions and doing psychology experiments www.josephjaywilliams.com/experiments-online-education Joseph Jay Williams, Stanford University Lytics Lab (lytics.stanford.edu), Graduate School of Education Office of the Vice Provost of Online Learning (vpol.stanford.edu) josephjaywilliams@stanford.edu www.josephjaywilliams.com Originally from Trinidad – apologies for thick accent! 1
  • 2. Overview • • • • • • MOOCs VS Digital Online Resources Experiments: Link Real World & Laboratory Motivational Messages Experiment Simultaneous Basic & Applied Research Ed-Surge Meetup to link Research & Practice (Wed Feb 19 @ 7) Current Directions Teaching Learning Strategies Learning Coach MOOC Experiment • MOOClets Research & Practice Group 2
  • 4. Moving from Physical to Digital World
  • 5. Experiments: Link Real World & Laboratory • Randomized assignment • Experimental Control • Rich data • Real-world environment • Authentic activities • Practical Challenges
  • 6. Experiment • Practical goal on Khan Academy • Extend motivation research 6
  • 7. Motivate students to practice math • Online (Math) Exercise Motivational Message: Keep up the good work! 1. Number of Problems Practiced 2. Improve Accuracy 7
  • 8. Add motivational messages Practice-as-usual Growth Mindset Message Remember, the more you practice the smarter you become! 8
  • 9. Which messages to add? • • • • • • "This might be a tough problem, but we know you can do it!” Remember, the more you practice the smarter you become! Some of these problems are hard. Just do your best.” Encourage “Growth Mindset” beliefs about intelligence (Dweck, 2008) Fixed Mindset Growth Mindset 9
  • 10. Embedded in vivo Experiment Jascha Sohl-Dickstein • Practice-as-usual • Growth Mindset Message • • "Remember, the more you practice the smarter you become.”, "Mistakes help you learn. Think hard to learn from them.” 10
  • 11. Results: More motivated? • Growth Mindset Message > Practice-asUsual • reduction in dropout rate • increase in Number of problems practiced • increase in Number Correct • Accuracy: Problems correct/Problems attempted • Accuracy improvement 11
  • 12. Interpreting results • Obvious? • Skepticism: Anything’s statistically signficant in a MOOC 12
  • 13. Different motivational messages Practice-as-usual Message Growth Mindset Message Positive Remember, the more you practice the smarter you become! Some of these problems are hard. Do your best! 13
  • 14. Does any positive message work? • Practice-as-usual • Growth Mindset Message • • "Remember, the more you practice the smarter you become.”, "Mistakes help you learn. Think hard to learn from them.” • Positive Message • • "Some of these problems are hard. Just do your best." "This might be a tough problem, but we know you can do it.” 14
  • 15. Effects of Positive Messages? • Positive Messages not better than Practice-as-Usual! • Growth Mindset better than Positive • reduction in dropout rate • increase in Number of problems practiced • increase in Number Correct • Accuracy improvement 15
  • 16. Next: Situationally appropriate messages • Message blast, can be ignored • Providing motivational message Justin-Time: after incorrect answer • Laboratory experiments • www.assistments.org Neil Heffernan 16
  • 17. Simultaneous Applied & Basic Research • Start with “Product” • Constraints Randomization (Technical, use Qualtrics) Behavioral/Learning Measures Williams & Williams (2013). Using Interventions to improve Online Learning. NIPS Data Driven Education workshop. tiny.cc/experimentsonlinelearning • Consider Synthesis Williams, J.J. (2013) Improving Learning in MOOCs by Applying Cognitive Science. AIED MOOCshop & NIPS Data-Driven Workshop. tiny.cc/cognitivescienceinmoocs Reading List: www.josephjaywilliams.com/education Improving outcome of psychosocial treatments by enhancing memory and learning.(in press) Perspectives in Psychological Science. • High Investment but High Return 17
  • 18. Multidisciplinary – Relevant Communities • • • • • Association for Psychological Science 2013. Online Learning and Psychological Science: Opportunities to integrate research and practice. Williams, Saxberg, Means, Mitros. Cognitive Science Society 2013. Online Education: A Unique Opportunity for Cognitive Scientists to Integrate Research and Practice. Williams, Renkl, Koedinger, Stamper. AERA 2014– American Education Research Association. How online resources can facilitate interdisciplinary collaboration. Williams, Thille, Trumbore, Stigler, Siemens. ICLS (under review) – International Conference for the Learning Sciences. Online & Blended Education: Opportunities and Obstacles for the Learning Sciences. Williams, Linn, Slotta, Rose, Dillenbourg. CHI 2014 – Computer Human Interaction. Learning Innovations at Scale. Williams, Kizilcec, Russell, Klemmer. • • • E-learning guild’s DevLearn Educational Data Mining Artificial Intelligence in Education • • • Computer Supported Collaborative Learning Professional & Organizational Development Network American Society for Training & Development
  • 19. Laboratory Paradigms & Online Products • A matching problem • www.edsurge.com, www.edudemic.com • josephjaywilliams@stanford.edu • Williams, J.J., Teachman, B.A., Richland, L., Brady, S.T, & Aleahmad, T. (2014). Leveraging the Internet to do Laboratory Research in the Real World. Symposium to be conducted at the annual convention of the Association for Psychological Science. San Francisco, CA. 19
  • 20. Example: Explanation & Learning • Williams & Lombrozo (2010, Cognitive Science; 2013, Cognitive Psychology; w Rehder, JEP: General) Glorp Right Drent Wrong 20
  • 21. EdSurge Meetup Wed Feb 19 7 pm CERAS • “Improving Ed-Tech using Research on Learning & Advancing Research on Learning using Ed-Tech” • Ed-Tech Startup Survey • Researcher Survey • Featured Company & Researcher • Panelist & Question suggestions? 21
  • 22. Completely New Horizons for Experimentation • “Bridging Studies” • www.josephjaywilliams.com/experime nts-online-education • Williams, J. J. (2013). Enhancing Educational Research & Practice using Experiments on Online Educational Resources. Talk at Pittsburgh Science of Learning Center LearnLab Summer School, Pittsburgh, PA. tiny.cc/onlineexperiments 22
  • 23. Next: Teaching Learning Strategies • Limited teaching time: Specific content or general strategies? • Online Advantage: Easier to run RCTs & collect longitudinal data Reinforce new educational habits & behaviors • Teach metacognitive self-questioning strategies (Palinscar & Brown, 1984, Cognition and instruction; McNamara, 2004, Discourse Processes; Williams & Lombrozo, 2010, Cognitive Science) 23
  • 24. Drop-down text about strategies Clickable link. [Click here to learn about the “What? Why? How?” strategy]
  • 26. MOOC experiment • Motivational Video Control Energetic Thank you & Encouragement Growth Mindset (by Instructor) Growth Mindset (outside Instructor) “Belonging” • Questions for reflection after video None VS Just Think VS Write response 26
  • 27. MOOClets Group – Develop Digital Modules • • • • • • • • • E.g. MOOClet: Math exercise, 5 minute interactive video, 30 minute lesson, Study Strategy coach, Quiz/Assessment Instructors can use for blended classroom education (High School, Community College, University) Continuously in use and being improved Easy to build, prototype, test & iterate Researchers directly implement & collect data Small scale & online access facilitates multidisciplinary collaboration & research-practitioner development (see AERA 2014 invited panel) Illustrate/Operationalize instructional design principles MOOClet Reading & Discussion Group meets Tue 11-12 PST @ Barnum (Stanford) or tiny.cc/lyticshangout Multidisciplinary group of researchers & practitioners (tiny.cc/mooclet) 27
  • 28. Review • • • • • • • MOOCs VS Digital Online Resources Experiments: Link Real World & Laboratory Motivational Messages Experiment Simultaneous Basic & Applied Research Ed-Surge Meetup to link Research & Practice (Wed Feb 19 @ 7) Current Directions Teaching Learning Strategies Learning Coach MOOC Experiment MOOClets Research & Practice Group Meets Tuesdays 11-12 PST (tiny.cc/lyticshangout, tiny.cc/mooclet) 28
  • 29. Acknowledgements • • • • • • Jascha Sohl-Dickstein Jace Kohlmeier & Khan Academy PERTS Sam Maldonado Lytics Lab (lytics.stanford.edu) VPOL (Vice Provost of Online Learning, online.stanford.edu) 29

Notas del editor

  1. Originally from Trinidad. I’m a Research Fellow in theLytics Lab & Office of Online Learning. Cognitive Science background – experimental psychology, statistical modeling and machine learning.Core interest is being a knowledge broker – reviewing and synthesizing the thousands of published studies in cognitive science, education research, the learning sciences to see which ones are relevant to real-world outcomes, and building products or conducting experiments that improve practically ad financially valuable outcomes. Illustrate this approach with an experiment that shows how to improve students’ motivation while learning from mathematics problems on Khan Academy.
  2. o.320.27