SlideShare una empresa de Scribd logo
1 de 33
Descargar para leer sin conexión
The Moodle Quiz at the Open University:
how we use it & how that helps students
Tim Hunt, Information Technology
MoodleMoot Dublin 2015
The Moodle Quiz at the Open University:
how we use it & how that helps students
Tim Hunt, Information Technology
MoodleMoot Dublin 2015
many dedicated
OU teaching staff
Alternate titles
Lots of interesting graphs
The scholarship of teaching and learning in practice
Overall use
It matters
End of module survey results
0%
20%
40%
60%
80%
100%
KPI 01: Overall, I am satisfied with the quality of this module.
Q33 Clear understanding of what was required to complete the assessed work
Q34 the assessment activities supported my learning
It makes a difference
Level 3 physics (SM358)
SM358 assessment tasks
0 TMAs 1 TMA 2 TMAs 3 TMAs 4 TMAs
0 iCMAs
1 iCMA
2 iCMAs
3 iCMAs
4 iCMAs
5 iCMAs
6 iCMAs
Students completing tasks
0 TMAs 1 TMA 2 TMAs 3 TMAs 4 TMAs
0 iCMAs 11.6% 3.4% 1.5% 0.5% 0.5%
1 iCMA 1.5% 1.0%
2 iCMAs 1.5% 2.4% 1.5%
3 iCMAs 1.5%
4 iCMAs 5.3% 2.4%
5 iCMAs 0.5% 3.9% 5.8% 8.2%
6 iCMAs 0.5% 0.5% 5.8% 5.8% 34.3%
Average exam scores
0 TMAs 1 TMA 2 TMAs 3 TMAs 4 TMAs
0 iCMAs 6.0
1 iCMA
2 iCMAs 17.0 24.0
3 iCMAs 60.0
4 iCMAs 43.7 62.0
5 iCMAs 23.0 46.0 62.6 69.5
6 iCMAs 35.3 60.8 77.5
Exam scores vs prediction
0 TMAs 1 TMA 2 TMAs 3 TMAs 4 TMAs
0 iCMAs −20.8
1 iCMA
2 iCMAs −43.9 −27.5
3 iCMAs −9.0
4 iCMAs −15.6 +1.8
5 iCMAs −3.8 −11.1 +1.4 +2.4
6 iCMAs −17.1 +3.4 +4.6
Changing assessment type
T184 robotics & meaning of life
Before
10% mid-course iCMA
90% final written EMA
part 1 short-answer
part 2 programming
& essays
After
10% mid-course iCMA
30% final iCMA
60% final written EMA
Module completion rates
T184 completion rates
60%
65%
70%
75%
80%
85%
90%
95%
100%
2004 2005 2006 2007 2008 2009 2010 2011
May
Oct
Introduction of
CME
T184 completion rates
60%
65%
70%
75%
80%
85%
90%
95%
100%
2004 2005 2006 2007 2008 2009 2010 2011
May
Oct
Introduction of
CME
Module completion rates
T184 completion rates
60%
65%
70%
75%
80%
85%
90%
95%
100%
2004 2005 2006 2007 2008 2009 2010 2011
May
Oct
Introduction of
CME
Module completion rates
Deadlines
SM358 iCMA51 submit date
2010 advisory deadline
SM358 iCMA51 submit date
2010 advisory deadline
2012 hard deadline
Grades
Optional quizzes
Compulsory quizzes
Do students read feedback?
Interactive with multiple tries
Same question in two quizzes
Can computers grade sentences?
Spoiler: Yes!
How good are humans?
Question Number of
responses in
analysis
Percentage of responses where
the human markers were in
agreement with question author
Percentage of
responses where
computer marking
was in agreement
with question
author
Range for the
6 human
markers
Mean of the 6
human
markers
A 189 97.4 – 100. 98.9 99.5
B 248 83.9 – 97.2 91.9 97.6
C 150 80.7 – 94.0 86.9 94.7
D 129 91.5 – 98.4 96.7 97.6
E 92 92.4 – 97.8 95.1 98.9
F 129 86.0 – 97.7 90.8 97.7
G 132 66.7 – 90.2 83.2 89.4
Comparing three algorithms
Question Number of
responses in
analysis
Percentage of responses where computer marking
was in agreement with question author
Computational
linguistics
Algorithmic manipulation of
keywords
IAT Pattern-match Regular
Expressions
A 189 99.5 99.5 98.9
B 248 97.6 98.8 98.0
C 150 94.7 94.7 90.7
D 129 97.6 96.1 97.7
E 92 98.9 96.7 96.7
F 129 97.7 88.4 89.2
G 132 89.4 87.9 88.6
Summary
References
• Overall iCMA usage numbers collated by Phil Butcher.
• End of module server results from Student Analytics in the Institute of Educational
Technology via Linda Price.
• SM358 data from John Bolton
• T184 data from Jon Rosewell
• SDK125, S141, S151 & S240 data from Sally Jordan
• Jordan, Sally (2014). Using e-assessment to learn about students and learning.
International Journal of e-Assessment, 4(1)
• Jordan, Sally (2014). Adult science learners’ mathematical mistakes: an analysis
of responses to computer-marked questions. European Journal of Science and
Mathematics Education, 2(2) pp. 63–86.
• Jordan, Sally (2012). Short-answer e-assessment questions : five years on. In:
2012 International Computer Assisted Assessment Conference, 10-11 July 2012,
Southampton.
• Pattern-match question type for Moodle
https://moodle.org/plugins/view/qtype_pmatch
Key points
Getting the assessment right is important
Online quizzes can be powerful learning tools
Dig into the data and you can learn
• Is this quiz working?
• What are my students’ misconceptions?
Computers can grade much more than multi-choice
but only on behalf of a teacher
Want more? I’m giving this talk again at #iMoot15

Más contenido relacionado

La actualidad más candente

Numberwise Info Guide
Numberwise Info GuideNumberwise Info Guide
Numberwise Info GuideGreg Swart
 
eLearning & Resuscitation Training -7th INTERNATIONAL SPARK OF LIFE CONFE...
eLearning & Resuscitation Training -7th INTERNATIONAL SPARK OF LIFE CONFE...eLearning & Resuscitation Training -7th INTERNATIONAL SPARK OF LIFE CONFE...
eLearning & Resuscitation Training -7th INTERNATIONAL SPARK OF LIFE CONFE...shlenson
 
Using the student voice to plan our Moodle - Mark Glynn, Eamon Costello
Using the student voice to plan our Moodle - Mark Glynn, Eamon CostelloUsing the student voice to plan our Moodle - Mark Glynn, Eamon Costello
Using the student voice to plan our Moodle - Mark Glynn, Eamon CostelloIreland & UK Moodlemoot 2012
 
Helen Johnston et al 2008
Helen Johnston et al 2008Helen Johnston et al 2008
Helen Johnston et al 2008Diana Quinn
 
Will time tell? Exploring the relationship between step duration and student ...
Will time tell? Exploring the relationship between step duration and student ...Will time tell? Exploring the relationship between step duration and student ...
Will time tell? Exploring the relationship between step duration and student ...Irene-Angelica Chounta
 
Online assessment and data analytics - Peter Tan - Institute of Technical Edu...
Online assessment and data analytics - Peter Tan - Institute of Technical Edu...Online assessment and data analytics - Peter Tan - Institute of Technical Edu...
Online assessment and data analytics - Peter Tan - Institute of Technical Edu...Blackboard APAC
 
A Research Study on the Use of Wimba Classroom
A Research Study on the Use of Wimba ClassroomA Research Study on the Use of Wimba Classroom
A Research Study on the Use of Wimba Classroomahornton
 
Blackboard Learning Analytics Research Update
Blackboard Learning Analytics Research UpdateBlackboard Learning Analytics Research Update
Blackboard Learning Analytics Research UpdateJohn Whitmer, Ed.D.
 
A poster presented in teaching and learning conference, University of Birmingham
A poster presented in teaching and learning conference, University of BirminghamA poster presented in teaching and learning conference, University of Birmingham
A poster presented in teaching and learning conference, University of BirminghamMohamed El-Adawy
 
How did participation in the SSAI ICM programme affect my career?
How did participation in the SSAI ICM programme affect my career?How did participation in the SSAI ICM programme affect my career?
How did participation in the SSAI ICM programme affect my career?scanFOAM
 
MoDLA 2011 - Online WDs at MCC
MoDLA 2011 - Online WDs at MCCMoDLA 2011 - Online WDs at MCC
MoDLA 2011 - Online WDs at MCCLeo Hirner
 
The use of ict in teaching and learning
The use of ict in teaching and learningThe use of ict in teaching and learning
The use of ict in teaching and learningitsnadia
 
IDEE Workshop: Applying the 4C-ID Model to the Design of a Digital Educationa...
IDEE Workshop: Applying the 4C-ID Model to the Design of a Digital Educationa...IDEE Workshop: Applying the 4C-ID Model to the Design of a Digital Educationa...
IDEE Workshop: Applying the 4C-ID Model to the Design of a Digital Educationa...Guilhermina Miranda
 
Assisting Students in Finding Their Own Bugs in Programming Exercises using V...
Assisting Students in Finding Their Own Bugs in Programming Exercises using V...Assisting Students in Finding Their Own Bugs in Programming Exercises using V...
Assisting Students in Finding Their Own Bugs in Programming Exercises using V...Mai Dinh
 

La actualidad más candente (19)

Numberwise Info Guide
Numberwise Info GuideNumberwise Info Guide
Numberwise Info Guide
 
eLearning & Resuscitation Training -7th INTERNATIONAL SPARK OF LIFE CONFE...
eLearning & Resuscitation Training -7th INTERNATIONAL SPARK OF LIFE CONFE...eLearning & Resuscitation Training -7th INTERNATIONAL SPARK OF LIFE CONFE...
eLearning & Resuscitation Training -7th INTERNATIONAL SPARK OF LIFE CONFE...
 
Question 1
Question 1Question 1
Question 1
 
Lit Survey Review
Lit Survey ReviewLit Survey Review
Lit Survey Review
 
Investigating E-Lerning Effects on Continuance Intentions of Hospitality Mana...
Investigating E-Lerning Effects on Continuance Intentions of Hospitality Mana...Investigating E-Lerning Effects on Continuance Intentions of Hospitality Mana...
Investigating E-Lerning Effects on Continuance Intentions of Hospitality Mana...
 
Using the student voice to plan our Moodle - Mark Glynn, Eamon Costello
Using the student voice to plan our Moodle - Mark Glynn, Eamon CostelloUsing the student voice to plan our Moodle - Mark Glynn, Eamon Costello
Using the student voice to plan our Moodle - Mark Glynn, Eamon Costello
 
Helen Johnston et al 2008
Helen Johnston et al 2008Helen Johnston et al 2008
Helen Johnston et al 2008
 
Learn Education for FIT
Learn Education for FITLearn Education for FIT
Learn Education for FIT
 
Intriduction to Data Science - Week 1
Intriduction to Data Science - Week 1Intriduction to Data Science - Week 1
Intriduction to Data Science - Week 1
 
Will time tell? Exploring the relationship between step duration and student ...
Will time tell? Exploring the relationship between step duration and student ...Will time tell? Exploring the relationship between step duration and student ...
Will time tell? Exploring the relationship between step duration and student ...
 
Online assessment and data analytics - Peter Tan - Institute of Technical Edu...
Online assessment and data analytics - Peter Tan - Institute of Technical Edu...Online assessment and data analytics - Peter Tan - Institute of Technical Edu...
Online assessment and data analytics - Peter Tan - Institute of Technical Edu...
 
A Research Study on the Use of Wimba Classroom
A Research Study on the Use of Wimba ClassroomA Research Study on the Use of Wimba Classroom
A Research Study on the Use of Wimba Classroom
 
Blackboard Learning Analytics Research Update
Blackboard Learning Analytics Research UpdateBlackboard Learning Analytics Research Update
Blackboard Learning Analytics Research Update
 
A poster presented in teaching and learning conference, University of Birmingham
A poster presented in teaching and learning conference, University of BirminghamA poster presented in teaching and learning conference, University of Birmingham
A poster presented in teaching and learning conference, University of Birmingham
 
How did participation in the SSAI ICM programme affect my career?
How did participation in the SSAI ICM programme affect my career?How did participation in the SSAI ICM programme affect my career?
How did participation in the SSAI ICM programme affect my career?
 
MoDLA 2011 - Online WDs at MCC
MoDLA 2011 - Online WDs at MCCMoDLA 2011 - Online WDs at MCC
MoDLA 2011 - Online WDs at MCC
 
The use of ict in teaching and learning
The use of ict in teaching and learningThe use of ict in teaching and learning
The use of ict in teaching and learning
 
IDEE Workshop: Applying the 4C-ID Model to the Design of a Digital Educationa...
IDEE Workshop: Applying the 4C-ID Model to the Design of a Digital Educationa...IDEE Workshop: Applying the 4C-ID Model to the Design of a Digital Educationa...
IDEE Workshop: Applying the 4C-ID Model to the Design of a Digital Educationa...
 
Assisting Students in Finding Their Own Bugs in Programming Exercises using V...
Assisting Students in Finding Their Own Bugs in Programming Exercises using V...Assisting Students in Finding Their Own Bugs in Programming Exercises using V...
Assisting Students in Finding Their Own Bugs in Programming Exercises using V...
 

Similar a The Moodle Quiz at the Open University: how we use it & how that helps students

ISETL presentation: Cafeteria Style Grading 10/13/16
ISETL presentation: Cafeteria Style Grading 10/13/16ISETL presentation: Cafeteria Style Grading 10/13/16
ISETL presentation: Cafeteria Style Grading 10/13/16Anne Arendt
 
IRJET- Outcome Based Education(OBE) Execution in Polytechnic Education System...
IRJET- Outcome Based Education(OBE) Execution in Polytechnic Education System...IRJET- Outcome Based Education(OBE) Execution in Polytechnic Education System...
IRJET- Outcome Based Education(OBE) Execution in Polytechnic Education System...IRJET Journal
 
IRJET- Teaching Learning Practices for Metrology & Quality Control Subject in...
IRJET- Teaching Learning Practices for Metrology & Quality Control Subject in...IRJET- Teaching Learning Practices for Metrology & Quality Control Subject in...
IRJET- Teaching Learning Practices for Metrology & Quality Control Subject in...IRJET Journal
 
Statement Of Results ITIL
Statement Of Results ITILStatement Of Results ITIL
Statement Of Results ITILMathew Ferris
 
An empirical study on assessment of co attainment for a diploma course
An empirical study on assessment of co attainment for a diploma courseAn empirical study on assessment of co attainment for a diploma course
An empirical study on assessment of co attainment for a diploma courseIAEME Publication
 
CER Template (1).docx
CER Template (1).docxCER Template (1).docx
CER Template (1).docxmariny79
 
Se be information technology rev 2016
Se   be information technology rev 2016Se   be information technology rev 2016
Se be information technology rev 2016SANJEEVKUMARSRIVASTA7
 
Se be information technology rev 2016
Se   be information technology rev 2016Se   be information technology rev 2016
Se be information technology rev 2016SANJEEVKUMARSRIVASTA7
 
Syllabus_Grading System_Combustion Engg.pdf
Syllabus_Grading System_Combustion Engg.pdfSyllabus_Grading System_Combustion Engg.pdf
Syllabus_Grading System_Combustion Engg.pdfMarcJairroGajudo
 
Student Usability Study of BlueQuill LMS
Student Usability Study of BlueQuill LMSStudent Usability Study of BlueQuill LMS
Student Usability Study of BlueQuill LMSTawana Washington
 
Colorado Community College Systems COETC presentation at Accelerated Learning...
Colorado Community College Systems COETC presentation at Accelerated Learning...Colorado Community College Systems COETC presentation at Accelerated Learning...
Colorado Community College Systems COETC presentation at Accelerated Learning...cccschamp
 
Higher Education in Pakistan: An ICT Integration Viewpoint
Higher Education in Pakistan: An ICT Integration ViewpointHigher Education in Pakistan: An ICT Integration Viewpoint
Higher Education in Pakistan: An ICT Integration ViewpointZaffar Ahmed Shaikh
 
Higher education in pakistan an ict integration
Higher education in pakistan an ict integrationHigher education in pakistan an ict integration
Higher education in pakistan an ict integrationTariq Ghayyur
 
Language Localisation of Tamil using Statistical Machine Translation - ICTer2015
Language Localisation of Tamil using Statistical Machine Translation - ICTer2015Language Localisation of Tamil using Statistical Machine Translation - ICTer2015
Language Localisation of Tamil using Statistical Machine Translation - ICTer2015Achchuthan Yogarajah
 
Data Collection and Analysis Overview - Martin Furner
Data Collection and Analysis Overview - Martin FurnerData Collection and Analysis Overview - Martin Furner
Data Collection and Analysis Overview - Martin FurnerIMI PQ NET Romania
 

Similar a The Moodle Quiz at the Open University: how we use it & how that helps students (20)

TAO DAYS - E-Assessment for Learning
TAO DAYS - E-Assessment for LearningTAO DAYS - E-Assessment for Learning
TAO DAYS - E-Assessment for Learning
 
Ijmet 10 01_025
Ijmet 10 01_025Ijmet 10 01_025
Ijmet 10 01_025
 
ISETL presentation: Cafeteria Style Grading 10/13/16
ISETL presentation: Cafeteria Style Grading 10/13/16ISETL presentation: Cafeteria Style Grading 10/13/16
ISETL presentation: Cafeteria Style Grading 10/13/16
 
IRJET- Outcome Based Education(OBE) Execution in Polytechnic Education System...
IRJET- Outcome Based Education(OBE) Execution in Polytechnic Education System...IRJET- Outcome Based Education(OBE) Execution in Polytechnic Education System...
IRJET- Outcome Based Education(OBE) Execution in Polytechnic Education System...
 
Hkeaa2009
Hkeaa2009Hkeaa2009
Hkeaa2009
 
IRJET- Teaching Learning Practices for Metrology & Quality Control Subject in...
IRJET- Teaching Learning Practices for Metrology & Quality Control Subject in...IRJET- Teaching Learning Practices for Metrology & Quality Control Subject in...
IRJET- Teaching Learning Practices for Metrology & Quality Control Subject in...
 
Statement Of Results ITIL
Statement Of Results ITILStatement Of Results ITIL
Statement Of Results ITIL
 
An empirical study on assessment of co attainment for a diploma course
An empirical study on assessment of co attainment for a diploma courseAn empirical study on assessment of co attainment for a diploma course
An empirical study on assessment of co attainment for a diploma course
 
CER Template (1).docx
CER Template (1).docxCER Template (1).docx
CER Template (1).docx
 
MCAT Playbook Excerpts
MCAT Playbook ExcerptsMCAT Playbook Excerpts
MCAT Playbook Excerpts
 
ICTVT-05-10-20.pptx
ICTVT-05-10-20.pptxICTVT-05-10-20.pptx
ICTVT-05-10-20.pptx
 
Se be information technology rev 2016
Se   be information technology rev 2016Se   be information technology rev 2016
Se be information technology rev 2016
 
Se be information technology rev 2016
Se   be information technology rev 2016Se   be information technology rev 2016
Se be information technology rev 2016
 
Syllabus_Grading System_Combustion Engg.pdf
Syllabus_Grading System_Combustion Engg.pdfSyllabus_Grading System_Combustion Engg.pdf
Syllabus_Grading System_Combustion Engg.pdf
 
Student Usability Study of BlueQuill LMS
Student Usability Study of BlueQuill LMSStudent Usability Study of BlueQuill LMS
Student Usability Study of BlueQuill LMS
 
Colorado Community College Systems COETC presentation at Accelerated Learning...
Colorado Community College Systems COETC presentation at Accelerated Learning...Colorado Community College Systems COETC presentation at Accelerated Learning...
Colorado Community College Systems COETC presentation at Accelerated Learning...
 
Higher Education in Pakistan: An ICT Integration Viewpoint
Higher Education in Pakistan: An ICT Integration ViewpointHigher Education in Pakistan: An ICT Integration Viewpoint
Higher Education in Pakistan: An ICT Integration Viewpoint
 
Higher education in pakistan an ict integration
Higher education in pakistan an ict integrationHigher education in pakistan an ict integration
Higher education in pakistan an ict integration
 
Language Localisation of Tamil using Statistical Machine Translation - ICTer2015
Language Localisation of Tamil using Statistical Machine Translation - ICTer2015Language Localisation of Tamil using Statistical Machine Translation - ICTer2015
Language Localisation of Tamil using Statistical Machine Translation - ICTer2015
 
Data Collection and Analysis Overview - Martin Furner
Data Collection and Analysis Overview - Martin FurnerData Collection and Analysis Overview - Martin Furner
Data Collection and Analysis Overview - Martin Furner
 

Más de Tim Hunt

Writing better Behat tests
Writing better Behat testsWriting better Behat tests
Writing better Behat testsTim Hunt
 
What’s next for the Quiz and Question bank and for Moodle community collabora...
What’s next for the Quiz and Question bank and for Moodle community collabora...What’s next for the Quiz and Question bank and for Moodle community collabora...
What’s next for the Quiz and Question bank and for Moodle community collabora...Tim Hunt
 
Question bank improvements in Moodle 4.0 : A successful community collaboration
Question bank improvements in Moodle 4.0 : A successful community collaborationQuestion bank improvements in Moodle 4.0 : A successful community collaboration
Question bank improvements in Moodle 4.0 : A successful community collaborationTim Hunt
 
Looking after the Open University's Moodle
Looking after the Open University's MoodleLooking after the Open University's Moodle
Looking after the Open University's MoodleTim Hunt
 
Embedding questions anywhere in Moodle
Embedding questions anywhere in MoodleEmbedding questions anywhere in Moodle
Embedding questions anywhere in MoodleTim Hunt
 
Hosting STACK at scale
Hosting STACK at scaleHosting STACK at scale
Hosting STACK at scaleTim Hunt
 
Moodle questions without the quiz
Moodle questions without the quizMoodle questions without the quiz
Moodle questions without the quizTim Hunt
 
MUC - Moodle Universal Cache
MUC - Moodle Universal CacheMUC - Moodle Universal Cache
MUC - Moodle Universal CacheTim Hunt
 
Hosting Moodle at the OU
Hosting Moodle at the OUHosting Moodle at the OU
Hosting Moodle at the OUTim Hunt
 
2012 Computer-Assisted Assessment
2012 Computer-Assisted Assessment2012 Computer-Assisted Assessment
2012 Computer-Assisted AssessmentTim Hunt
 
A basic introduction to the Moodle architecture
A basic introduction to the Moodle architectureA basic introduction to the Moodle architecture
A basic introduction to the Moodle architectureTim Hunt
 

Más de Tim Hunt (11)

Writing better Behat tests
Writing better Behat testsWriting better Behat tests
Writing better Behat tests
 
What’s next for the Quiz and Question bank and for Moodle community collabora...
What’s next for the Quiz and Question bank and for Moodle community collabora...What’s next for the Quiz and Question bank and for Moodle community collabora...
What’s next for the Quiz and Question bank and for Moodle community collabora...
 
Question bank improvements in Moodle 4.0 : A successful community collaboration
Question bank improvements in Moodle 4.0 : A successful community collaborationQuestion bank improvements in Moodle 4.0 : A successful community collaboration
Question bank improvements in Moodle 4.0 : A successful community collaboration
 
Looking after the Open University's Moodle
Looking after the Open University's MoodleLooking after the Open University's Moodle
Looking after the Open University's Moodle
 
Embedding questions anywhere in Moodle
Embedding questions anywhere in MoodleEmbedding questions anywhere in Moodle
Embedding questions anywhere in Moodle
 
Hosting STACK at scale
Hosting STACK at scaleHosting STACK at scale
Hosting STACK at scale
 
Moodle questions without the quiz
Moodle questions without the quizMoodle questions without the quiz
Moodle questions without the quiz
 
MUC - Moodle Universal Cache
MUC - Moodle Universal CacheMUC - Moodle Universal Cache
MUC - Moodle Universal Cache
 
Hosting Moodle at the OU
Hosting Moodle at the OUHosting Moodle at the OU
Hosting Moodle at the OU
 
2012 Computer-Assisted Assessment
2012 Computer-Assisted Assessment2012 Computer-Assisted Assessment
2012 Computer-Assisted Assessment
 
A basic introduction to the Moodle architecture
A basic introduction to the Moodle architectureA basic introduction to the Moodle architecture
A basic introduction to the Moodle architecture
 

Último

Patient Counselling. Definition of patient counseling; steps involved in pati...
Patient Counselling. Definition of patient counseling; steps involved in pati...Patient Counselling. Definition of patient counseling; steps involved in pati...
Patient Counselling. Definition of patient counseling; steps involved in pati...raviapr7
 
PISA-VET launch_El Iza Mohamedou_19 March 2024.pptx
PISA-VET launch_El Iza Mohamedou_19 March 2024.pptxPISA-VET launch_El Iza Mohamedou_19 March 2024.pptx
PISA-VET launch_El Iza Mohamedou_19 March 2024.pptxEduSkills OECD
 
Human-AI Co-Creation of Worked Examples for Programming Classes
Human-AI Co-Creation of Worked Examples for Programming ClassesHuman-AI Co-Creation of Worked Examples for Programming Classes
Human-AI Co-Creation of Worked Examples for Programming ClassesMohammad Hassany
 
How to Use api.constrains ( ) in Odoo 17
How to Use api.constrains ( ) in Odoo 17How to Use api.constrains ( ) in Odoo 17
How to Use api.constrains ( ) in Odoo 17Celine George
 
Diploma in Nursing Admission Test Question Solution 2023.pdf
Diploma in Nursing Admission Test Question Solution 2023.pdfDiploma in Nursing Admission Test Question Solution 2023.pdf
Diploma in Nursing Admission Test Question Solution 2023.pdfMohonDas
 
Maximizing Impact_ Nonprofit Website Planning, Budgeting, and Design.pdf
Maximizing Impact_ Nonprofit Website Planning, Budgeting, and Design.pdfMaximizing Impact_ Nonprofit Website Planning, Budgeting, and Design.pdf
Maximizing Impact_ Nonprofit Website Planning, Budgeting, and Design.pdfTechSoup
 
CapTechU Doctoral Presentation -March 2024 slides.pptx
CapTechU Doctoral Presentation -March 2024 slides.pptxCapTechU Doctoral Presentation -March 2024 slides.pptx
CapTechU Doctoral Presentation -March 2024 slides.pptxCapitolTechU
 
2024.03.23 What do successful readers do - Sandy Millin for PARK.pptx
2024.03.23 What do successful readers do - Sandy Millin for PARK.pptx2024.03.23 What do successful readers do - Sandy Millin for PARK.pptx
2024.03.23 What do successful readers do - Sandy Millin for PARK.pptxSandy Millin
 
HED Office Sohayok Exam Question Solution 2023.pdf
HED Office Sohayok Exam Question Solution 2023.pdfHED Office Sohayok Exam Question Solution 2023.pdf
HED Office Sohayok Exam Question Solution 2023.pdfMohonDas
 
DUST OF SNOW_BY ROBERT FROST_EDITED BY_ TANMOY MISHRA
DUST OF SNOW_BY ROBERT FROST_EDITED BY_ TANMOY MISHRADUST OF SNOW_BY ROBERT FROST_EDITED BY_ TANMOY MISHRA
DUST OF SNOW_BY ROBERT FROST_EDITED BY_ TANMOY MISHRATanmoy Mishra
 
CHUYÊN ĐỀ DẠY THÊM TIẾNG ANH LỚP 11 - GLOBAL SUCCESS - NĂM HỌC 2023-2024 - HK...
CHUYÊN ĐỀ DẠY THÊM TIẾNG ANH LỚP 11 - GLOBAL SUCCESS - NĂM HỌC 2023-2024 - HK...CHUYÊN ĐỀ DẠY THÊM TIẾNG ANH LỚP 11 - GLOBAL SUCCESS - NĂM HỌC 2023-2024 - HK...
CHUYÊN ĐỀ DẠY THÊM TIẾNG ANH LỚP 11 - GLOBAL SUCCESS - NĂM HỌC 2023-2024 - HK...Nguyen Thanh Tu Collection
 
General views of Histopathology and step
General views of Histopathology and stepGeneral views of Histopathology and step
General views of Histopathology and stepobaje godwin sunday
 
Patterns of Written Texts Across Disciplines.pptx
Patterns of Written Texts Across Disciplines.pptxPatterns of Written Texts Across Disciplines.pptx
Patterns of Written Texts Across Disciplines.pptxMYDA ANGELICA SUAN
 
How to Add Existing Field in One2Many Tree View in Odoo 17
How to Add Existing Field in One2Many Tree View in Odoo 17How to Add Existing Field in One2Many Tree View in Odoo 17
How to Add Existing Field in One2Many Tree View in Odoo 17Celine George
 
CAULIFLOWER BREEDING 1 Parmar pptx
CAULIFLOWER BREEDING 1 Parmar pptxCAULIFLOWER BREEDING 1 Parmar pptx
CAULIFLOWER BREEDING 1 Parmar pptxSaurabhParmar42
 
What is the Future of QuickBooks DeskTop?
What is the Future of QuickBooks DeskTop?What is the Future of QuickBooks DeskTop?
What is the Future of QuickBooks DeskTop?TechSoup
 
The basics of sentences session 10pptx.pptx
The basics of sentences session 10pptx.pptxThe basics of sentences session 10pptx.pptx
The basics of sentences session 10pptx.pptxheathfieldcps1
 
Practical Research 1 Lesson 9 Scope and delimitation.pptx
Practical Research 1 Lesson 9 Scope and delimitation.pptxPractical Research 1 Lesson 9 Scope and delimitation.pptx
Practical Research 1 Lesson 9 Scope and delimitation.pptxKatherine Villaluna
 
How to Make a Field read-only in Odoo 17
How to Make a Field read-only in Odoo 17How to Make a Field read-only in Odoo 17
How to Make a Field read-only in Odoo 17Celine George
 
How to Show Error_Warning Messages in Odoo 17
How to Show Error_Warning Messages in Odoo 17How to Show Error_Warning Messages in Odoo 17
How to Show Error_Warning Messages in Odoo 17Celine George
 

Último (20)

Patient Counselling. Definition of patient counseling; steps involved in pati...
Patient Counselling. Definition of patient counseling; steps involved in pati...Patient Counselling. Definition of patient counseling; steps involved in pati...
Patient Counselling. Definition of patient counseling; steps involved in pati...
 
PISA-VET launch_El Iza Mohamedou_19 March 2024.pptx
PISA-VET launch_El Iza Mohamedou_19 March 2024.pptxPISA-VET launch_El Iza Mohamedou_19 March 2024.pptx
PISA-VET launch_El Iza Mohamedou_19 March 2024.pptx
 
Human-AI Co-Creation of Worked Examples for Programming Classes
Human-AI Co-Creation of Worked Examples for Programming ClassesHuman-AI Co-Creation of Worked Examples for Programming Classes
Human-AI Co-Creation of Worked Examples for Programming Classes
 
How to Use api.constrains ( ) in Odoo 17
How to Use api.constrains ( ) in Odoo 17How to Use api.constrains ( ) in Odoo 17
How to Use api.constrains ( ) in Odoo 17
 
Diploma in Nursing Admission Test Question Solution 2023.pdf
Diploma in Nursing Admission Test Question Solution 2023.pdfDiploma in Nursing Admission Test Question Solution 2023.pdf
Diploma in Nursing Admission Test Question Solution 2023.pdf
 
Maximizing Impact_ Nonprofit Website Planning, Budgeting, and Design.pdf
Maximizing Impact_ Nonprofit Website Planning, Budgeting, and Design.pdfMaximizing Impact_ Nonprofit Website Planning, Budgeting, and Design.pdf
Maximizing Impact_ Nonprofit Website Planning, Budgeting, and Design.pdf
 
CapTechU Doctoral Presentation -March 2024 slides.pptx
CapTechU Doctoral Presentation -March 2024 slides.pptxCapTechU Doctoral Presentation -March 2024 slides.pptx
CapTechU Doctoral Presentation -March 2024 slides.pptx
 
2024.03.23 What do successful readers do - Sandy Millin for PARK.pptx
2024.03.23 What do successful readers do - Sandy Millin for PARK.pptx2024.03.23 What do successful readers do - Sandy Millin for PARK.pptx
2024.03.23 What do successful readers do - Sandy Millin for PARK.pptx
 
HED Office Sohayok Exam Question Solution 2023.pdf
HED Office Sohayok Exam Question Solution 2023.pdfHED Office Sohayok Exam Question Solution 2023.pdf
HED Office Sohayok Exam Question Solution 2023.pdf
 
DUST OF SNOW_BY ROBERT FROST_EDITED BY_ TANMOY MISHRA
DUST OF SNOW_BY ROBERT FROST_EDITED BY_ TANMOY MISHRADUST OF SNOW_BY ROBERT FROST_EDITED BY_ TANMOY MISHRA
DUST OF SNOW_BY ROBERT FROST_EDITED BY_ TANMOY MISHRA
 
CHUYÊN ĐỀ DẠY THÊM TIẾNG ANH LỚP 11 - GLOBAL SUCCESS - NĂM HỌC 2023-2024 - HK...
CHUYÊN ĐỀ DẠY THÊM TIẾNG ANH LỚP 11 - GLOBAL SUCCESS - NĂM HỌC 2023-2024 - HK...CHUYÊN ĐỀ DẠY THÊM TIẾNG ANH LỚP 11 - GLOBAL SUCCESS - NĂM HỌC 2023-2024 - HK...
CHUYÊN ĐỀ DẠY THÊM TIẾNG ANH LỚP 11 - GLOBAL SUCCESS - NĂM HỌC 2023-2024 - HK...
 
General views of Histopathology and step
General views of Histopathology and stepGeneral views of Histopathology and step
General views of Histopathology and step
 
Patterns of Written Texts Across Disciplines.pptx
Patterns of Written Texts Across Disciplines.pptxPatterns of Written Texts Across Disciplines.pptx
Patterns of Written Texts Across Disciplines.pptx
 
How to Add Existing Field in One2Many Tree View in Odoo 17
How to Add Existing Field in One2Many Tree View in Odoo 17How to Add Existing Field in One2Many Tree View in Odoo 17
How to Add Existing Field in One2Many Tree View in Odoo 17
 
CAULIFLOWER BREEDING 1 Parmar pptx
CAULIFLOWER BREEDING 1 Parmar pptxCAULIFLOWER BREEDING 1 Parmar pptx
CAULIFLOWER BREEDING 1 Parmar pptx
 
What is the Future of QuickBooks DeskTop?
What is the Future of QuickBooks DeskTop?What is the Future of QuickBooks DeskTop?
What is the Future of QuickBooks DeskTop?
 
The basics of sentences session 10pptx.pptx
The basics of sentences session 10pptx.pptxThe basics of sentences session 10pptx.pptx
The basics of sentences session 10pptx.pptx
 
Practical Research 1 Lesson 9 Scope and delimitation.pptx
Practical Research 1 Lesson 9 Scope and delimitation.pptxPractical Research 1 Lesson 9 Scope and delimitation.pptx
Practical Research 1 Lesson 9 Scope and delimitation.pptx
 
How to Make a Field read-only in Odoo 17
How to Make a Field read-only in Odoo 17How to Make a Field read-only in Odoo 17
How to Make a Field read-only in Odoo 17
 
How to Show Error_Warning Messages in Odoo 17
How to Show Error_Warning Messages in Odoo 17How to Show Error_Warning Messages in Odoo 17
How to Show Error_Warning Messages in Odoo 17
 

The Moodle Quiz at the Open University: how we use it & how that helps students

  • 1. The Moodle Quiz at the Open University: how we use it & how that helps students Tim Hunt, Information Technology MoodleMoot Dublin 2015
  • 2. The Moodle Quiz at the Open University: how we use it & how that helps students Tim Hunt, Information Technology MoodleMoot Dublin 2015 many dedicated OU teaching staff
  • 3. Alternate titles Lots of interesting graphs The scholarship of teaching and learning in practice
  • 6. End of module survey results 0% 20% 40% 60% 80% 100% KPI 01: Overall, I am satisfied with the quality of this module. Q33 Clear understanding of what was required to complete the assessed work Q34 the assessment activities supported my learning
  • 7. It makes a difference
  • 8. Level 3 physics (SM358)
  • 9. SM358 assessment tasks 0 TMAs 1 TMA 2 TMAs 3 TMAs 4 TMAs 0 iCMAs 1 iCMA 2 iCMAs 3 iCMAs 4 iCMAs 5 iCMAs 6 iCMAs
  • 10. Students completing tasks 0 TMAs 1 TMA 2 TMAs 3 TMAs 4 TMAs 0 iCMAs 11.6% 3.4% 1.5% 0.5% 0.5% 1 iCMA 1.5% 1.0% 2 iCMAs 1.5% 2.4% 1.5% 3 iCMAs 1.5% 4 iCMAs 5.3% 2.4% 5 iCMAs 0.5% 3.9% 5.8% 8.2% 6 iCMAs 0.5% 0.5% 5.8% 5.8% 34.3%
  • 11. Average exam scores 0 TMAs 1 TMA 2 TMAs 3 TMAs 4 TMAs 0 iCMAs 6.0 1 iCMA 2 iCMAs 17.0 24.0 3 iCMAs 60.0 4 iCMAs 43.7 62.0 5 iCMAs 23.0 46.0 62.6 69.5 6 iCMAs 35.3 60.8 77.5
  • 12. Exam scores vs prediction 0 TMAs 1 TMA 2 TMAs 3 TMAs 4 TMAs 0 iCMAs −20.8 1 iCMA 2 iCMAs −43.9 −27.5 3 iCMAs −9.0 4 iCMAs −15.6 +1.8 5 iCMAs −3.8 −11.1 +1.4 +2.4 6 iCMAs −17.1 +3.4 +4.6
  • 14. T184 robotics & meaning of life Before 10% mid-course iCMA 90% final written EMA part 1 short-answer part 2 programming & essays After 10% mid-course iCMA 30% final iCMA 60% final written EMA
  • 15. Module completion rates T184 completion rates 60% 65% 70% 75% 80% 85% 90% 95% 100% 2004 2005 2006 2007 2008 2009 2010 2011 May Oct Introduction of CME
  • 16. T184 completion rates 60% 65% 70% 75% 80% 85% 90% 95% 100% 2004 2005 2006 2007 2008 2009 2010 2011 May Oct Introduction of CME Module completion rates
  • 17. T184 completion rates 60% 65% 70% 75% 80% 85% 90% 95% 100% 2004 2005 2006 2007 2008 2009 2010 2011 May Oct Introduction of CME Module completion rates
  • 19. SM358 iCMA51 submit date 2010 advisory deadline
  • 20. SM358 iCMA51 submit date 2010 advisory deadline 2012 hard deadline
  • 24. Do students read feedback?
  • 26. Same question in two quizzes
  • 27. Can computers grade sentences?
  • 29. How good are humans? Question Number of responses in analysis Percentage of responses where the human markers were in agreement with question author Percentage of responses where computer marking was in agreement with question author Range for the 6 human markers Mean of the 6 human markers A 189 97.4 – 100. 98.9 99.5 B 248 83.9 – 97.2 91.9 97.6 C 150 80.7 – 94.0 86.9 94.7 D 129 91.5 – 98.4 96.7 97.6 E 92 92.4 – 97.8 95.1 98.9 F 129 86.0 – 97.7 90.8 97.7 G 132 66.7 – 90.2 83.2 89.4
  • 30. Comparing three algorithms Question Number of responses in analysis Percentage of responses where computer marking was in agreement with question author Computational linguistics Algorithmic manipulation of keywords IAT Pattern-match Regular Expressions A 189 99.5 99.5 98.9 B 248 97.6 98.8 98.0 C 150 94.7 94.7 90.7 D 129 97.6 96.1 97.7 E 92 98.9 96.7 96.7 F 129 97.7 88.4 89.2 G 132 89.4 87.9 88.6
  • 32. References • Overall iCMA usage numbers collated by Phil Butcher. • End of module server results from Student Analytics in the Institute of Educational Technology via Linda Price. • SM358 data from John Bolton • T184 data from Jon Rosewell • SDK125, S141, S151 & S240 data from Sally Jordan • Jordan, Sally (2014). Using e-assessment to learn about students and learning. International Journal of e-Assessment, 4(1) • Jordan, Sally (2014). Adult science learners’ mathematical mistakes: an analysis of responses to computer-marked questions. European Journal of Science and Mathematics Education, 2(2) pp. 63–86. • Jordan, Sally (2012). Short-answer e-assessment questions : five years on. In: 2012 International Computer Assisted Assessment Conference, 10-11 July 2012, Southampton. • Pattern-match question type for Moodle https://moodle.org/plugins/view/qtype_pmatch
  • 33. Key points Getting the assessment right is important Online quizzes can be powerful learning tools Dig into the data and you can learn • Is this quiz working? • What are my students’ misconceptions? Computers can grade much more than multi-choice but only on behalf of a teacher Want more? I’m giving this talk again at #iMoot15