Invited presentation given at the OpenStax Creatorfest, April 2018, Houston TX

1. Students as producers
of high quality, engaging assessments to support learning
Simon Bates
simon.bates@ubc.ca @simonpbates bit.ly/batestalks
OpenStax creatorfest – April 2018 – Houston TX

3. April 2016

8. Selected results & analysis
Engagement - how do students use the system?
Benefits - what is the impact on learning?
Question quality - how good is what students produce?
Relevant publications:
Scaffolding student engagement via online peer learning - European Journal of Physics 35 (4), 045002
(2014)
Student-Generated Content: Enhancing learning through sharing multiple-choice questions. International
Journal of Science Education, 1-15 (2014).
Assessing the quality of a student-generated question repository - Phys Rev ST PER (2014) 10, 020105
Student-generated assessment - Education in Chemistry (2013) 13 1

9. Typical implementation
Minimum participation requirements for each of two
assessment exercises (PW1, PW2)
Write 1 Answer 5 Rate / comment 3
5% course credit
Physics 101, Energy & Waves
Winter Semester: 3 sections, ~800 students

10. Not so typical implementation
Writing original questions is a
demanding activity
Extensive scaffolding exercises
Revisited in subsequent
tutorials

12. Engagement with PeerWise
Number Multiplier Number Multiplier
Questions 1105 [1.7] 998 [1.6]
Answers 11393 [17.2] 11807 [18.7]
Comments 4901 [7.4] 5509 [8.7]
PW 1 PW 2

13. Engagement with PeerWise

15. Engagement with PeerWise

16. Engagement with PeerWise

17. Engagement with PeerWise
Generally, students did
• Participate beyond minimum requirements
• Engage in community learning, correcting errors
• Create problems, not exercises
• Provide positive feedback
Generally, students did not
• Contribute trivial or irrelevant questions
• Obviously plagiarize
• Participate much beyond assessment periods
• Leave it to the last minute (sort of….)

19. 19
Correlation with learning

20. 20
Correlation with learning

21. 21
Correlation with learning

22. 22

23. 23
Question/Explanation Quality
Bloom’s Taxonomy of levels in the cognitive domain
Score Level Description
1 Remember Factual knowledge, trivial plugging in of numbers
2 Understand Basic understanding of content
3 Apply Implement, calculate / determine. Typically one-stage
problem
4 Analyze
Typical multi-step problem; requires identification of
strategy
Evaluate
Compare & assess various option possibilities; often
conceptual
Synthesize
Ideas and topics from disparate course sections combined.
Significantly challenging problem.

24. 24
Textp>0.05, NS

25. 25
Question/Explanation Quality
Score Level Description
0 Missing No explanation provided or explanation
incoherent/irrelevant
1 Inadequate Wrong reasoning and/or answer; trivial or flippant
2 Minimal
Correct answer but with insufficient
explanation/justification/ Some aspects may be
unclear/incorrect/confused.
3 Good Clear and detailed exposition of correct method &
answer.
4 Excellent
Thorough description of relevant physics and solution
strategy. Plausibility of all answers considered. Beyond
normal expectation for a correct solution

26. 26

27. 27
Results (UoE 2010-11)
2 successive years of the same course (N=150, 350)
‘High quality’ questions: 78%, 79%
Over 90% (most likely) correct, and 3/5 of those wrong were
identified by students.
69% (2010) and 55% (2011) rated 3 or 4 for explanations
Only 2% (2010) and 4% (2011) rated 1/ 6 for taxonomic level.

28. 28
Bottomley & Denny Biochem and Mol Biol Educ. 39(5) 352-
361 (2011)
107 Year 2 biochem students
56 / 35 / 9 % of questions in lowest 3 levels.
Momsen et al CBE-Life Sci Educ 9, 436-440 (2010)
“9,713 assessment items submitted by 50 instructors in the
United States reported that 93% of the questions asked on
examinations in introductory biology courses were at the
lowest two levels of the revised Bloom’s taxonomy”
Comparison with literature

29. 29

30. 30

31. 31
Resources
Student-facing system
http://peerwise.cs.auckland.ac.nz/
All the research studies referenced and scaffolding materials
referred to are accessible through the PeerWise community
site http://www.peerwise-community.org/

33. Beyond MCQs
Why not short answer Qs?
Why not …. anything?

34. Beyond MCQs
Why not short answer Qs?
Why not …. anything?

35. 35
Hands on!
http://bit.ly/openstax2018