A pilot study in a large enrolment
introductory science course
Expert-guided crowdsourced
learning content
Simon Bates
sim...
~2010
~2006
2013
Context - flipped classrooms
CC BY-NC 2.0 https://flic.kr/p/f3ynHx
Derek Bruff: Class time reconsidered
http://prezi.com/do...
Weekly rhythm for our 1A class
Poh,	
  M.Z.,	
  Swenson,	
  N.C.,	
  Picard,	
  R.W.,	
  "A	
  Wearable	
  Sensor	
  for	
...
Weekly rhythm for our 1A class
Weekly rhythm for our 1A class
Learning gains on PI
Learning gains on PI
1
2
3
A ball initially at rest is
thrown upwards, comes
back down & is caught
!
Which of the following is a
plausible grap...
Freeman et al PNAS www.pnas.org/cgi/doi/10.1073/pnas.1319030111	

Wieman commentary PNAS
http://www.pnas.org/cgi/doi/10.10...
~2010
~2006
2013
a"web&based"MCQ"repository"created"by"students"
Ins$tu$ons((signing(up(per(year:(
!
2009:! ! !22(
2010:! ! !66(
2011:! ! !204(
2012:! ! !266(
2013!(Jan,Jun):! !214(
Growi...
Student'ownership'over'
learning'resource'
Student'familiarity'with'
social'so7ware'
Leveraging'student'
energy'and'crea9v...
Badges' Points' Leaderboards'
Typical implementation
Minimum participation requirements for each
of two assessment exercises (PW1, PW2)
Write 1 Answer 5...
Scaffolding
TOCCLT'.'
-
QestIsct
How 1o...
q/t4osTer
Submit ond onswer questions
on topics in lhe torget region,
just obov...
Engagement with PeerWise
Number Multiplier Number Multiplier
Questions 1105 [1.7] (1.4) 998 [1.6] (1.3)
Answers 11393 [17....
Engagement with PeerWise
Engagement with PeerWise
PW1 PW2
Contributed and met minimum requirements
Contributed but did not meet mininum requirement...
Question/Explanation Quality
Bloom’s Taxonomy of levels in the cognitive domain
Score Level Description
1 Remember Factual...
1 2 3 4 1 2 3 4
0
20
40
60
Numberofquestions
Assessment 1 Assessment 2
Bloom's Taxonomy: Question Quality
Textp>0.05, NS
0...
Question/Explanation Quality
Description of explanation quality
Score Level Description
0 Missing
No explanation provided ...
0 1 2 3 4 0 1 2 3 4
0
20
40
60
Numberofquestions
Assessment 1 Assessment 2
Explanation Quality
p=0.02
~2010
~2006
2013
PHYS101:
Energy and Waves
Logistics
!
!
Cohort split into 4 groups	

!
Each week one group tasked with creating LOs	

!
Each submission counts for 2...
Results: engagement
0 100 200 300
LO 1
LO 2
LO 3
LO 4
LO 5
LO 6
LO 7
LO 8
Number of students
Assigned
Optional
0 10 20 30 40 50 60 70 80 90 100
0
100
200
300
Grade (%)
Numberofstudents LO Grade Distribution
0 10 20 30 40 50 60 70 80 90 100
0
100
200
300
Grade (%)
Numberofstudents LO Grade Distribution
These students
didn’t even...
Which students aren’t submitting LOs?
Non-participants are from all walks of life!
0 10 20 30 40 50 60 70 80 90 100
0
5
10...
Results: types of Los
PPT
DOC
Prezi
Ted-Ed
PDF
Image
Other
Results: time on task
0 100 200 300 400
Less than 0.5h
0.5 to 1 h
1 to 2h
2 to 3h
3 to 4h
4 to 5h
More than 5h
Number of s...
Results: self reported change in
understanding
0 200 400 600 800
None
Little
Moderate
Good
Excellent
Number of students
02...
PHYS101:
Energy and Waves
3. Successes	

http://youtu.be/BObyt_NJYrE
Sample 2 - Standing Wave in a bowl
Sample 3 - Colour Loss Underwater
Student generated exam content
3. Successes
simon.bates@ubc.ca @simonpbates bit.ly/batestalks
Relevant publications:
!
Scaffolding student engagement via online peer ...
Image credits
Crowd http://www.flickr.com/photos/18378655@N00/613445810/
Expert Guided Crowd Sourced Learning Content
Expert Guided Crowd Sourced Learning Content
Expert Guided Crowd Sourced Learning Content
Expert Guided Crowd Sourced Learning Content
Expert Guided Crowd Sourced Learning Content
Expert Guided Crowd Sourced Learning Content
Expert Guided Crowd Sourced Learning Content
Expert Guided Crowd Sourced Learning Content
Expert Guided Crowd Sourced Learning Content
Expert Guided Crowd Sourced Learning Content
Expert Guided Crowd Sourced Learning Content
Próxima SlideShare
Cargando en…5
×

Expert Guided Crowd Sourced Learning Content

1.444 visualizaciones

Publicado el

Keynote presented at the Improving University Teaching Conference (http://www.iutconference.com/) Vancouver July 2014

Publicado en: Educación
  • Sé el primero en comentar

  • Sé el primero en recomendar esto

Expert Guided Crowd Sourced Learning Content

  1. 1. A pilot study in a large enrolment introductory science course Expert-guided crowdsourced learning content Simon Bates simon.bates@ubc.ca @simonpbates bit.ly/batestalks
  2. 2. ~2010 ~2006 2013
  3. 3. Context - flipped classrooms CC BY-NC 2.0 https://flic.kr/p/f3ynHx Derek Bruff: Class time reconsidered http://prezi.com/donq036eunko/class-time-reconsidered/
  4. 4. Weekly rhythm for our 1A class Poh,  M.Z.,  Swenson,  N.C.,  Picard,  R.W.,  "A  Wearable  Sensor  for  Unobtrusive,  Long-­‐term  Assessment  of  Electrodermal  AcEvity,"  IEEE  TransacEons  on  Biomedical  Engineering,  vol.57,  no.5,  pp.1243-­‐1252,     May  2010.  doi:  10.1109/TBME.2009.2038487  
  5. 5. Weekly rhythm for our 1A class
  6. 6. Weekly rhythm for our 1A class
  7. 7. Learning gains on PI
  8. 8. Learning gains on PI
  9. 9. 1 2 3 A ball initially at rest is thrown upwards, comes back down & is caught ! Which of the following is a plausible graph of the acceleration of the ball with time?
  10. 10. Freeman et al PNAS www.pnas.org/cgi/doi/10.1073/pnas.1319030111 Wieman commentary PNAS http://www.pnas.org/cgi/doi/10.1073/pnas.1407304111 “This meta-analysis makes a powerful case that any college or university that is teaching its STEM courses by traditional lectures is providing an inferior education to its students”
  11. 11. ~2010 ~2006 2013
  12. 12. a"web&based"MCQ"repository"created"by"students"
  13. 13. Ins$tu$ons((signing(up(per(year:( ! 2009:! ! !22( 2010:! ! !66( 2011:! ! !204( 2012:! ! !266( 2013!(Jan,Jun):! !214( Growing(content(repository:( ! Courses:! !2,500( Logins/month:! !75,000( Ques<ons:! !600,000( Answers:! !12,000,000( Answers (20,000,000) Questions (1,000,000)
  14. 14. Student'ownership'over' learning'resource' Student'familiarity'with' social'so7ware' Leveraging'student' energy'and'crea9vity'
  15. 15. Badges' Points' Leaderboards'
  16. 16. Typical implementation Minimum participation requirements for each of two assessment exercises (PW1, PW2) Write 1 Answer 5 Rate / comment 3 5% course credit Physics 101, Energy & Waves Winter Semester: 3 sections, 791 students
  17. 17. Scaffolding TOCCLT'.' - QestIsct How 1o... q/t4osTer Submit ond onswer questions on topics in lhe torget region, just obove lhe physics you hove olreody mostered. Dislroclers ) l*"*"ò rNrrltng Ih s region corrk:ins llte plrysics knowledgo ond colrco¡ls you connol leorn yel becouse the louncalions crrc nol in plocc PHYSICS TOPICS IN YOUR TARG T REGION buo'5ot't1 'Den¡¡lu B¿:ovvì 1 J F:rcq - YOUR CHOSEN TOPIC à.. auC'q^ x@ bôo'osro pn ro, d logroo, o 9tS COMMON MISCONCEPÎIONS AND ERRORS (Sec hHÞ:/jÞhy¡.udofo¡.cdu/CJP/trêconcoÞllo¡t.pdt tor o llil ol common mhconcepllons) ¿K N(,qht %cce Açxs no exi¡þ , c.rrrçrJ On oblec'-t 5 õÞYJrnr cìü?À I in rtuiò. ü lrh'brot4ont eprç,e .I¡1enÀs crq den-s,{^ c>Ç c.,þ¡ec , cìo+ .*¿n:r oÇ Çtu,8, fr- btrrqon! $crCe c¿cs Àoc¡:'cr fr¿lt{e- r *hcrn ú? Trr¿e. $ììfifflsz hrrr: ¿lÕoolr-g lnnS (**= tcpo þ/rnl {otr-* qrÇ bq.:. 0" Ooz Physics knowledge ond conccpluol underslonding you lrove olrcody conslrucleC ¡n your heod ¡,<'t.lerJ rt)^,¿z! Þ.rrp r- rr't¡ ,l? Chcck lhol yovr on3Íêr ¡t rêo!ànoblê ðnd potrlble 'f<,,cc Photo by Seth Casteel http://www.littlefriendsphoto.com Permission to use agreed Introduced in tutorials ! Extensive scaffolding exercises ! Revisited in subsequent tutorials ! Tutorials delivered by 24 TAs
  18. 18. Engagement with PeerWise Number Multiplier Number Multiplier Questions 1105 [1.7] (1.4) 998 [1.6] (1.3) Answers 11393 [17.2] (14.4) 11807 [18.7] (15.0) Comments 4901 [7.4] (6.2) 5509 [8.7] (5.0) PW 1 PW 2
  19. 19. Engagement with PeerWise
  20. 20. Engagement with PeerWise PW1 PW2 Contributed and met minimum requirements Contributed but did not meet mininum requirements Did not participate 0.84 0.80 Total: 791
  21. 21. Question/Explanation Quality Bloom’s Taxonomy of levels in the cognitive domain Score Level Description 1 Remember Factual knowledge, trivial plugging in of numbers 2 Understand Basic understanding of content 3 Apply Implement, calculate / determine. Typically one-stage problem 4 Analyze Typical multi-step problem; requires identification of strategy Evaluate Compare &assess various option possibilities; often conceptual Synthesize Ideas and topics from disparate course sections combined. Significantly challenging problem. Text
  22. 22. 1 2 3 4 1 2 3 4 0 20 40 60 Numberofquestions Assessment 1 Assessment 2 Bloom's Taxonomy: Question Quality Textp>0.05, NS 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 1 2 3 4 5 6 Taxonomic Category PercentageofSubmittedQuestions First semester N = 350 Second semester N = 252 Bates et al, PRST PER, in press 2014 http://arxiv.org/abs/1308.2202
  23. 23. Question/Explanation Quality Description of explanation quality Score Level Description 0 Missing No explanation provided or explanation incoherent/ irrelevant 1 Inadequate Wrong reasoning and/or answer; trivial or flippant 2 Minimal Correct answer but with insufficient explanation/ justification/ Some aspects may be unclear/incorrect/ confused. 3 Good Clear and detailed exposition of correct method & ans 4 Excellent Thorough description of relevant physics and solution strategy. Plausibility of all answers considered. Beyond normal expectation for a correct solution
  24. 24. 0 1 2 3 4 0 1 2 3 4 0 20 40 60 Numberofquestions Assessment 1 Assessment 2 Explanation Quality p=0.02
  25. 25. ~2010 ~2006 2013
  26. 26. PHYS101: Energy and Waves
  27. 27. Logistics ! ! Cohort split into 4 groups ! Each week one group tasked with creating LOs ! Each submission counts for 2.5% of final grade ! Repeat cycle twice per Semester ! Students can submit >2 LOs & receive grade for best 2 ! Short survey on submission ! Students encourage to apply CC licenses
  28. 28. Results: engagement 0 100 200 300 LO 1 LO 2 LO 3 LO 4 LO 5 LO 6 LO 7 LO 8 Number of students Assigned Optional
  29. 29. 0 10 20 30 40 50 60 70 80 90 100 0 100 200 300 Grade (%) Numberofstudents LO Grade Distribution
  30. 30. 0 10 20 30 40 50 60 70 80 90 100 0 100 200 300 Grade (%) Numberofstudents LO Grade Distribution These students didn’t even try! 0 10 20 30 40 50 60 70 80 90 100 0 100 200 300 Grade (%) Numberofstudents LO Grade Distribution
  31. 31. Which students aren’t submitting LOs? Non-participants are from all walks of life! 0 10 20 30 40 50 60 70 80 90 100 0 5 10 15 20 25 Final Exam (%) Numberofstudents Final Exam marks of students that did not participate in LOs
  32. 32. Results: types of Los PPT DOC Prezi Ted-Ed PDF Image Other
  33. 33. Results: time on task 0 100 200 300 400 Less than 0.5h 0.5 to 1 h 1 to 2h 2 to 3h 3 to 4h 4 to 5h More than 5h Number of students
  34. 34. Results: self reported change in understanding 0 200 400 600 800 None Little Moderate Good Excellent Number of students 0200400600800 Number of students
  35. 35. PHYS101: Energy and Waves 3. Successes http://youtu.be/BObyt_NJYrE
  36. 36. Sample 2 - Standing Wave in a bowl
  37. 37. Sample 3 - Colour Loss Underwater
  38. 38. Student generated exam content 3. Successes
  39. 39. simon.bates@ubc.ca @simonpbates bit.ly/batestalks Relevant publications: ! Scaffolding student engagement via online peer learning - European Journal of Physics 35 (4), 045002 (2014) ! Student-Generated Content: Enhancing learning through sharing multiple-choice questions. International Journal of Science Education, 1-15 (2014). ! Assessing the quality of a student-generated question repository - Phys Rev ST PER (2014) in press ! Student-generated assessment - Education in Chemistry (2013) 13 1
  40. 40. Image credits Crowd http://www.flickr.com/photos/18378655@N00/613445810/

×