NOVICE summer school 2011 (www.noviceproject.eu) Workshop: E-assessment

1. Jan Ehlers E-Assessment

3. Someexperiences

4. Whatcouldbedoneelectronically?

7. formative

9. Diagnostic/formative/summative

10. Changingflipchartsevery 3-5 minutes26.08.2011 5

11. 26.08.2011 6 RUEDEL C (2009): Was ist eAssessment? Hamburger eLMAGAZIN (02), 22-24

12. Traditional vs. electronical These examsarequitethe same: Traditional Electronical Minutes / term papers ---------- Written tests (open or closed questions) ---- Oral exams ------------------------ Portfolios --------------------------- Peer-assessment ---------------- Scientific practicals -------------- Study diary ------------------------ 26.08.2011 7 Electronic submissions E-assessment, e-exam Adaptive testing E-portfolios Peer-assessment Simulations Weblog

13. Traditional vs. electronical These examsarequitesimilar: Traditional Electronical Oral collaboration ---------------- Group examination -------------- Presentations---------------------- 26.08.2011 8 Discussionboards/voting Wikis Digital storytelling

15. 883 firstyearstudentsfrom 6,553 applicants

16. 180 items/30 min

18. What are your exams like? 0 1 - Oral exams 2 - Practicalexams 3 - Writtenexams 4 - Electronic Exams 5 - Essay questions 6 - Multiple choice 7 - Others

19. What are your exams like? 26.08.2011 12 0% 1 - Oral exams 0% 2 - Practicalexams 0% 3 - Writtenexams 0% 4 - Electronic Exams 5 - Essay questions 0% 6 - Multiple choice 0% 7 - Others 0%

20. You want your students to document their EMS and present these documentations afterwards to others. Which tool could help you here? 26.08.2011 13 0 1 - Online surveys 2 - Wikis 3 - Discussionboards 4 - Blogs 5 - Portfolios 6 - SocialBookmarking 7 - Facebook

21. You want your students to document their EMS and present these documentations afterwards to others. Which tool could help you here? 26.08.2011 14 1 - Online surveys 0% 2 - Wikis 0% 0% 3 - Discussionboards 4 - Blogs 0% 5 - Portfolios 0% 6 - SocialBookmarking 0% 0% 7 - Facebook 26.08.2011 14

22. Summative assessment Q [kju:] (Codiplan) 26.08.2011 15 Start: August 2008 126 exams 25,443 individual exams 9,602 questions Partlyblendedassessment EHLERS JP, CARL T, WINDT K-H, MÖBS D, REHAGE J, TIPOLD A: BlendedAssessment: Mündliche und elektronische Prüfungen im klinischen Kontext. In: Zeitschrift für Hochschulentwicklung 4, 3 (2009) 24-36

23. Requirementsfor summative assessments Quality criteria 26.08.2011 16 Utility Testfairness Testeconomy Transparency Reasonabilty Objectivity Reliability Validity

24. What is for you most important? 26.08.2011 17 00 1 - Objectivity 2 - Reliability 3 - Validity 4 - Utility 5 - Testfairness 6 - Testeconomy 7 - Transparency 8 - Reasonability

25. What is for you most important? 26.08.2011 18 1 - Objectivity 0% 2 - Reliability 0% 3 - Validity 0% 4 - Utility 0% 5 - Testfairness 0% 6 - Testeconomy 0% 7 - Transparency 0% 8 - Reasonability 0% 26.08.2011 18

27. Advatages/disadvantages/“visions“

28. Changingflipchartsevery 3-5 minutes26.08.2011 19

29. Students´ view

30. Acceptance summative e-assessment 26.08.2011 21 Was the system easy to use? (n = 5328, mean: 1,8) 1 2 3 How do you rate thewholeexam? (n = 5155, mean: 3,0) 1 2 3 4 5 6 (1= „yes, very, a lot, verygood“ to 6= „no, not at all, nothing, verybad“)

31. Acceptance summative e-assessment 26.08.2011 22 Do youtrust in theusedtechniques? (n = 5268, mean: 3,0) 1 2 3 4 5 6 Doesitmake sense touse an electronic systemforwrittentests? (n = 5249, mean: 3,1) 4 5 6 1 2 3 (1= „yes, very, a lot, verygood“ to 6= „no, not at all, nothing, verybad“)

32. Acceptance summative e-assessment 26.08.2011 23 Doesitmake sense touse MCQs? (n = 5283, mean: 3,7) 1 2 3 4 5 6 Shouldtherebemorewrittenthan oral exams? (n = 5228, mean: 3,8) 1 2 3 4 5 6 Do youthingwrittenexamsaremor fair than oral exams? (n = 5217, mean: 3,1) 1 2 3 4 5 6 (1= „yes, very, a lot, verygood“ to 6= „no, not at all, nothing, verybad“)

33. Post-review Statisticsforeach item 26.08.2011 24

34. Post-review Statisticsforthewholeexam 26.08.2011 25

35. Post-review Log-files 26.08.2011 26

36. Post-review Item-database 26.08.2011 27

37. Pyramide ofcompetencies (MILLER) 26.08.2011 28 Perfomance„in vivo“: MiniCEX (undercov. stand. patients) Does Performance „in vitro“: OSCE (stand. patients) Shows Prozedurale knowledge: Key-feature, oral exams Knows how Deskriptive/ elaborativeKnowledge: MCQ, oral exams Knows MILLER GE (1990): The Assessment of Clinical Skills/Competence/Performance. AcadMed 65, 63-67

38. Proceduraleknowledge Useofmedia 26.08.2011 29

39. Proceduraleknowledge Picture analysis 26.08.2011 30

40. Proceduraleknowledge/skills Botany „o-test“ 26.08.2011 31 E-exampartlywith„peer-generated“ MCQ Identification-exercise in theuniversity´sbotanicalgarden ABOLING S, WINDT K-H, POHL D, EHLERS JP: Lehr- und Prüfungsmethoden im Fach veterinärmedizinische Botanik mit besonderer Berücksichtigung des Konzepts o-test. ZFHE 6/1, 19-33

41. Last days…. 26.08.2011 32 Histology Botany

42. Key-featureassessment 26.08.2011 33 Answer question 1 Answer question 2 Continuationvignette Continuationvignette Vignette Question 3 Question 1 Question 2 SCHAPER E, FISCHER MR, TIPOLD A, EHLERS JP: Fallbasiertes, elektronisches Lernen und Prüfen in der Tiermedizin - auf der Suche nach einer Alternative zu Multiple-Choice Prüfungen. Tierärztl. Umschau 66, 261-268

43. Key-featureexample 26.08.2011 34 11 KFQ, 40 examinees -> Cronbach´sα = 0,68

44. VP-Assessment Vetsuisse-Faculty University Bern European School ofAdvancedVeterinary Studies Examinationswith CASUS About 15-20 cards per case 2-5 cases per exam 26.08.2011 35

45. Questionformats MCQ TypeA und PickN Short Answer Long Menu Picture analysis (fill-in-the-blank) (drag ´n drop)

47. Time: about 15 minutes26.08.2011 37

48. Whatis an ideal exam?

50. Definedlistoflearninggoals

51. Congruenceoflearning/teachingandassessment26.08.2011 39

53. Just portfolios

54. Ifexams: just unheralded26.08.2011 40

56. Time neartolectures

57. Objective (w/o fear)

58. Interdisciplinary, broadknowledge

59. Feedback forteachersandlearners

60. Progress test, OSCE, kea-feature, portfolio

61. Noexams – evaluationduringcourses26.08.2011 41

62. Conclusions… E-assessmentispossible! Regardtheir potential!(andcreateyourvision) Develop e-assessmentfurther! (don´tonlycopywrittentests) 26.08.2011 42

63. Thankyouforyourattention! 26.08.2011 43 Contact Dr. Jan P. Ehlers, M.A. E-learningconsultant University ofVeterinaryMedicine Hannover Bünteweg 2 – D-30559 Hannover +49-511-953-8054 jan.ehlers@tiho-hannover.de www.tiho-hannover.de

65. Veterinarystudents

66. ICT education26.08.2011 44