Paper(ACM): http://bit.ly/1GCwK63 We introduce a new grading system, the Cloud Teaching Assistant System (CTAS), as an additional element to instructor grading, peer grading and automated validation in massive open online courses (MOOCs). The grading distributions of the different approaches are compared in an experiment consisting of 476 exam participants. 25 submissions were graded by all four methods. 451 submissions were graded only by peer grading and automated validation. The results of the experiment suggest that both CTAS and peer grading do not simulate instructor grading (Pearson’s correlations: 0.36, 0.39). If the CTAS and not the instructor is assumed to deliver accurate grading, peer grading is concluded to be a valid grading method (Pearson’s correlation: 0.76).

1. On the Validity of Peer Grading
and a Cloud Teaching Assistant
System
Tim Vogelsang - March 2015
Lara Ruppertz
Paper (ACM): http://dx.doi.org/10.1145/2723576.2723633

2. On the Validity of Peer Grading
and a Cloud Teaching Assistant
System
Tim Vogelsang - March 2015
Lara Ruppertz
Paper (ACM): http://dx.doi.org/10.1145/2723576.2723633

3. Q3: How do students’ performances and the grading types differ
amend different scoring criteria?
Q4: How valid are CTAS and peer grading?
MOOC
INTRODUCTION
“MOOC, every letter is negotiable“ by Mathieu Plourde is licensed under CC-BY.

4. Team
INTRODUCTION
4

5. Team
INTRODUCTION
5

6. Team
INTRODUCTION
6

7. On The Validity of Peer
Grading and a Cloud Teaching
Assistant System
On The Validity of Peer
Grading and a Cloud Teaching
Assistant System

8. Peer Grading
INTRODUCTION

9. Peer Grading
INTRODUCTION

10. Peer Grading
INTRODUCTION

11. Peer Grading
INTRODUCTION
11

12. Peer Grading Systems
RELATED WORK

13. Peer Grading Systems
RELATED WORK
SWoRD
Peer Grader

14. On The Validity of Peer Grading and a
Cloud Teaching Assistant
System
On The Validity of Peer Grading and a
Cloud Teaching Assistant
System

15. Cloud Teaching Assistants (CTAs)
METHODS
Platform

16. Cloud Teaching Assistants (CTAs)
METHODS
Platform

17. Cloud Teaching Assistants (CTAs)
METHODS
Platform

18. Cloud Teaching Assistants (CTAs)
METHODS
Platform

19. Cloud Teaching Assistants (CTAs)
METHODS
Platform

20. On The Validity of Peer
Grading and a Cloud Teaching
Assistant System
On The Validity of Peer
Grading and a Cloud Teaching
Assistant System

21. Reliability and Validity
METHODS
„Validity & Reliability“ by Nevit Dilmen is licensed under CC BY-SA 3.0.

22. Reliability and Validity
METHODS
„Validity & Reliability“ by Nevit Dilmen is licensed under CC BY-SA 3.0.

23. Reliability and Validity
INTRODUCTION
Assignment

24. Reliability and Validity
INTRODUCTION
Few Profs
Assignment

25. Reliability and Validity
INTRODUCTION
Many Peers
Few Profs
Assignment

26. Reliability and Validity
INTRODUCTION
Many Peers Some CTAs
Few Profs
Assignment

27. Reliability and Validity
INTRODUCTION
Many Peers Some CTAs
Few Profs
Assignment
Compute Reliability

28. Reliability and Validity
INTRODUCTION
Many Peers Some CTAs
Few Profs
Assignment
Compute Reliability

29. Reliability and Validity
INTRODUCTION
Many Peers Some CTAs
Few Profs
Assignment
Compute Reliability

30. Reliability and Validity
INTRODUCTION
Many Peers Some CTAs
Few Profs
Compute Validity
Assignment
Compute Reliability

31. On The Validity of Peer
Grading and a Cloud Teaching
Assistant System
Experiment Setup & Methods

32. Course: Public Privacy
EXPERIMENT SETUP
Source: 11 pt, regular
32

33. Professor: Anja Mihr
EXPERIMENT SETUP
Screenshot taken from iversity.org platform. Image courtesy Anja Mihr.

34. Sample Size
EXPERIMENT SETUP
Many Peers Some CTAs
Few Profs
Compute Validity
Compute Reliability
Final Exam

35. Many Peers Some CTAs
Few Profs
Compute Validity
Compute Reliability
- 25 paid exam participants
- 16 multiple choice questions
- 476 exam participants
- 4620 active students
- 30k enrolments
- 1 essay question
Sample Size
Final Exam
EXPERIMENT SETUP

36. Many Peers (476) Some CTAs (4)
Few Profs (1)
Final Exam
Compute Validity
Compute Reliability
- 25 paid exam participants
- 16 multiple choice questions
- 476 exam participants
- 4620 active students
- 30k enrolments
- 1 essay question
25
Sample Size
EXPERIMENT SETUP

37. Final Exam
EXPERIMENT SETUP
Screenshot taken from iversity.org platform. Image courtesy Anja Mihr.
Source: 11 pt, regular
37

38. Grading Rubrics
EXPERIMENT SETUP

39. THESIS / CENTRAL IDEA
Grading Rubrics
EXPERIMENT SETUP
I

40. THESIS / CENTRAL IDEA
ORGANISATION
Grading Rubrics
EXPERIMENT SETUP
I
II

41. THESIS / CENTRAL IDEA
ORGANISATION
CONTENT
Grading Rubrics
EXPERIMENT SETUP
I
II
III

42. THESIS / CENTRAL IDEA
ORGANISATION
CONTENT
BALANCED ARGUMENTATION
Grading Rubrics
EXPERIMENT SETUP
I
II
III
IV

43. I
II
III
IV
V
THESIS / CENTRAL IDEA
ORGANISATION
CONTENT
BALANCED ARGUMENTATION
SOURCES
Grading Rubrics
EXPERIMENT SETUP

44. Q3: How do students’ performances and the grading types differ
amend different scoring criteria?
Q4: How valid are CTAS and peer grading?
Research Questions
METHODS

45. Q1: How are grades given by peers, the CTAs and the instructor
distributed?
Q3: How does the grading differ between the grading rubrics?
Q4: How valid are CTAS and peer grading?
Research Questions
METHODS

46. Q1: How are grades given by peers, the CTAs and the instructor
distributed?
Q2: How does the grading vary among the grading rubrics?
Q4: How valid are CTAS and peer grading?
Research Questions
METHODS

47. Q1: How are grades given by peers, the CTAs and the professor
distributed?
Q2: How does the grading vary among the grading rubrics?
Q3: How valid are peer and CTA grading?
Research Questions
METHODS

48. On The Validity of Peer
Grading and a Cloud Teaching
Assistant System
Result
s

49. Q1: How are grades given by peers, the CTAs and
the professor distributed?

50. RESULTS
Histogram of grades given by peers, CTAs and
professor
Range: 0 (worst grade) - 4 (best grade)
Grade
CTAs
Peers
Professor
How are grades distributed?

51. Q2: How does the grading
vary among the grading rubrics?

52. RESULTS
Thesis Organisation Content Argumentation Sources
Histogram of grades given by peers, CTAs and professor.
Range: 0 (worst grade) - 4 (best grade).
Distinguished by grading rubrics.
CTAs
Peers
Professor
Grade
How does the grading vary among the rubrics?
52

53. RESULTS
Average grade assigned by peers, CTAs and professor to a specific criterion.
Range: 0 (worst grade) - 4 (best grade).
Peers CTAs Professor
How does the grading vary among the rubrics?
53

54. Q3: How valid are peer and CTA grading?

55. Many Peers (476) Some CTAs (4)
Few Profs (1)
Final Exam
Compute Validity
Compute Reliability
- 25 paid exam participants
- 16 multiple choice questions
- 476 exam participants
- 4620 active students
- 30k enrolments
- 1 essay question
25
RESULTS
How valid are peer and CTA grading?

56. RESULTS
How valid are peer and CTA grading?
Correlation matrix using Pearson’s correlation.
375 data points: (25 exams) x (5 criteria) x (3 grading groups).

57. RESULTS
Scatterplot : One dot represents one
final exam graded by peers, CTAs
and professor.
Range: 0 (worst grade) - 4 (best grade)
Average CTA Grade
How valid are peer and CTA grading?

58. RESULTS
How valid are peer and CTA grading?
Correlation matrix using Pearson’s correlation.
375 data points: (25 exams) x (5 criteria) x (3 grading groups).

59. RESULTS
How valid are peer and CTA grading?
Peers CTAs Prof.
Peers
CTAs
Prof.
Correlation matrix using Pearson’s correlation.
375 data points: (25 exams) x (5 criteria) x (3 grading groups).

60. Many Peers (476) Some CTAs (4)
Few Profs (1)
Final Exam
Compute Validity
Compute Reliability
- 25 paid exam participants
- 16 multiple choice questions
- 476 exam participants
- 4620 active students
- 30k enrolments
- 1 essay question
25
RESULTS
How valid are peer and CTA grading?

61. TheCooperReview.com.Image courtesy: Sarah Cooper.
61

62. DISCUSSION
Hypotheses / Future Work
Repeat Experiment Adjust Experiment
62

63. Hypotheses / Future Work
DISCUSSION
Repeat Experiment Adjust Experiment
Disagreement on
‚argumentation‘?
63

64. Hypotheses / Future Work
DISCUSSION
Repeat Experiment Adjust Experiment
Disagreement on
‚argumentation‘?
Peer’s bias?
64

65. Hypotheses / Future Work
DISCUSSION
Repeat Experiment Adjust Experiment
Disagreement on
‚argumentation‘?
Peer’s bias?
Advanced CTA
selection process.
65

66. Hypotheses / Future Work
DISCUSSION
Repeat Experiment Adjust Experiment
Disagreement on
‚argumentation‘?
Peer’s bias?
Advanced CTA
selection process.
Peer calibration
process.
66

67. How to define
validity in the
context of massive
grading?
Hypotheses / Future Work
DISCUSSION
Repeat Experiment Adjust Experiment
Disagreement on
‚argumentation‘?
Peer’s bias?
Advanced CTA
selection process.
Peer calibration
process.
67

68. Disagreement on
‚argumentation‘?
Peer’s bias?
Advanced CTA
selection process.
Peer calibration
process.
How to define
validity in the
context of massive
grading?
How to design a valid grading
process including professor,
peers and CTAs?
Hypotheses / Future Work
DISCUSSION
Repeat Experiment Adjust Experiment
68

69. Hypotheses / Future Work
DISCUSSION
69

70. Thank you!
t.vogelsang@iversity.org
l.ruppertz@iversity.org