Predicting Peer-Review Participation at Large
Scale Using an Ensemble Learning Method
Erkan Er | Eduardo Gómez-Sánchez | M...
Peer Reviews in MOOCs
• Learning benefits for both parties [1, 2].
• Impossible to assess thousands of student artefacts.
...
Peer Reviews in MOOCs
• Problems with peer reviews at large scale
LOW
PARTICIPATION
Diversity among
MOOC Learners No Instr...
Proposed Solution and Motivation
4
Predicting student engagement in peer-reviews
The number
of peer work
that students
wil...
Proposed Solution and Motivation
5
Predicting student engagement in peer-reviews
Peer reviews #1 Peer reviews #2 Peer revi...
Proposed Solution and Motivation
Effective peer-review
sessions
Effective collaborative
learning activities
• Peer reviews...
Our preceding work
• Predicting Student Participation in Peer Reviews in MOOCs
5th European MOOCs Stakeholders Summit,
Mad...
Our preceding work
• Limitations:
• The model was built with
• A large part of the error was accounted for students:
8
Ass...
Current Work
• Reduced yet predictive feature set
• Helps minimize the probability of overfit,
• Enhances transferability
...
Current Work
• Students with ZERO participation in peer reviews
10
REGRESSION
MODEL
PREDICTIONS
PERFORMANCE
ALL DATA
STUDE...
Context
• Canvas Network course (from dataverse)
• No contextual information regarding the courses,
• we do not know their...
Context
• 3567 enrollments
• 4 assignments requiring peer reviews
12
Feature Generation
Discussion features
disc_activity_count,
disc_active_days,
disc_avg_time_inbtwn,
disc_entry_count,
disc...
Method
• 10-fold cross-validation,
• LASSO as the regression method,
• Logistic regression as the classifier,
• Scikit-lea...
Results
15
SESSION APPROACH 0 1 2 3 4 TOTAL PREV
1st Peer
Reviews
Regression 2.06 1.08 0.24 0.75 1.68 1.04
1.02
Ensemble 2...
Discussion
• Better prediction performance with ensemble approach
• A classification phase before the regression model
• S...
Discussion
• However, error increased in the prediction of other levels
• Due to poor classification performance.
• More w...
PEER-REVIEW PREDICTION
PART II
18
Classification and Transferability
Classification: Under-participation vs…
Recommended
participation
[e.g., 3 reviews]
19
Practicality!
20
Course
Start
Course
End
PREDICTION
MODEL
Practicality!
• Experimentation with post-hoc predictions [3]
• Useful in demonstrating relationships
• Exam scores  Drop...
Practicality!
• Using only the information available at the time of
prediction,
• Making prediction when it is useful to i...
Transfer learning: Practical predictions
• In-situ learning: Transferring a model across different weeks
23
Course
start
n...
Transfer learning: Practical predictions
• Across contexts: Transferring a model across different courses
Course A
Course ...
Context
• Course #1
• 3567 enrollments
• 4 peer-review sessions
25
Context
• Course #2
• 3632 enrollments
• 4 peer-review sessions
26
Context
• Course #3
• 5248 enrollments
• 7 peer-review sessions
27
Feature Generation
Discussion entries
dentry_count,
dentry_charcount_mean
dentry_charcount_ttl,
dentry_depth_mean,
dentry_...
Method
• 10-fold cross-validation vs transfer learning approaches,
• Logistic regression as the classifier,
• Scikit-learn...
Results
Course #1
Course #2
Course #3
30
1st Peer reviews 2nd Peer reviews 3rd Peer reviews 4th Peer reviews
CV - 0.629 0....
Results
• Transfer between Course #1 and Course #2
31
Using Course #1 to predict Course #2 on a weekly base
Week #1 Week #...
Results
• Transfer between Course #1 and Course #2
32
Using Course #2 to predict Course #1 on a weekly base
Week #2 Week #...
Discussion
• Using in-situ approach, the classification model has potential to be
utilized in an ongoing MOOC.
• Transfera...
Discussion
• Transferable in both directions, but accuracy was not the same
• Training using whole course seems to perform...
Future Work
Limitation -> Context was UNKNOWN
Connecting with Learning Design
• Replicating the model in one of our own MO...
Future Work
• Using this classification model in practice
• to produce a relevant input to the group-formation task,
• to ...
References
[1] Topping, K.: Peer assessment between students in colleges and universities. Rev. Educ. Res.
68, 249–276 (19...
Thanks!
• Questions?
38
Próxima SlideShare
Cargando en…5
×

VII Jornadas eMadrid "Education in exponential times". Erkan Er: "Predicting Peer-Review Participation at Large Scale Using an Ensemble Learning Method". 04/07/2017.

33 visualizaciones

Publicado el

VII Jornadas eMadrid "Education in exponential times". Erkan Er: "Predicting Peer-Review Participation at Large Scale Using an Ensemble Learning Method". 04/07/2017.

Publicado en: Educación
  • Sé el primero en comentar

  • Sé el primero en recomendar esto

VII Jornadas eMadrid "Education in exponential times". Erkan Er: "Predicting Peer-Review Participation at Large Scale Using an Ensemble Learning Method". 04/07/2017.

  1. 1. Predicting Peer-Review Participation at Large Scale Using an Ensemble Learning Method Erkan Er | Eduardo Gómez-Sánchez | Miguel Luis Bote-Lorenzo Yannis Dimitriadis | Juan Ignacio Asensio-Pérez Project ID: VA082U16, Funded by Junta de Castilla y León. GSIC-EMIC Research Group, Universidad de Valladolid
  2. 2. Peer Reviews in MOOCs • Learning benefits for both parties [1, 2]. • Impossible to assess thousands of student artefacts. • Receive constructive feedback, • BUT ALSO get graded to continue Feedback to improve! Higher-order thinking! 2
  3. 3. Peer Reviews in MOOCs • Problems with peer reviews at large scale LOW PARTICIPATION Diversity among MOOC Learners No Instructor Facilitation DROPOUTS UNGRADED SUBMISSIONS MISSED LEARNING BENEFITS 3
  4. 4. Proposed Solution and Motivation 4 Predicting student engagement in peer-reviews The number of peer work that students will review
  5. 5. Proposed Solution and Motivation 5 Predicting student engagement in peer-reviews Peer reviews #1 Peer reviews #2 Peer reviews #3 Peer reviews #4
  6. 6. Proposed Solution and Motivation Effective peer-review sessions Effective collaborative learning activities • Peer reviews based on the expected level of engagement • Allotting adaptive time periods • Providing incentives to motivate students • Inter-homogeneous groups in terms of having some members • with desire to review teammates’ work 6
  7. 7. Our preceding work • Predicting Student Participation in Peer Reviews in MOOCs 5th European MOOCs Stakeholders Summit, Madrid, Spain, May 2017.
  8. 8. Our preceding work • Limitations: • The model was built with • A large part of the error was accounted for students: 8 Assignment time Peer-review time versus Particular to the context. A large feature set Overfit small data set
  9. 9. Current Work • Reduced yet predictive feature set • Helps minimize the probability of overfit, • Enhances transferability 9 MOOC 1
  10. 10. Current Work • Students with ZERO participation in peer reviews 10 REGRESSION MODEL PREDICTIONS PERFORMANCE ALL DATA STUDENTS WITH ZERO PEER REVIEW STUDENTS WITH PEER REVIEWS CLASSIFICATION MODEL ALL DATA
  11. 11. Context • Canvas Network course (from dataverse) • No contextual information regarding the courses, • we do not know their learning design, • we do not know the content/purpose of the activities. • Therefore, we make some inferences based on the log data at hand. 11
  12. 12. Context • 3567 enrollments • 4 assignments requiring peer reviews 12
  13. 13. Feature Generation Discussion features disc_activity_count, disc_active_days, disc_avg_time_inbtwn, disc_entry_count, disc_charcount_mean disc_entry_days, 13 Quiz features quiz_activity_count, quiz_active_days, quiz_avg_time_inbtwn, quiz_counts_uncomp, quiz_total_attempts_ttl, quiz_timespent_ttl, quiz_scores_mean Assignment features assign_activity_count, assign_active_days, assign_avg_time_inbtwn, assign_score Peer-review features pr_subs_count, pr_count, pr_avg, Sequence features da_count, qa_count, ca_count, ad_count, qd_count, cd_count, ac_count, ac_count, dc_count
  14. 14. Method • 10-fold cross-validation, • LASSO as the regression method, • Logistic regression as the classifier, • Scikit-learn implementation of the predictors. • Mean absolute error (MAE) as the performance metric 14
  15. 15. Results 15 SESSION APPROACH 0 1 2 3 4 TOTAL PREV 1st Peer Reviews Regression 2.06 1.08 0.24 0.75 1.68 1.04 1.02 Ensemble 2.06 1.08 0.24 0.76 1.68 1.04 2nd Peer Reviews Regression 1.73 0.71 0.76 0.23 1.31 0.60 0.66 Ensemble 1.59 0.83 0.79 0.24 1.30 0.59 3rd Peer Reviews Regression 1.19 0.78 0.82 0.20 0.88 0.49 0.56 Ensemble 0.74 1.08 1.05 0.20 1.06 0.45 4th Peer Reviews Regression 1.06 1.03 0.73 0.21 0.98 0.52 0.58 Ensemble 0.73 1.28 0.97 0.23 0.98 0.50
  16. 16. Discussion • Better prediction performance with ensemble approach • A classification phase before the regression model • Standard Deviation of actual peer-review participation • Ranging between 2.41-2.62, • The performance of the ensemble method seem to be promising, • Ranging between 0.45 to 1.04. 16
  17. 17. Discussion • However, error increased in the prediction of other levels • Due to poor classification performance. • More work needed on the classification model. 17
  18. 18. PEER-REVIEW PREDICTION PART II 18 Classification and Transferability
  19. 19. Classification: Under-participation vs… Recommended participation [e.g., 3 reviews] 19
  20. 20. Practicality! 20 Course Start Course End PREDICTION MODEL
  21. 21. Practicality! • Experimentation with post-hoc predictions [3] • Useful in demonstrating relationships • Exam scores  Dropouts • No practical use. 21
  22. 22. Practicality! • Using only the information available at the time of prediction, • Making prediction when it is useful to instructor. 22 • Using TRANSFER LEARNING approaches [4] to create operational prediction models.
  23. 23. Transfer learning: Practical predictions • In-situ learning: Transferring a model across different weeks 23 Course start nth peer reviews (features)n (MODEL) n Predicting (n+1)th peer-reviews (n+1)th assignment submissions (features)n+1 using (MODEL)n fed with (features)n+1
  24. 24. Transfer learning: Practical predictions • Across contexts: Transferring a model across different courses Course A Course B nth peer reviews1st peer reviews 2nd peer reviews (MODEL)1 (MODEL)2 (MODEL)n nth peer reviews 1st peer reviews 2nd peer reviews
  25. 25. Context • Course #1 • 3567 enrollments • 4 peer-review sessions 25
  26. 26. Context • Course #2 • 3632 enrollments • 4 peer-review sessions 26
  27. 27. Context • Course #3 • 5248 enrollments • 7 peer-review sessions 27
  28. 28. Feature Generation Discussion entries dentry_count, dentry_charcount_mean dentry_charcount_ttl, dentry_depth_mean, dentry_linecount_mean, dentry_linecount_ttl, dentry_wordcount_mean, dentry_wordcount_ttl, 28 Quiz activities finished_quiz_how_early, quiz_scores_mean, quiz_scores_ttl, quiz_timespent_mean, quiz_timespent_ttl, quiz_total_attempts_mean, quiz_total_attempts_ttl, uncomp_quiz_counts Assignment activities assign_attempt assign_score assign_submt_how_early Peer-review activities PR_SUBS_COUNT, PR_COUNT, PR_AVG, message_size_ttl, message_size_avg, message_size_multby_timespent, message_size_dvdby_timespent, pr_timespent_mean, pr_timespent_ttl, review_days_after_submission
  29. 29. Method • 10-fold cross-validation vs transfer learning approaches, • Logistic regression as the classifier, • Scikit-learn • Offers parameters to deal with imbalanced class distributions • AUC scores were used as the metrics of performance. • More rigorous when the class distributions are imbalanced. • 0.9-1.0: Excellent, 0.8-0.9: Good, 0.7-0.8: Fair, 0.6-0.7: Poor, 0.5-06: Fail 29
  30. 30. Results Course #1 Course #2 Course #3 30 1st Peer reviews 2nd Peer reviews 3rd Peer reviews 4th Peer reviews CV - 0.629 0.895 0.891 I-S - 0.606 0.863 0.849 1st Peer reviews 2nd Peer reviews 3rd Peer reviews 4th Peer reviews CV - 0.690 0.766 0.821 I-S - 0.678 0.741 0.797 1st Peer reviews 2nd Peer reviews 3rd Peer reviews 4th Peer reviews 5th Peer reviews 6th Peer reviews 7th Peer reviews CV - 0.680 0.681 0.724 0.735 0.787 0.822 I-S - 0.631 0.615 0.720 0.723 0.742 0.751
  31. 31. Results • Transfer between Course #1 and Course #2 31 Using Course #1 to predict Course #2 on a weekly base Week #1 Week #2 Week #3 Week #4 Week #1 0.559 Week #2 0.681 Week #3 0.656 Week #4 0.717 Using Course #2 to predict Course #1 on a weekly base Week #1 Week #2 Week #3 Week #4 Week #1 0.552 Week #2 0.735 Week #3 0.779 Week #4 0.762
  32. 32. Results • Transfer between Course #1 and Course #2 32 Using Course #2 to predict Course #1 on a weekly base Week #2 Week #3 Week #4 0.787 0.802 0.806 Using Course #1 to predict Course #2 on a weekly base Week #2 Week #3 Week #4 0.672 0.716 0.751
  33. 33. Discussion • Using in-situ approach, the classification model has potential to be utilized in an ongoing MOOC. • Transferable over the weeks of the same course. 33
  34. 34. Discussion • Transferable in both directions, but accuracy was not the same • Training using whole course seems to perform better. • More data points  better trained model. • Prediction of the participation in the first peer reviews is a challenge. 34 The classification model seems to be transferable across different courses.
  35. 35. Future Work Limitation -> Context was UNKNOWN Connecting with Learning Design • Replicating the model in one of our own MOOCs: • Features that are informed by the learning design • Transferring across KNOWN courses • The role of the learning design and the context 35
  36. 36. Future Work • Using this classification model in practice • to produce a relevant input to the group-formation task, • to offer peer-review related interventions • Exploring other characteristics of peer-review engagement: • Whether students are likely to provide feedback or not, • The quality of feedback: related factors? 36
  37. 37. References [1] Topping, K.: Peer assessment between students in colleges and universities. Rev. Educ. Res. 68, 249–276 (1998). [2] Wen, M.L., Tsai, C.C., Chang, C.Y.: Attitudes towards peer assessment: a comparison of the perspectives of pre‐service and in‐service teachers. Innov. Educ. Teach. Int. 43, 83–92 (2006). [3] Kurka, D.B., Godoy, A., Von Zuben, F.J.: Delving Deeper into MOOC Student Dropout Prediction. CEUR Workshop Proc. 1691, 21–27 (2016). [4] Champaign, J., McCalla, G.: Transfer Learning for Predictive Models in Massive Open Online Courses. Lect. Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics). 9112, 883 (2015). 37
  38. 38. Thanks! • Questions? 38

×