Jennifer Imazeki, Economics
Scaffolded Writing and Reviewing in the Disciplines(SWoRD) is a web-based peer-review system. One of the primary innovations of SWoRD, relative to other peer review tools, is the scoring algorithm through which peer review scores are converted into student grades for both writing and reviewing. In this session, I will discuss my experience with SWoRD, which I used in Spring 2011 for an upper-division writing course for economics majors, replacing my previous system of ‘manual’ peer review (i.e., students swapping papers)
Peer review in an economics writing course: the Pen is still mightier than the SWoRD,
1. Peer review in an economics writing course: the Pen is still mightier than the SWoRD Jennifer Imazeki Department of Economics jimazeki@gmail.com http://economicsforteachers.blogspot.com
2. SWoRD Scaffolded Writing and Reviewing in the Disciplines Students submit papers Students review papers of 4-6 peers Comments and 7-point numeric rating Students ‘back evaluate’ reviews they receive 5-point numeric rating
3. What makes SWoRD different? Student review and backevaluation scores grades for both writing and reviewing Reviewing grade: “Accuracy” + “Helpfulness” Accuracy: Consistency compared to other reviewers Helpfulness: Back evaluation ratings Writing grade: adjusted average of numeric ratings from reviewers Scoring algorithm to remove bias
4. Econ 449W: Economic Literacy “The goal of this course is to make you better economists by making you better writers… This course will focus on how to apply and use [economic] knowledge, by incorporating economic material and the economic way of thinking into writing for a more general audience.”
5. Econ 449W: Structure 2010: 6 peer-reviewed papers, exchanged in class 1 reviewer for each paper, plus instructor feedback No feedback for reviewers, participation points only
6. Potential benefits of SWoRD(vs. other peer review tools) Students read multiple papers, receive feedback from multiple reviewers Back evaluations provide incentive to take reviews seriously, give students feedback for improving reviews (without requiring instructor intervention) Rubric scores converted to grades with adjustment for bias
7. Econ 449W: 2010 vs. 2011 2010: 6 peer-reviewed papers, exchanged in class 1 reviewer for each paper, plus instructor feedback No feedback for reviewers, participation points only 2011: 4 peer reviewed papers, all online submission 4-5 reviewers for each paper, no/less instructor feedback on draft 1 Writers back-evaluate reviewers, reviewing score based 50% on those back evaluations
8. Assignment timeline 5 days 5 days 5 days 9pm Monday 9pm Sunday 9pm Friday 9pm Wednesday First Draft Final draft Final reviews (5) Reviews (5) Back reviews by Thurs Back reviews by Wednesday
9. Issues Lack of flexibility Lack of transparency Scoring algorithm questionable
10.
11.
12. Peer review in an economics writing course: the Pen is still mightier than the SWoRD Jennifer Imazeki Department of Economics jimazeki@gmail.com http://economicsforteachers.blogspot.com
Editor's Notes
http://www.lrdc.pitt.edu/schunn/sword/index.html
Review process itself similar to other options (e.g., PeerMark, doing it manually)
Key point here is that the focus is not JUST writing but incorporating economic ideas into writing (so they have to know some econ)
- Explain process (half class writing/half reviewing)- I already had rubrics and peer review instructions for each assignment, so fixed costs low
This was the main reason I wanted to try SWoRD – already did some peer review but SWoRD a) allows for more feedback overall and b) potentially less grading for me
- the biggest change in the structurewas that because you need at least 4 or more reviews in SWoRD to get meaningful composite scores, I reduced the number of writing assignments and stretched out the timeline for each paperI had hoped to give less feedback on the first draft but after first assignment, realized that wasn’t going to happen – partly a problem with my comment and rating prompts, partly students don’t know how to give good feedback, partly that students don’t feel ‘qualified’ to give feedback (since emphasis in this course is as much on content as writing) These are challenges for peer review in general (not necessarily SWoRD): students need training in how to give good comments; prompts and ratings need to be well-designed, not vague, get students to recognize their own ability to be ‘experts’But the problems students have with giving good reviewing feedback is even more important with if student grades are going to be based on those reviews and that feedbackBack evaluations did not provide good assessment or sufficient feedback for improving comments
One thing I want to point out is that my class met on Mondays and Wednesdays – issues with SWoRD deadlines
Interface overall is not very flexible (deadlines biggest problem but also admin)Explain about how SWoRD requires curve but not transparent (show screenshots)Can see raw scores, final scores, but can’t see scores without curve applied
Stats for each draft
Draft grades (cannot change order of presentation in SWoRD)
