This document discusses peer review and how to conduct constructive peer reviews. It defines peer review as the appraisal of reported research by experts in the field. Peer review is used to verify results, guide editors on publishing decisions, and help authors improve their work. The document outlines what peer review does and does not do. It also provides guidance on how to conduct strategic and helpful peer reviews, including using techniques like questioning the research, assessing what is needed to make the work publishable, and providing evidence-based feedback in a respectful manner. The goal is to foster constructive criticism that helps strengthen scholarship.
If you can't be kind, be scholarly. Constructive peer reviewing - Emma Coonan
1. If you can’t be kind,
be scholarly
Constructive peer reviewing
EMMA COONAN
JOURNAL OF INFORMATION LITERACY
GrouphugbyJorisLouwes,CCBY2.0
2.
3. Aims
• Explore what peer review is and what it’s for
• Demystify what it involves
• Foster constructive reviewers and critical friends to
scholarship
7. Discussion
1. What is peer review?
Appraisal of reported research by expert in the field
May be ‘double blind’ – author’s name is not revealed
May be 2 or more reviewers
9. Discussion
2. What’s it for?
Verification of reported results as far as possible
Guide the editor in a decision on whether to publish
Help authors make the best possible presentation of their research to their
community of practice
10. The $64,000 question
“What’s needed to bring this up to
publishable standard?”
The point is not to eliminate but to include
14. What to look for
• Research informed and evidence based
• Designed around an arguable research question
• Contextualised with reference to previous and current advances in IL
thinking
• Methodologically robust with a demonstrable research design
• Investigation not description
16. Guess the headings
• Relevance to JIL
• Originality and interest to audience
• Title and abstract
• Methodology
• Use of literature and referencing
• Clarity of expression and structure
17. Outcomes for each criterion
• Appropriate
• Needs amendment
• Needs major rewriting or adjustment
18. Overall recommendation
• Accept for publication without amendment
• Revisions required
• Major revisions required followed by peer review
• Decline submission
19. How to look
• Critically and analytically
- not descriptively / not at sentence level
• Test for weakness in argument and structure
- use the what/why/how framework
• Detached mindset
- evaluate integrity of argument, not how far it matches your own view of IL
• Don’t just review what you see
- what is the author not saying? What literature hasn’t been cited?
24. Reviewer’s toolkit
• JIL’s 4 bullet points
Question-led, evidence-based investigation
• The $64,000 question
“What’s needed to bring this up to publishable standard?”
25. Reviewer’s toolkit
• JIL’s 4 bullet points
Question-led, evidence-based investigation
• The $64,000 question
“What’s needed to bring this up to publishable standard?”
• Strategic reading techniques
26. Reviewer’s toolkit
• JIL’s 4 bullet points
Question-led, evidence-based investigation
• The $64,000 question
“What’s needed to bring this up to publishable standard?”
• Strategic reading techniques
Including reverse outlining
27. Reviewer’s toolkit
• JIL’s 4 bullet points
Question-led, evidence-based investigation
• The $64,000 question
“What’s needed to bring this up to publishable standard?”
• Strategic reading techniques
Including reverse outlining
• What/why/how
28. What/why/how
• What is the research?
What questions does it address? What contribution does it make?
• Why has it been done?
Why does it matter? What will it change?
• How has it been done?
What’s the method? How does it frame the findings? How has it helped the
researcher mitigate bias?
30. “I would like to thank you again for all the constructive and
benevolent effort that you and your reviewers put into this review
and for the graciousness with which you did it.
“I have been through several submission processes that have
been quite impersonal and where the critical feedback has been
either on the verge of cruelty or entirely neglectful. You and your
reviewers stand apart …”
32. On being helpful and humane
• Check your privilege - unequal power relationship
• You don’t have to agree, just to check if the position is adequately
grounded and defended
33. On being helpful and humane
• Use what’s well done as a yardstick
• “What I think would make this even better is …”
34. On being helpful and humane
• “Show your workings” (be evidence-based!)
• Give practical and workable suggestions for how to implement your
amendments
35.
36. 1. “This article is riddled with assumptions.”
2. “The writing is often arrestingly pedestrian.”
3. “It is clear that the author has read way too much and
understood way too little.”
4. “Something is missing.”
5. “Not only does this strike me as the worst kind of postmodern
legerdemain, but if true the statement would transform ethics
into a hopelessly muddled enterprise.”
From http://shitmyreviewerssay.tumblr.com/
37. Further reading
JIL author guidelines
Lowell, Seri (2002) Helpful hints for effective peer reviewing
Raff, Jennifer (2013) How to become good at peer review
Schneiderhan, Erik (2013) Why you gotta be so mean?
This tweet is really what started off the idea for this workshop -
… it prompted me to think (and tweet) “Be kind, always. And if you can’t be kind, be scholarly”. Meaning primarily: make your reviewing not only humane, but objective and evidence-based: just what we are looking for in all scholarly communications.
And hopefully inspire confidence to review!
Discuss in small groups and feed back to the whole group
Single blind: reviewers are anonymous, but they know who the author is.
JIL always uses double blind peer review (sometimes more than 2 reviewers).
Looking for: analysis, verification, testing structure, meeting scholarly writing conventions, checking use of evidence etc.
NB, this is my definition – other brands are (probably) available : )
From the JIL Author Guidelines.
Guess the headings game (from JIL form) – there are 6 aspects that the editor needs to know about: what do you think they are?
NB – they aren’t the same as the bullet points on the previous slide – but some of them broadly correspond to the structure of the paper
(Caveat: if you come up with a really good new aspect we may include it :D )
Appropriate – needs amendment – needs major rewriting.
On not proofreading: clarity of expression and sentence construction DO matter – but view them in larger terms, as demonstrating how well the author understands their own argument and is able to unfold it logically (see ‘structural integrity’ below).
Structural integrity: is there a connection between the research question and the method chosen, between the method and the findings, and between findings and conclusions drawn? Do all parts of the paper point back to the research question and title? (for more on method, see Alison Pickard’s ‘guidebook’.
How many can you remember?
In a nutshell!
Which is … ?
Not just for your own sake as a reader: also the best way to test the structural integrity of a work.
Name me some strategic reading techniques?
The last two are also great for helping with writing!
Discuss in small groups and feed back to the whole group
Also: not being personal; not being absolutist; non-dismissive phrase constructions;
Classic feedback sandwich manoeuvre
Don’t just give vague directives! State what the issue is, why it’s not working for you, and how the author might go about making it work better.