The document describes a research project conducted by librarians and instructional designers at UC Santa Cruz to develop a checklist for student research papers, which was found to improve students' citation skills, source evaluation, and bibliographies when used. The checklist was tested on students in a psychology course, and results showed significant improvements in attribution of facts and ideas for all students, especially for students from marginalized backgrounds.

1. Clear expectations and habits of
mind: A self-evaluation checklist for
student writing
Annette Marines, Research Support Librarian
Aaron Zachmeier, Instructional Designer
University of California, Santa Cruz
LILAC 2017 Swansea University

2. Project Context
● Campus-wide assessment
○ Active program outcomes assessment
○ Focus: Written communications and information literacy
● Teaching support efforts
○ Liaison Librarian: Collections and Information Literacy Instruction
○ Instructional Designer: Online Education
○ Assessment Assistant Director: Assessment Support
● This Research Team
○ Liaison Librarian
○ Instructional Designer
○ Institutional Research, Assessment, and Policy Studies (IRAPS)

3. Introduction
● What we were hearing and seeing
○ Students were not citing; were citing incorrectly; were not evaluating sources
○ Assignment handouts avoided guidance
● The intervention
○ Opportunity for Librarian and Instructional Designer to collaborate
○ Developed a checklist to use as a research paper cover sheet
● The research project
○ Teamed up with IRAPS and a Psychology professor
○ Assessed the checklist

4. Why it’s a problem?
● Expert versus novice
● Information deluge
● Communication
● Time management

5. Some Literature
● Assigning Inquiry: How Handouts for Research Assignments Guide Today's
College Students, Alison J. Head & Michael B. Eisenberg, Project Information
Literacy
● They can find it but they don't know what to do with it: describing the use of
scholarly literature by undergraduate students, Stephanie Rosenblatt
● Checklist manifesto: How to get things right, Atul Gawande
● Teaching Research Processes, William B. Badke

6. Hypothesis
Students’ proficiencies in citation, attribution, and source evaluation can improve
with a checklist

7. Research Strategy
With full cooperation with the course instructor, the checklist intervention was
assigned in a senior seminar course. Students were required to turn in the
checklist as a cover sheet to the draft of a research paper
A rubric addressing learning outcomes of the checklist was used to evaluate the
draft research papers of two classes: one using no checklist and one using the
checklist

8. Method
Participants
● Upper division, undergraduate seniors
● Psychology majors
● Senior seminar course enrollees

9. Intervention : The Checklist
● We provided the course instructor with a PDF copy of the checklist
● The instructor:
○ Assigned the checklist along with the draft research paper
○ And made it available in the LMS
○ No additional instruction was provided
● Students handed in the checklist with the paper
● The intervention was handled exclusively by the course instructor

11. Sample
● No checklist n = 24 ; Checklist n = 27
● Unit of analysis: Draft Research Paper
● 100% of the research papers from each class were included
○ Started by reading 50% (n=12 and n=14). IRAPS asked us to read all the papers to increase
significance
● Classes were taught three year apart by the same instructor

12. Measurement Instrument : Rubric
● Informed by AACU Value Rubrics and others
● 9 criteria
○ Each checklist item is matched to a rubric criterion, Ex. Evaluation separated into 3 domains to
capture
■ Makes rubric more precise in evaluating information literacy and critical thinking
● 4 levels
○ Unacceptable (no signs), Beginning, Developing, Proficient (mastered)

15. Data Collection
● 2 rubric norming sessions
○ Discussed discrepancies until we came to an agreement
● 2 readers, read through and scored papers
● Readers evaluated:
○ Citation style, formatting, consistency
○ Attribution
○ Evaluation of sources
● Blind review

16. Results
IRAPS ran the data analysis. The questions covered:
● Setting a baseline
● Comparing no checklist and checklist classes
● Areas that remained low
● Effect of checklist on specific populations

17. Key Decision
● The Developing & Proficient levels were combined
● The counts for Proficient were too low to analyze

18. In what areas do most students have developing or proficient skills and in
what areas do they need improvement?
(the baseline)
MEDIUM
60-79%
C
HIGH
80-100%
D2, A [ F, D3 ]
LOW
0-59 %
B1, D1, B2, E
A - Source Types
B&C - Bibliography
D - Attribution
E - Evaluation
F - Quoted Text

19. LOW
0-59 %
D1*, B2**, E
In what areas did the checklist lead to improvement?
MEDIUM
60-79%
B1***, C
HIGH
80-100%
D2, A [ F, D3 ]
* Significance at p <.05
** at p <.07
*** Not significant
A - Source Types
B&C - Bibliography
D - Attribution
E - Evaluation
F - Quoted Text

20. LOW
0-59 %
D1, B2, E
In what areas do students still lack skills?
MEDIUM
60-79%
B1, C
A - Source Types
B&C - Bibliography
D - Attribution
E - Evaluation
F - Quoted Text

21. LOW
0-59 %
D1 Study 1 - SOC &
Bilingual
D1 Study 2* - SOC
Were there any differences related to background characteristics (ethnicity,
language, and transfer status)?
MEDIUM
60-79%
D2 Study 1 - SOC
D1 Study 2* - Bilingual
HIGH
80-100%
D2 Study 2* - SOC
* Significance differences Fisher’s exact test p < .08
Study 1 findings in blue
Study 2 findings in purple
SOC - Students of color
D1 - attribution of facts
D2 - attribution of ideas

22. Results Sample Overview
● Both classes were comparable in terms of ethnicity and language status
○ Bilingual students included those who at the age of 5 spoke English and another language or
learned English later in school.
● Class 2 (checklist) had fewer transfer students than Class 1 (no checklist)
● There were more first generation students in Class 2 than in Class 1
● There were greater number of women in both classes, with fewer men in
Class 2

23. Discussion
● The results support our hypothesis: Students proficiencies can and did
improve with the checklist
● Students improved in every area, with significant differences in 3 of these
● Clear guidelines and steps help students

24. Evaluating Sources
MED-HIGH LOW
21-59%
BOTTOM LOW
0-20 %
E
A - Source Types
B&C - Bibliography
D - Attribution
E - Evaluation
F - Quoted Text

25. New text
❏ Have you included an evaluation of your sources?
Pay particular attention to the (1) the authoritativeness, perspective, and potential bias
of the author; (2) credibility or validity of the findings; (3) relevance and purpose of the
source itself.
Evaluating Sources
Old text
❏ Have you evaluated your sources?
Pay particular attention to the (1) the authoritativeness, perspective, and potential bias
of the author; (2) credibility or validity of the findings; (3) relevance and purpose of the
source itself.

26. Why it’s a problem, again?
Expert versus novice, Information deluge, Communication, Time management
Limitations
● The checklist is not a teaching tool
● No evidence yet that repeated use will develop habits of mind
Unexpected findings
● Closed gaps seen in ethnic and language populations
○ The checklist helped everyone, but helped these groups more

27. Extending the Project
● We’re working with the same instructor with their lower-division, introduction
to the major course.
● We’re currently reading papers for a Senior Seminar course in the History
Department
Teaching Support
● We promote the checklist in our teaching support consultations
● Our new rubric experience has opened doors with assessment conversations,
where I wouldn’t have brought it up before

28. Photo Credits
● The University of California Santa Cruz, Doc Searls, (CC BY-SA 2.0), https://flic.kr/p/jmLrq
● Understand context, Paul Downey, (CC BY 2.0), https://flic.kr/p/eTB94c
● Ginger man... NAILED IT!, Sharon Vicino, (CC BY-SA 2.0), https://flic.kr/p/quA8aW
● Rubik's Cubes, CC0 Public Domain, http://maxpixel.freegreatpicture.com/photo-1243091
● College Students, CollegeDegrees360, (CC BY-SA 2.0), https://flic.kr/p/cEJAgJ
● cmyk color management linen tester, Markus Spiske, (CC0 1.0), https://flic.kr/p/RS7kbT
● Crunches, Everkinetic (http://everkinetic.com/), (CC BY-SA 3.0),
https://commons.wikimedia.org/wiki/File%3ADecline-crunch-2.png &
https://commons.wikimedia.org/wiki/File%3ADecline-crunch-2.png
● Crystal Wynwood Mural, Elvert Barnes, (CC BY-SA 2.0), https://flic.kr/p/pic2z7
● Hooray Penguin, David Goehring, (CC BY 2.0), https://flic.kr/p/JZ4Dmn
● Students in Class, Tulane Public Relations, (CC BY 2.0),
https://commons.wikimedia.org/wiki/File%3AStudent_in_Class_(3618969705).jpg
● The Questioning Roboto, Matthew Hutchinson, (CC BY 2.0), https://flic.kr/p/7VhPft

29. Annette Marines
amarines@ucsc.edu
Aaron Zachmeier
azachmei@ucsc.edu
More information (checklists)
http://guides.library.ucsc.edu/checklists