The iterative engagement between curation and evaluation in an open research project: A utilization-focused approach Presentation for the AVU Conference 1-3 July 2015 by Sarah Goodier

1. Sarah Goodier
Thomas King
Cheryl Hodgkinson-Williams
ROER4D Project, Centre for Innovation in Learning and Teaching,
University of Cape Town
2nd International Conference of the AVU
2 July 2015
The iterative engagement between curation and
evaluation in an open research project:
A utilization-focused approach
7/3/20151

2. Curation in research
 Curation of research – usually commercial
e.g. journal articles; book chapters
 Move towards researching in the open:
 practice of releasing interim or draft outputs, early
data sets and project planning documents during the
course of project activity
 Change in types of research products
being shared and when as well as ways of
engaging with these research products

3. in the Global South
 In what ways, and under what circumstances can the adoption of OER
address the increasing demand for accessible, relevant, high-quality
and affordable education and what is its impact in the Global South?
 Researching in the open
The case of ROER4D:
Research on OER for Development

4. 1. Build an empirical knowledge base on the
use and impact of OER in education
2. Develop the capacity of OER researchers
3. Build a network of OER scholars
4. Communicate research to inform
education policy and practice
5. Curate output as open content
ROER4D Objectives

5. 1. Build an empirical knowledge base on the
use and impact of OER in education
2. Develop the capacity of OER researchers
3. Build a network of OER scholars
4. Communicate research to inform
education policy and practice
5. Curate output as open content
ROER4D Key Evaluation Areas
EVALUATION
Using Utilization
Focused Evaluation
(UFE) supported by
DECI-2

6. UFE in 12 steps:
1. Assessing program readiness
2. Assessing evaluator readiness
3. Identifying primary intended users
4. Situational analysis
5. Identification of primary intended uses
6. Focusing the evaluation
7. Evaluation design
8. Simulation of use
9. Data collection
10. Data analysis
11. Facilitation of use
12. Meta evaluation
Steps in the
Utilization
Focused
Evaluation
(UFE)
process

7. Curation in the ROER4D project
Outputs from the
project:
Open approach has a considerable influence on the
curation strategy and how it is shaped

8. Curation in the ROER4D project
Open approach has a considerable influence on the
curation strategy and how it is shaped

9. Curation platforms
Platform Function Capabilities
Content Management
System (Sakai - Vula)
Long-term, private and secure
storage of project document drafts,
raw data and confidential/internal
documentation
Security
Privacy
Long term curation
Open access repository
(OpenUCT)
Long-term, open access to project
outputs and linked data
Security
Metadata
Accessibility
Long-term curation
Curation spaces:
1. FigShare, Zenodo
2. SlideShare
Short to medium-term access to:
1. Project outputs and linked data
2. Presentations
Security
Various levels of metadata
Broader accessibility
Short to medium-
term curation
Cloud-based
Collaboration Platform
(Google Docs)
Collaborative creation and editing
of documents; sharing of interim
documents and documents under
development
Accessibility
Ease-of-use
Simultaneous creation,
editing and commenting on
documents

10. Evaluating curation in ROER4D

11. Extract from ROER4D evaluation plan for curation
Objective 5:
Curate output
as open
content
(outputs =
documents
and data)
Use(s) KEQ(s) Evidence What will
measured
To:
- better
curate the
content.
- ensure
ROER4D is
using the
best
metadata
standard
across
outputs.
5.1.1 How well is
our own
research curated
in order to be
shared?
5.1.2 Where is
the research
that has been
curated and
shared getting
the most hits?
Measure
against
metadata
standards
best practice
- process
review;
Views and
downloads
Measure
against the e.g.
Digital
Curation
Centre’s (DCC)
standards of
best practice:
Checklist for
data
management
plan, etc.
Process of
curation of
documents and
data against
checklist
Monitor
curation
spaces to
gather views
and
downloads
data
e.g. From the
OpenUCT
publically
available
statistics: Views
and downloads

12. Lessons Learnt: Opportunities & Challenges
 Opportunities
Transparency & Adaptability:
 Open and inclusive team dynamics
 Capitalising on new insights
Reciprocity:
 Connections between curation and
communication work and its
evaluation
 Within the evaluation plan
Accessibility, Adaptability &
Discoverability:
 Multiple platforms and outputs
 Evaluation can inform which
platforms are the adequate
 Challenges
Readiness & Adaptability :
 When to be open
 Evaluation needs to adapt to a
changing curation strategy
Cohesiveness:
 Across platforms
 Within the evaluation plan
Vulnerability/Instability:
 3rd-party platforms
 Extract analytics data regularly

13. Curation platforms & evaluation plan – an example
Curation
Platform
Platform
Function
Relevant KEQs
Evaluation
measures
Uses for the evaluation
results
Content
Management
System
(Sakai)
Long-term,
private and
secure storage
of project
document
drafts, raw data
and
confidential/
internal
documentation
5.1.1 How well
is our own
research
curated in
order to be
shared?
5.1.2 Where is
the research
that has been
curated and
shared getting
the most hits?
Measure
against the
e.g. DCC
standards of
best practice
Available
statistics:
Views and
downloads
- better curate the
content.
- ensure ROER4D is
using the best
metadata standard
across outputs.
- determine if it is
productive to produce
multiple formats of
outputs
- ensure cohesion
across platforms.

14. Evaluating curation in ROER4D

15. Further reading:
• DCC. (2013). Checklist for a Data Management Plan. v.4.0. Edinburgh: Digital Curation Centre.
Available online: http://www.dcc.ac.uk/resources/data-management-plans [Last accessed 10 February
2015].
• Hodgkinson-Williams, C. (2013). Research on Open Educational Resources for Development in Post-
secondary Education in the Global South (ROER4D) - Scoping Document. Available online:
http://hdl.handle.net/11427/8430 [Last accessed 10 February 2015].
• Hodgkinson-Williams, C. and Arinto, P. (2014). Open Education for a Multicultural World: A report from
the Research on Open Educational Resources for Development (ROER4D) project in the Global South.
Available online: http://www.slideshare.net/ROER4D/roer4-d-hodgkinson-williams-arinto-ocwc-2014-
33930519 [Last accessed 11 February 2015].
• Hodgkinson-Williams, C. and Cartmill, T. (2014). Research on Open Educational Resources for
Development in the Global South: 1st Technical Report 23 June 2013 to 27 August 2014. Available
online: http://hdl.handle.net/11427/9695 [Last accessed 11 February 2015].
• Monastersky, R. (2013). Publishing Frontiers: The Library reboot. Available online:
http://www.nature.com/news/publishing-frontiers-the-library-reboot-1.12664#/data (Accessed 10
February 2015)
• National Information Standards Organisation (NISO) (2004). Understanding Metadata. Bethesda:
National Information Standards Organisation. Available online: http://www.niso.org/
publications/press/UnderstandingMetadata.pdf [Last accessed 11 February 2015].
• Patton, M. Q. (2008). Utilization-focused evaluation. California: Sage Publications Inc.
• Ramirez, R. and Brodhead, D. (2013). Utilization Focused Evaluation: A Primer for Evaluators. Penang:
Southbound.
• Scriven, M. (1991). Evaluation Thesaurus. California: Sage Publications Inc.
• Tenopir C, Allard S, Douglass K, Aydinoglu AU, Wu L, et al. (2011). Data Sharing by Scientists: Practices
and Perceptions. PLoS ONE, Vol. 6, no. 6, e21101. doi:10.1371/journal.pone.0021101

16. Links and license
This work is licensed under a Creative Commons Attribution 4.0 International License.
Website: http://roer4d.org
sarah.goodier@uct.ac.za
@SarahGoodier
Excluding images,
screenshots and
logos and/or
unless otherwise
indicated on
content