Sally Rumsey's presentation from the ORCID and Casrai joint conference, Barcelona, May 2015

1. Organizational complexity as a
challenge to research assessment
Sally Rumsey
The Bodleian Libraries
University of Oxford
A case study of the University of Oxford

2. Complexity
A result of
• Size of the institution
and the amount of
research produced
(research intensive)
Coupled with
• Organisational
complexity of the
institution
(administrative
structure; governance;
etc)

3. University of Oxford
Facts and figures*
• > 5800 academic and research staff
• > 5600 research students
• 4 university academic divisions (> 70 departments)
– Humanities: 9 faculties + Ruskin School of Art
– Social Sciences: 16 departments
– MPLS (Mathematical, Physical & Life Sciences): 12 departments
– Medical Sciences: 16 departments
– Plus cross disciplinary institutes eg Oxford Martin School and others
• 44 colleges and PPHs (Permanent Private Halls) (separate legal
entities to University)
• External research funding £478.3m
• Approx 4700 active research awards
• Estimated 16,000 journal articles published p.a.
• > 120 libraries
* 2013 and 2013/14 figures: Annual Review 2013-14

4. Democracy
• Wellington Square as the
centre of ‘the University’
• Democratic
• Decision-making devolved
to the four Academic
Divisions and their
departments/faculties
• Decision-making can
progress slowly
• Example: Symplectic
adoption for REF 2014
• REF OA Service design
project (for REF 2020)
Cppyright Clare Byers

5. Colleges
• Separate legal
entities to the
university
• Dual appointments
• College-only
appointed staff
• Not on HR system
• Submitted to REF
Copyright Jason Partridge

6. Governance and infrastructure
• Complex committee structure
• Routes and timing of committees
• Systems integration (WIP)
• RIM data quality
• Lack of unique identifiers and controlled vocabularies
across the entire institution (Divisions, Colleges, NHS
etc)
• Devolved model
• Internal matters also relevant to external integration
• Compliance with standards (eg RIOXX)

7. Facts and figures
Oxford REF 2014 submission
Submitted to
31 Units of
Assessment
(UoA)*
(out of 36)
c.2400
research staff
submitted
c. 8500
outputs
submitted
Returned
380,000 data
fields
REF 2008
resulted in £1.9
billion research
income over 6
year period
Reported
4892.5 doctoral
degrees
awarded
Took 2½ years
to manage and
compile return
* http://www.ref.ac.uk/panels/unitsofassessment/

8. REF complexity illustrative case
OII – Oxford Internet Institute
• A department of Social Sciences Division
• Multi/cross disciplinary
• Submitted under 9 UoAs (inc psychology,
economics, law, geography)
• Difficulty in identifying which are the ‘best’
outputs. Best for whom?

9. Quality control for REF Case studies
• Quality control was a challenge
• 31 UoAs split across 4 academic divisions
• Needed consistency across all submitted case
studies across:
– Unit
– Within the division
– Across all 4 divisions

10. REF complexity illustrative case:
Impact
• Medical sciences – broad areas such as
‘cancer’ or ‘neuroscience’ over a large
academic division
• Do not ‘sit’ in any one department
• Administrative staff cannot know all the
details to report impact
• Impact can be reported back 20 years
previously. Difficult to identify and track back
so far – people leave. Generally cite only 10
years

11. The effect on assessment
• Locating data required for
assessment
• Dealing with a variety of
systems
• Lack of single location for
each data type
• Data entry/import
(quantity)
• Consistency
• Size of the operation and
numbers of people involved
Copyright Clare Byers

12. • REF OA Service
Design Project
• Pilot Jan – June 2015
• Simple, short
message
– For individuals
– For support staff
(including directive)
• Comms is key
• RS/Bod/Divisional
staff

13. ORCIDs
• How ORCIDs should help
– De facto standard
– Common across all units of the University
– Supports integration externally
– To be required for REF 2020?
– Ultimately link individuals, publications, funding
– Assist automation
• Difficulties eg College staff not on HR/CUD system
• Limitations
• ORCIDs at Oxford service

14. ORCIDs at Oxford service is live
Released 12 May 2015

15. Points for discussion
• Data quality
• Systems integration
• Standards
compliance
• Common framework
between agencies
• Reduce
administrative
burden
• Automation to cope
with scale
Sally Rumsey, the Bodleian Libraries, 2015