Despite being controversial, research metrics are becoming a key component of research evaluation processes globally. Nevertheless, accessing research metrics to support these processes in a timely manner is not a straightforward task, as it requires either having access to expensive commercial solutions such as Elsevier SciVal or Clarivate Analytics' InCites, or having substantial knowledge of existing APIs and data sources as well as the ability and skills needed to analyse large amounts of raw scholarly data in-house. This is especially the case on a department or institutional level where large amounts of data have to be aggregated prior to analysis. To alleviate this problem we have designed and prototyped CORE Analytics Dashboard – a tool for analytical evaluation of research outputs of universities. The aim of the CORE Analytics Dashboard is to help universities analyse their performance using a variety of metrics captured from openly available data sources, including citation counts and social media metrics, and to help them compare their performance with other institutions. This paper presents the motivation behind developing this dashboard and its main features.

1. CORE Analytics Dashboard
Petr Knoth
Jozef Harag
Drahomira Herrmannova
June 13, 2019 – OR 2019, Hamburg, Germany
CORE
Big Scientific Data and Text Analytics Group
Knowledge Media Institute, The Open University

2. Introduction
• Research metrics becoming a key component of evaluation
processes
• Application of research indicators controversial, but demand for
them is increasing
• Universities want to understand and monitor the impact of their
research outputs
• This is evidenced by subscriptions to expensive products like
Elsevier’s SciVal, Clarivate’s InCites
• Typical cost: negotiated, it is believed £35-£50k per annum per
institution

3. Existing solutions
• Elsevier SciVal
• Clarivate InCites
• Digital Science Dimensions

4. Limitations of existing solutions 1/2
• Proprietary data (Scopus and Web of Science)
• Data or results of analyses cannot be easily downloaded and shared
with others
• Algorithms used for citation data acquisition not open/known
(Scopus and Web of Science)
• Complicates confirming validity/replicating results

5. Limitations of existing solutions 2/2
• Data sparsity
• Scopus/Web of Science are known to have lower publication/citation
coverage than Google Scholar and Microsoft Academic [Harzing, 2017]
• Dimensions use primarily citations from I4OC which is an even sparser
citation dataset
• Lack of APIs for integration of results into existing systems
(Scopus and Web of Science)
• Slice-and-dice operations on titles/abstracts only
• Papers without DOI have metrics rarely available

6. CORE Analytics Dashboard
• CORE is the largest full text Open Access aggregation service
[Notay, 2018]
• Because it is based on Open Access data, it can offer/enable:
• Slice-and-dice operations based on full text
• Full access to underlying data
• API integration with internal university systems

7. CORE Analytics Dashboard
• A tool for analytical evaluation of universities’ research outputs
based primarily on CORE data
• Research impact insights on institutional level
• Comparison with other institutions or groups of institutions
• Based on citation data, social impact data
• Enables users to design and create custom graphs and add
these to their user area
• Intended for university research officers, repositories, university
management

8. Use cases
• A university's Research Office manager wants to see how
her institution compares to another institution
• A university's Research Office manager wants to have an
overlook on research impact of a hot-topic (e.g. Artificial
Intelligence, DNA mapping, global warming, etc.) across UK
universities
• A non-Russell Group university wants to demonstrate (for
marketing reasons) that they have the same research and
social impact (or perhaps better) as any of Russell
Group universities in a given field.

9. Data sources
1. CORE dataset
• Primary pivot, provides publication metadata, affiliation information
2. Microsoft Academic Graph
• Citation data
3. Crossref event data
• Publications’ Wikipedia and Twitter mentions
4. Mendeley
• Readership data, i.e. information about how many Mendeley users
have added a certain publication in their Mendeley library

10. Data
• Data from across different systems were matched using DOIs
• Prototype version:
• UK institutions
• Outputs with a DOI
• Long-term goal is to expand to the whole word

11. Interface overview
• Individual for each user
• Each user assigned a specific
institution
• Customizable user area:
users can define metrics and
graphs to track & display
• Institutional ranking

12. User interface: statistics bar
• Each user assigned a specific institution
• Quick summary overview of institution’s performance

13. User interface: dashboard
• Customizable area, user-defined charts

14. User interface: institutional ranking
• Ranking according to a number of metrics

15. Customizable area
• Enables creating user-defined
graphs
• Chart data can be
downloaded:
• Original (raw) data
• Aggregate chart data
• Chart PNG
• Full customizable, enables
resizing and reorganizing
charts

16. User area: customizing the dashboard
• A step-by-step tool (benchmarking wizard) for creating custom
visualizations tailored to user’s needs
• Works in three steps:
1. Define publications sample filtered to select search criteria
2. Define what indicators to inspect
3. Select visualization chart to apply on the filtered data

17. User interface: benchmarking wizard
1. Filter data – user specifies which articles to work with
• E.g. using keywords or phrases of interest that will be searched in
publications’ titles, abstracts, and full texts
• Also enables filtering, currently by year and publication type (research
article, thesis, etc.)
2. Selecting metrics to track – user defines which institutions to
compare and based on what metrics to compare them
3. Designing the chart – select chart type and name the chart
• Several chart types available including line charts, bar charts, scatter
plots, etc.
• User can customize colors, names of axes, etc.

18. User interface: benchmarking wizard
1. Filter data – user specifies which articles to work with

19. User interface: benchmarking wizard
1. Filter data – user specifies which articles to work with

20. User interface: benchmarking wizard
2. Selecting metrics to track

21. User interface: benchmarking wizard
3. Designing the chart

22. Conclusions
• We presented CORE Analytics Dashboard – a tool designed to
enable users to analyse and compare the performance of
research outputs between universities along a variety of metrics
• Key difference from existing solutions – focus on collecting
performance indicators from openly available sources
• Goal – add a layer of transparency to research evaluation
• Goal – extend to whole world and add additional features
• E.g. collaboration graphs, UK REF predictions [Pride, 2018]

23. Feedback
• www.menti.com
• Code 23 36 84

24. References
• Harzing, Anne-Wil, and Satu Alakangas. (2017). "Microsoft
Academic is one year old: The Phoenix is ready to leave the
nest." Scientometrics 112.3: 1887-1894.
• Notay, Balviar. (2018). "CORE becomes the world’s largest
aggregator". Jisc scholarly communications.
https://scholarlycommunications.jiscinvolve.org/wp/2018/06/01/c
ore-becomes-the-worlds-largest-aggregator/. Accessed: 2019-
01-16.
• Pride, D. and Knoth, P. (2018). "Peer review and citation data in
predicting university rankings, a large-scale analysis". Lecture
Notes in Computer Science. Springer.

25. Thank you!
Demo: bit.ly/core-analytics-dashboard