Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.

CORE Analytics Dashboard

360 visualizaciones

Publicado el

Despite being controversial, research metrics are becoming a key component of research evaluation processes globally. Nevertheless, accessing research metrics to support these processes in a timely manner is not a straightforward task, as it requires either having access to expensive commercial solutions such as Elsevier SciVal or Clarivate Analytics' InCites, or having substantial knowledge of existing APIs and data sources as well as the ability and skills needed to analyse large amounts of raw scholarly data in-house. This is especially the case on a department or institutional level where large amounts of data have to be aggregated prior to analysis. To alleviate this problem we have designed and prototyped CORE Analytics Dashboard – a tool for analytical evaluation of research outputs of universities. The aim of the CORE Analytics Dashboard is to help universities analyse their performance using a variety of metrics captured from openly available data sources, including citation counts and social media metrics, and to help them compare their performance with other institutions. This paper presents the motivation behind developing this dashboard and its main features.

Publicado en: Educación
  • Sé el primero en comentar

  • Sé el primero en recomendar esto

CORE Analytics Dashboard

  1. 1. CORE Analytics Dashboard Petr Knoth Jozef Harag Drahomira Herrmannova June 13, 2019 – OR 2019, Hamburg, Germany CORE Big Scientific Data and Text Analytics Group Knowledge Media Institute, The Open University
  2. 2. Introduction • Research metrics becoming a key component of evaluation processes • Application of research indicators controversial, but demand for them is increasing • Universities want to understand and monitor the impact of their research outputs • This is evidenced by subscriptions to expensive products like Elsevier’s SciVal, Clarivate’s InCites • Typical cost: negotiated, it is believed £35-£50k per annum per institution
  3. 3. Existing solutions • Elsevier SciVal • Clarivate InCites • Digital Science Dimensions
  4. 4. Limitations of existing solutions 1/2 • Proprietary data (Scopus and Web of Science) • Data or results of analyses cannot be easily downloaded and shared with others • Algorithms used for citation data acquisition not open/known (Scopus and Web of Science) • Complicates confirming validity/replicating results
  5. 5. Limitations of existing solutions 2/2 • Data sparsity • Scopus/Web of Science are known to have lower publication/citation coverage than Google Scholar and Microsoft Academic [Harzing, 2017] • Dimensions use primarily citations from I4OC which is an even sparser citation dataset • Lack of APIs for integration of results into existing systems (Scopus and Web of Science) • Slice-and-dice operations on titles/abstracts only • Papers without DOI have metrics rarely available
  6. 6. CORE Analytics Dashboard • CORE is the largest full text Open Access aggregation service [Notay, 2018] • Because it is based on Open Access data, it can offer/enable: • Slice-and-dice operations based on full text • Full access to underlying data • API integration with internal university systems
  7. 7. CORE Analytics Dashboard • A tool for analytical evaluation of universities’ research outputs based primarily on CORE data • Research impact insights on institutional level • Comparison with other institutions or groups of institutions • Based on citation data, social impact data • Enables users to design and create custom graphs and add these to their user area • Intended for university research officers, repositories, university management
  8. 8. Use cases • A university's Research Office manager wants to see how her institution compares to another institution • A university's Research Office manager wants to have an overlook on research impact of a hot-topic (e.g. Artificial Intelligence, DNA mapping, global warming, etc.) across UK universities • A non-Russell Group university wants to demonstrate (for marketing reasons) that they have the same research and social impact (or perhaps better) as any of Russell Group universities in a given field.
  9. 9. Data sources 1. CORE dataset • Primary pivot, provides publication metadata, affiliation information 2. Microsoft Academic Graph • Citation data 3. Crossref event data • Publications’ Wikipedia and Twitter mentions 4. Mendeley • Readership data, i.e. information about how many Mendeley users have added a certain publication in their Mendeley library
  10. 10. Data • Data from across different systems were matched using DOIs • Prototype version: • UK institutions • Outputs with a DOI • Long-term goal is to expand to the whole word
  11. 11. Interface overview • Individual for each user • Each user assigned a specific institution • Customizable user area: users can define metrics and graphs to track & display • Institutional ranking
  12. 12. User interface: statistics bar • Each user assigned a specific institution • Quick summary overview of institution’s performance
  13. 13. User interface: dashboard • Customizable area, user-defined charts
  14. 14. User interface: institutional ranking • Ranking according to a number of metrics
  15. 15. Customizable area • Enables creating user-defined graphs • Chart data can be downloaded: • Original (raw) data • Aggregate chart data • Chart PNG • Full customizable, enables resizing and reorganizing charts
  16. 16. User area: customizing the dashboard • A step-by-step tool (benchmarking wizard) for creating custom visualizations tailored to user’s needs • Works in three steps: 1. Define publications sample filtered to select search criteria 2. Define what indicators to inspect 3. Select visualization chart to apply on the filtered data
  17. 17. User interface: benchmarking wizard 1. Filter data – user specifies which articles to work with • E.g. using keywords or phrases of interest that will be searched in publications’ titles, abstracts, and full texts • Also enables filtering, currently by year and publication type (research article, thesis, etc.) 2. Selecting metrics to track – user defines which institutions to compare and based on what metrics to compare them 3. Designing the chart – select chart type and name the chart • Several chart types available including line charts, bar charts, scatter plots, etc. • User can customize colors, names of axes, etc.
  18. 18. User interface: benchmarking wizard 1. Filter data – user specifies which articles to work with
  19. 19. User interface: benchmarking wizard 1. Filter data – user specifies which articles to work with
  20. 20. User interface: benchmarking wizard 2. Selecting metrics to track
  21. 21. User interface: benchmarking wizard 3. Designing the chart
  22. 22. Conclusions • We presented CORE Analytics Dashboard – a tool designed to enable users to analyse and compare the performance of research outputs between universities along a variety of metrics • Key difference from existing solutions – focus on collecting performance indicators from openly available sources • Goal – add a layer of transparency to research evaluation • Goal – extend to whole world and add additional features • E.g. collaboration graphs, UK REF predictions [Pride, 2018]
  23. 23. Feedback • • Code 23 36 84
  24. 24. References • Harzing, Anne-Wil, and Satu Alakangas. (2017). "Microsoft Academic is one year old: The Phoenix is ready to leave the nest." Scientometrics 112.3: 1887-1894. • Notay, Balviar. (2018). "CORE becomes the world’s largest aggregator". Jisc scholarly communications. ore-becomes-the-worlds-largest-aggregator/. Accessed: 2019- 01-16. • Pride, D. and Knoth, P. (2018). "Peer review and citation data in predicting university rankings, a large-scale analysis". Lecture Notes in Computer Science. Springer.
  25. 25. Thank you! Demo: