1. Usage Factor
a new metric to measure journal quality
Jayne Marks
Co-Chair, Usage Factor Project
UKSG
March 27, 2012
Usage Factor – a COUNTER project

2. Usage Factor: agenda
• Why usage-based measures?
• Strengths and weaknesses of the Impact Factor
• Background and aims of the Usage Factor project
• The story so far:
• Stage 1 – market research
• Stage 2 - modelling an analysis
• Results
• Recommendations
• Next steps
2

3. Usage Factor: background
Why usage-based measures?
• A growing body of reliable journal usage statistics
• The burgeoning availability of reliable usage data for online journals has
opened the door to usage-based measures of impact, value and status.
• Since 2002 COUNTER has provided a global standard for usage statistics
• A complement to citation-based measures
• Impact Factors, based on citation data, have become generally accepted as a
valid measure of the impact and status of scholarly journals
• Widely used by publishers, authors, funding agencies and librarians
• There are misgivings about an over-reliance on Impact Factor alone
• Online availability of content combined with the availability of reliable
COUNTER-compliant online usage statistics, raises the possibility of a parallel
usage-based measure of journal performance becoming a viable additional
metric – the Usage Factor for journals
3

4. Usage Factor - background
Journal Impact Factors: strengths
• well-established, widely recognised, accepted and understood
• difficult to defraud
• endorsed by funding agencies and scientists
• simple and accessible quantitative measure
• independent
• global
• journals covered are measured on the same basis
• comparable data available over a period of decades
• broadly reflect the relative scientific quality of journals in a given
field
• faults are generally known
4

5. Usage Factor: Background
Journal Impact Factors: weaknesses
• bias towards US journals; non-English language journals poorly covered
• optimized for biomedical sciences, work less well in other fields
• as an arithmetic mean, a journal IF can be dramatically influenced by
one highly cited article
• can be manipulated by, e.g., self-citation
• over-used, mis-used and over-interpreted
• formula is flawed; two year time window too short for most fields
• only a true reflection of the value of a journal in pure research fields
• impact of practitioner-oriented journal is understated
• does not cover all fields of scholarship
• over-emphasis on IF distorts the behaviour of authors and publishers
• time-lag before IFs are calculated and reported; new journals have no IF
• emphasis on IF masks the great richness of citation data
5

6. Usage Factor: background
Usage Factor: providing a new perspective
UF is a complementary measure that will compensate for the
weaknesses of Impact Factors in several important ways:
• UFs will be available for a much larger number of journals
• coverage of all fields of scholarship that have online journals
• impact of practitioner-oriented journals is better reflected in usage
• usage is recorded and reported immediately upon publication of an
article
• availability of UF will reduce the current over-emphasis of IFs
• authors would welcome a usage-based measure for journals
6

7. Who will benefit from the Usage Factor?
Four major groups will benefit from the introduction of Usage Factors:
• Authors, especially those in practitioner-oriented fields, where citation-based
measures understate the impact of journals, as well as those in areas outside
the core STM fields of pure research, where coverage of journals by citation-
based measures is weak.
• Publishers, especially those with large numbers of journals outside of the
core STM research areas, where there is no reliable, universal measure of
journal impact, because citation-based measures are either inadequate or
non-existent for these fields
• Librarians, when deciding on new journal acquisitions, have no
reliable, global measures of journal impact for fields outside the core STM
research fields. They would use usage-based measures to help them
prioritise journals to be added to their collections.
• Research Funding Agencies, who are seeking a wider range of
credible, consistent quantitative measures of the value and impact of the
outputs of the research that they fund.
7

8. Usage Factor Project
- aims and objectives
The overall aim of this project was to explore how online journal usage
statistics might form the basis of a new measure of journal impact and
quality, the Usage Factor for journals.
Specific objectives were to answer the following questions:
• Will Usage Factor be a statistically meaningful measure?
• Will Usage Factor be accepted by researchers, publishers, librarians and
research institutions?
• Will Usage Factor be statistically credible and robust?
• Is there an organizational and economic model for its implementation
that would cost-effective and be acceptable to the major stakeholder
groups.
The project is being carried out in three Stages:
• Stage 1 ( 2007-2008): market research
• Stage 2 (2009-2011): modelling and analysis
• Stage 3 (2011-2012): further tests based on draft Code of Practice
8

9. Usage Factor Stage 1
- market research
Interviews with 29 key authors/editors, librarians and publishers; web-
based survey of 155 librarians; web-based survey of 1400 authors.
Key findings:
• the majority of publishers were supportive of the UF concept, appeared
to be willing, in principle, to participate in the calculation and
publication of UFs, and were prepared to see their journals ranked
according to UF
• the great majority of authors in all fields of academic research would
welcome a new, usage-based measure of the value of journals
• UF, were it available, would be a highly ranked factor by librarians, not
only in the evaluation of journals for potential purchase, but also in the
evaluation of journals for retention or cancellation
• COUNTER was on the whole trusted by librarians and publishers and
was seen as having a role in the development and maintenance of the
UF
9

10. Usage Factor Stage 2
- modelling and analysis
• Real journal usage data analysed by John Cox
Associates, Frontline GMS and CIBER
• Participating publishers:-
• American Chemical Society
• Emerald
• IOP
• Nature Publishing
• OUP
• Sage
• Springer
10

11. The data
• 326 journals
• 38 Engineering
• 32 Physical Sciences
• 119 Social Sciences
• 29 Business and Management
• 35 Humanities
• 102 Medicine and Life Sciences
• 57 Clinical Medicine
• c.150,000 articles
11

12. Recommendations: the metric
• Usage Factors should be calculated using the median rather than
the arithmetic mean
• Usage data re highly skewed; most items attract relatively low use
and a few are used many times. As a result, the use of the
arithmetic mean is not appropriate
• A range of Usage Factors should ideally be published for each
journal: a comprehensive UF ( all items, all versions) plus
supplementary factors for selected items
• There is considerable variation in the relative use made of
different articles types and versions. This means that the UF will
be affected significantly by the particular mix of items included in
a given journal, all other things being equal
12

13. Recommendations: the metric
• Usage Factors should be published as integers with no decimal
places
• Monthly patterns of use at the item level are quite volatile and
usage factors therefore include a component of statistical noise
• Usage Factors should be published with appropriate confidence
levels around the average to guide their interpretation
• As a result of this statistical noise, the mean usage factor should
be interpreted within intervals of plus or minus 22 per cent
(Amin and Mabe* have pointed out that ISI Journal Impact Factors are
subject to similar statistical factors and recommend that Impact Factors
that differ by less than 25% belong together in the same rank)
* Amin, M and Mabe, M., Impact Factors: Use and Abuse, Elsevier, Perspectives in
Publishing, October 2000.
http://www.elsevier.com/framework_editors/pdfs/Perspectives1.pdf
13

14. Recommendations: the metric
• The Usage Factor should be calculated initially on the basis of a
maximum usage time window of 24 months.
• It might be helpful later on to consider a 12-month window as well (or
possibly even a 6-month window) to provide further insights
• This study shows that relatively short time windows capture a
substantial proportion of the average lifetime interest in full
journal content.
• Longer windows than 24-months are not recommended and this
should be considered a maximum.
• There is possibly a case for considering a 12-month window, but
there are counter-arguments here. On average only 25-30% of
lifetime usage takes place during the first twelve
months, compared with 60-65% during the first 24 months
14

15. Recommendations: the metric
• The Usage Factor is not directly comparable across subject groups
and should therefore be published and interpreted only within
appropriate subject groupings.
• Usage, in months 1-12 especially, follows different patterns in
different subject areas
• The Usage Factor should be calculated using a publication window
of 2 years
• Usage factors will tend to inflate across the board year-on-year as
a result of many factors, including greater item discoverability
through search engines and gateways. The use of a two-year
publication window would ameliorate some of these effects by
providing a moving average as well as a greater number of data
points for calculating the usage factor.
15

16. Recommendations: the metric
• There seems to be no reason why ranked lists of journals by usage
factor should not gain acceptance
• The usage factor delivers journal rankings that are comparable in
terms of their year-on-year stability with those generated from
citation metrics such as the ISI impact factor and SNIP
• Small journals and titles with less than 100 downloads per item are
unsuitable candidates for Journal Usage Factors: these are likely to
be inaccurate and easily gamed
• Usage factors below a certain threshold value (perhaps 100 but
research is needed on a larger scale to explore this further) are
likely to be inaccurate due to statistical noise. The size of the
journal should also be taken into account
16

17. Recommendations: the metric
• The Usage Factor provides very different information from the citation
Impact Factor and this fact should be emphasised in public
communications.
• The usage factor does not appear to be statistically associated with
measures of citation impact
• Further work is needed on usage factor gaming and on developing
robust forensic techniques for its detection
• Attempts to game the usage factor are highly likely.
• CIBER’s view is that the real threat comes from software agents
rather than human attack.
• The first line of defence has to be making sure that COUNTER
protocols are robust against machine attack.
• A second line of defence would be to develop statistical forensics to
identify suspicious behaviour, whether human or machine in origin.
17

18. Recommendations: the metric
• Further work is needed to broaden the scope of the project over
time to include other usage-based metrics
• Although the scope of this study was to consider the journal
Usage Factor only, future work could look at the other indicators
that mimic other aspects of online use, such as a ‘journal usage
half-life’ or a ‘reading immediacy index’.
18

19. Recommendations: infrastructure
• Development of systems to automate the extraction and collation
of data needed for UF calculation is essential if calculation of this
metric is to become routine
• Development of an agreed standard for content item types, to
which journal specific item types would be mapped, is desirable
as it would allow for greater sophistication in UF calculation
• Development or adoption of a simple subject taxonomy to which
journal titles would be assigned by their publishers
• Publishers should adopt standard “article version” definitions
based on ALPSP/NISO recommendations
19

20. Next steps: Stage 3
Objectives
• Publication of a draft Code of Practice for the Usage Factor
• Further testing of the recommended methodology for calculating
Usage Factors for journals
• Investigation of an appropriate, resilient subject taxonomy for
the classification of journals
• Exploration of the options for an infrastructure to support the
sustainable implementation of Usage Factor
• Investigate the feasibility of applying the Usage Factor concept to
other categories of publication in addition to journals
20

21. Usage Factor
The draft Code of Practice
• The Code of Practice will be consistent with COUNTER and will provide:
• A list of Definitions and other terms that are relevant to Usage Factor
• A methodology for the calculation of Usage Factor as a median
value, including specifications for the metadata to be recorded, the
content types and article versions whose usage may be counted, as well
as the Publication Period and Usage Period to be used.
• Specifications for the reporting of the Usage Factor
• Data processing rules to ensure that Usage Factors are
credible, consistent and compatible, including protocols for identifying
and dealing with attempts to game the Usage Factor
• Specifications for the independent auditing of Usage Factors
• A description of the role of the Central Registry for Usage Factors in the
consolidation of usage data and in the publication of Usage Factors
• The draft Code of Practice will be published in Q1 2012
• Publishers are being invited to prepare UFs using the draft Code of
Practice
21

22. Usage Factor: organization
Project Co-Chairs:
• Jayne Marks, Wolters Kluwer, USA
• Hazel Woodward, Cranfield University, UK
International Advisory Board Members
• Mayur Amin, Elsevier, UK
• Kim Armstrong, CIC Center for Library Initiatives, USA
• Peter Ashman, BMJ Group, UK
• Terry Bucknell, University of Liverpool, UK
• Ian Craig, Wiley, UK
• Joanna Cross, Taylor & Francis, UK
• David Hoole, Nature Publishing Group, UK
• Tim Jewell, University of Washington, USA
• Jack Ochs, ACS Publications, USA
• Tony O’Rourke, IOP Publishing, UK
• Clive Parry, Sage Publications, UK
• Jason Price, Claremont College, USA
• Ian Rowlands, CIBER, UK
• Bill Russell, Emerald, UK
• Ian Russell, Oxford University Press, UK
• John Sack, HighWire Press, USA
• David Sommer, COUNTER, UK
• Harald Wirsching, Springer, Germany
Project Director
• Peter Shepherd, COUNTER
22

23. Usage Factor project
Acknowledgements
• Major funding from UKSG and RIN
• Additional funding from:
• ALPSP
• American Chemical Society
• STM
• Nature Publishing Group
• Springer
• Usage data for Stage 2 analysis provided by the American Chemical
Society, Emerald, IOP, Nature Publishing Group, OUP, Sage, Springer
For further information
Access the full report on Stages 1 and 2 of the Usage Factor project, as well as
information on the progress of Stage 3 on the COUNTER website at:
http://www.projectcounter.org/usage_factor.html
23