1. A study analyzed the impact of four discovery services on journal article usage across six publishers and 33 libraries over two years. 2. The study found that every discovery service increased journal usage compared to the control group, but the size of the increase differed between services and across libraries and publishers. 3. Usage changes were influenced by factors like library configurations, metadata quality, and availability of full text in aggregators. More research is needed on how discovery impacts other content types and why certain results occurred.

1. Discovery or Displacement?
A Large-Scale Longitudinal Study of the
Effect of Discovery Systems on Online
Journal Usage
December 2014
Combined Presentation
Michael Levine-Clark, University of Denver
John McDonald, University of Southern California
Jason Price, SCELC Consortium

2. …our customers insist that usage of our content
decreased after implementation of discovery service “X”.
A publisher told us . . .
https://www.flickr.com/photos/kongping/7192138660/in/gallery-flickr-72157645846953449/

3. Librarians speculate . . .
…of course discovery vendors direct their users
to their own aggregated content.
http://www.flickr.com/photos/
hypotekyfidler/15012731920

4. Goals of our research
• Determine whether discovery services impact
usage
• Help librarians and publishers understand
how their choices impact use
• Help librarians, publishers, and vendors
improve the discovery experience for end
users

5. Why Do We Use Discovery Services?
• Too many sources of information
– Specialized
– Confusing
– What’s the right specialized database for my
subject?
• Why do you search in one place for an article,
another place for a different article, another
for a book?

6. Terminology
Discovery Tool = Discovery Layer =
Discovery Service = Discovery System

8. Web-scale discovery services
• Single source for
finding information
– Books
– Articles
– Local content
• Metadata and/or full
text
• Content is pre-indexed
and/or pre-harvested
• Single fast search
ILS
HathiTrust
MLA
Bibliograph
y
Institutional
Repository
Publisher
Metadata
Discovery Service

9. Does implementation of a discovery
service impact usage of publisher-
hosted journal content?

10. What did we measure?
• Whether there is an
effect
• NOT why that effect
exists (that’s a future
study!)

11. Caveat 1: Publisher-hosted journals are
only part of the picture
eBooks, pBooks, newspaper articles, aggregator
journal content, etc.
publisher journal content
The six publishers in this study
Not to scale!

12. 52.3
12.4
1
2.8
32.5
Open web search
Library referrals
Social media
Academic
N/A
Journals Traffic Sources
(SAGE, Conrad ALPSP 2013)

13. An assumption
• At any given institution, given a relatively
stable user base, the total search effort will
remain roughly the same.

14. Discovery services
 Will take up an increasing amount of a finite
time for searching
 Will draw users from other (more or less
efficient) search tools
 Will alter the overall productivity of searches
(users will find more or less)
 Will alter the overall efficiency of users (users
will access more or less full-text)

15. Caveat 2: More usage may not be better!
• Decreased usage might be a sign of greater
efficiency
– Relevant articles found faster = fewer articles to examine
OR
– Fewer articles examined because other relevant content
types found

16. Did Journal Usage Change?
(and if so, to what extent?)
• 4 discovery services
– 6 libraries in each group
• A control group
– 9 libraries that did not implement a discovery
service in this time period
• 6 major journal publishers
– 9,206 Journals in the study
– 163,545 Observations (Library + Journal)

17. Participating libraries
• 157 asked for permission, 155 granted permission
– 124 from the US, 33 from other English-speaking countries
• Has your library used a different discovery service in
the past?
– Only libraries answering “No” were selected
• Is your discovery service featured on your homepage?
– All participants answered, “Yes, with a search box”
• To what extent did your library market the discovery
service at its release?
– 4 said “None”, one from each vendor
– 12 said “A limited extent”, 2 WCL, 2 EDS, 4 Summon, 4 Primo
– 8 said “A significant extent, 4 WCL, 2 EDS, 1 Summon, 1 Primo

18. Dataset
• 33 Libraries
– 28 US, 2 CA, 1 each from UK, AUS, NZ
– WorldCat book holdings
> Average: 1,114,193 ; Range: ~300k to ~2.6mil
– 4 discovery groups, of 6 libraries each
– 1 control group, 9 libraries
• Implementation dates (Discovery Libraries):
> 2010 (3), 2011 (19), 2012 (2)
• 6 Publishers
• 9,206 Journals
• 163,545 Usable Observations

19. Methodology
Compared COUNTER JR1 total full text article views for the
12 months before vs 12 months after implementation date
June2010Start
Implementation
May2011
May2012
End
Year 1 Year 2
Included implementation month in Year 1 to ensure that
both periods included an entire academic year

20. Examine Data for Outliers

21. Analyzing Usage Change: % vs Total
Use 12
months
before
Use 12
months
after
% Change
Total
Change
Journal A 500 600 20% 100
Journal B 5 15 200% 10
Which is the better measure?
Is it the same for publisher- & journal-level data?

22. Observations by Publisher

23. TotalStudentFTE

24. Journals by Library & Service
EDS Primo Summon WorldCat Control

25. Average Journal Usage by Library

26. Testable Effects
• Discovery Service
– Implemented by multiple libraries
– Used to find content from all publishers
• Publisher
– Accessible in all discovery services
– Accessible across all libraries
• Library
– Uses content from multiple publishers
– Uses only one discovery service

27. Full Model
Including Discovery Service, Publisher, and Library
Including Discovery Service, Publisher, and Library

28. Nested ANOVA Model Results

29. How does usage change differ across discovery services?
A
BB
C
D
Letters indicate statistically significant differences (Tukey multiple comparisons, p <
.05)

30. How does usage change differ across publishers?
Publisher (sorted by Mean Change)
C
Letters indicate statistically significant differences (Tukey multiple comparisons, p <
.01)
D
BBB
A

31. How does usage change differ across publishers?

32. Does usage change vary across libraries?
Institution (sorted by Mean Change)

33. Usage Change Per Institution: All Journals
Control EDS Primo Summon WCL

34. Usage Change By Institution: Pub 1
6.32
Primo WCLSummonEDSControl

35. Usage Change By Institution: Pub 2
Control EDS Primo Summon WCL

36. Usage Change By Institution: Pub 3
WCLEDS PrimoControl Summon

37. Usage Change By Institution: Pub 4
WCLEDSControl Primo Summon

38. Usage Change Per Institution: Pub 5
Control EDS Primo Summon WCL

39. Usage Change by Institution: Pub 6
WCLSummonPrimoEDSControl

40. Publisher 1 by Discovery System

46. Publisher 2 by Discovery System

52. Publisher 3 by Discovery System

58. Publisher 4 by Discovery System

64. Publisher 5 by Discovery System

70. Publisher 6 by Discovery System

76. Conclusions
1. Each discovery system caused an increase in usage
of publisher-hosted journal articles
– The size of the increase differed across discovery systems
2. Usage change differed among libraries using the
same discovery system
3. Discovery systems had stronger effects on some
publisher’s content than on others
4. [These results provide a snapshot of a very small
portion of the content that discovery services are
intended to expose, and should not be used to
assess discovery service quality]

77. Summary of Results
• Discovery Service
– Every service increased usage compared to control
– Some services increased usage more than others
• Library
– The degree of usage change differed among libraries
using the same discovery service
• Publisher
– Usage change differed across publishers:
• 1 of 6 publishers saw a significant decrease
• 2 of 6 publishers saw significant increases
• 3 of 6 publishers saw no detectable change

78. Conclusions to be avoided…
Our research does not indicate that:
a) one service increases usage more than another for
every library or publisher
 effects vary across libraries and publishers
b) one service is better than another
 libraries or their users may benefit from increased usage of
other content instead
 higher usage may indicate lower efficiency

79. Next Steps
• Design & test for effects of:
– Aggregator full text availability
– Linking configuration options in discovery services
• Expand pool of libraries
• Explore the why?
• Other possibilities
– Journal Subject?
– Journal age (archive vs current)?
– eBook usage?

80. 1. What is the best service: Summon, Primo, EDS or
Worldcat?
• Our research can’t prove one is better than another,
and usage is only one reason to install a discovery
system
2. Why was there so much variation in the effect of
implementation across publishers at my
institution? Why did some increase and some
decrease?
Potential Librarian Concerns (1)

81. 3. What about configuration differences?
• This undoubtedly contributes to some variance, so be
careful to review and maintain your configurations and
preferences.
4. What about other resource types (ebooks, print
books, etc)?
• They may be affected, we haven’t studied these.
5. Was it missing metadata that caused the
differences?
• Maybe, we can’t know for sure, so the publishers and
vendors need to come up with best practices for
metadata exchange to ensure everyone is on an equal
playing field (encourage participation in NISO ODI)
Potential Librarian Concerns (2)

82. Potential Publisher Issues (1)
1. Are users being directed to Aggregated full text
before publisher hosted full text?
• Maybe, but it might be the Library’s configuration that is to
blame.
• This could be counterproductive for aggregators.
2. Does implementation of abstract-based discovery
reduce the ranking of publisher content that is not
indexed (or available full text) in a library’s
aggregator databases?
• Maybe, but it its up to the Library to decide if that is a
desired effect or not.

83. Potential Publisher Issues (2)
3. Why was there so much variation in the effect of
implementation on change in usage of our content
across institutions using the same discovery
service?
• Because usage can affect the publisher bottom line, this is
a key question. Some publishers are finding it of value to
invest in research and engagement of customers at the
extremities of change after implementation.
4. How can publishers know what is being done with
the metadata they send?
• Discovery vendors need to be more proactive in proving
the positive benefits of providing more robust metadata

84. 1. How do we prove we’re content neutral?
• Develop best practices that indicate to libraries their
configuration and linking choices.
• Allow for independent studies
• Make the case for the risks related to bias?
2. Our systems make other resource types (ebooks,
print books, etc) more discoverable, does anyone
value that?
• More research needs to be done to assess
effectiveness for other content types
3. Does increased usage necessarily mean a
discovery system is better?
• Definitely not! Depends on what a library values.
Potential Discovery Vendor Concerns

85. What is the value of increased usage?
• For publishers and libraries:
– lower cost per use
– greater return on investment
– but there may be usage tradeoffs with other content types
• For users, usage decreases might be preferable
– Decreased usage might be a sign of greater efficiency
• Relevant articles found faster = fewer articles to
examine
OR
• Fewer articles examined, but other (more?) relevant
content types found

86. Long Term Goals
1. Determine whether discovery services impact usage
of various content types
• Journal articles first, but more to come
2. Reveal key factors that explain why some of the
differences exist
3. Help librarians and publishers understand how their
choices impact use
4. Help librarians, publishers, and vendors improve the
discovery experience for end users

87. Next Steps
• Design & test for effects of:
– Aggregator full text availability
– Journal age (archive vs current)
– Journal Subject
– Overall usage trends
– Configuration options in Discovery services
• Expand pool of libraries
• Explore WHY

88. Questions?
Michael Levine-Clark
michael.levine-clark@du.edu
John McDonald
johndmcdc@usc.edu
Jason Price
jason@scelc.org