Presentation by Carol Tenopir at the LIS DREaM final conference.
More information about this event is available at http://lisresearch.org/dream-project/dream-event-5-conference-monday-9-july-2012/
Enhancing Worker Digital Experience: A Hands-on Workshop for Partners
DREaM 5: Building evidence of the value and impact of library information services
1. DREaM Project Conference
British Library, 9th July 2012
Opening keynote presentation
Carol Tenopir
University of Tennessee, USA
Center for Information and Communication Studies
2. Building evidence of the value and
impact of library and information
services: Methods, metrics, and ROI
Carol Tenopir
University of Tennessee
ctenopir@utk.edu
DREaM Conference
July 9, 2012
Center for Information and Communication Studies
4. Today‟s topics…
• Lib-Value project overview
• Defining value in the library and
information context
• Techniques and examples:
– Critical incident
– ROI and contingent valuation
– Qualitative and personas
Center for Information and Communication Studies
5. Lib-Value: Multiple academic institutions using
multiple methods to measure multiple values for
multiple stakeholders
Center for Information and Communication Studies
6. Current projects
Special Information
Ebooks
Collections Commons
Methods for
Journal Teaching and
Measuring
Collections Learning
Value
Website and
Reading and
Digitization Value
Scholarship
Bibliography
Center for Information and Communication Studies
7. In the information context economist
Machlup described 2 types of value:
1. purchase or exchange value: what one
is willing to pay for information in money
and/or time, and
2. use value: the favorable consequences
derived from reading and using the
information.
Center for Information and Communication Studies
8. Lib-Value comprehensive library value
study (Bruce Kingma)
• Economic (private)
– What is the value to an individual to use the library
resources?
• Social (public)
– What is the value to the institution of the library?
• Environmental (externality)
– What is the value of the environmental savings of
library provision of electronic resources?
– Have libraries gone green without knowing it?
Center for Information and Communication Studies
9. Value of reading can be
measured in many ways*
1) Implied value (i.e., usage, downloads)
2) Explicit value (i.e., outcomes, critical incident)
3) Derived values (i.e., contingent valuation, ROI)
*These methods are useful in any type of library, but
most of my examples are from academic libraries
Center for Information and Communication Studies
11. Article readings 1977 to present by
scientists and social scientists
Readings per year
*
*2011-2012 (UIUC) n=639, (UK),n=1013;
2005,n=932; 2000-03, n=397; 1993, n=70;
1984, n=865; 1977, n=2350 Center for Information and Communication Studies
12. Going beyond implied value to
show…
Exchange
Use/Outcomes
Center for Information and Communication Studies
13. Tenopir & King scholarly reading
studies, 4 types of questions:
1. Demographic
2. Recollection Therefore, insights into
both READERS and
3. Critical Incident READINGS
4. Comments
Center for Information and Communication Studies
14. Critical incident of last reading
“The following questions in this section
refer to the SCHOLARLY ARTICLE YOU
READ MOST RECENTLY, even if you
had read the article previously. Note that
this last reading may not be typical, but
will help us establish the range of
patterns in reading.”
Center for Information and Communication Studies
15. Time spent (exchange value) reading
•Article
•49 min/article X 22 read per month X 12 months =
216 hours
•Book
•106 min/book X 7 per month X 12 months=
148 hours
•Other Publication
•42 min/publication X 10 per month X 12 months=
84 hours
U.K. academics spend on average per year nearly
three months of their work time reading scholarly
material.
Center for Information and Communication Studies
16. A majority of articles are from
the library:
Percent
UK, n=1189, June 2011;
UIUC, n=256 April 4, 2012 Center for Information and Communication Studies
17. Use of library collections for articles
N=775, 6 UK universities, June 6 2011 Center for Information and Communication Studies
18. Environmental and exchange value
Average last 30 % of
Activity days respondents
Physical visits 2.9 visits 73%
Remote visits 14.2 visits 88%
Average total resources used:
in-person visit to the library 7.3 uses 80%
remotely online 14.9 uses 89%
Center for Information and Communication Studies
19. Source of reading by purpose of
reading: Faculty in UK
Teaching
Research
Current
Awareness
n=1161, 2011 Center for Information and Communication Studies
20. Outcomes of journal article reading
1st Inspire new thinking or ideas 54%
2nd Improve results 38%
3rd Narrow/broaden/change the focus 28%
4th Resolve technical problems 10%
5th Save time or other resources 10%
6th Aid in faster completion 5%
7th Assist or result in collaboration/joint 4%
research
n=2117, 6 UK universities,
June 2011
Center for Information and Communication Studies
21. Return on investment in a strict sense…
…is a quantitative measure expressed as a ratio of
the value returned to the institution for each
monetary unit invested in the library.
For every $/€/£ spent on the library,
the university receives ‘X’ $/€/£ in return.
Demonstrate that library collections contribute to
income-generating activities
Center for Information and Communication Studies
22. Library‟s value to the grants process
9 institutions in 8 countries
http://libraryconnect.elsevier.com/whitepaper
s/roi2/lcwp021001.html Center for Information and Communication Studies
23. ROI from access to journal articles cited in grant
proposals
Research
• 13.2:1 to 15.5:1
STM
Research and
Teaching • 1.3:1 to 3.4:1
STM/Hum/SS
Research and
Teaching • Under 1:1
Center for Information and Communication Studies
24. ROI through contingent valuation
an economic method of evaluation of services
and goods which looks at the implications of
not having the services.
Center for Information and Communication Studies
25. National Network of Libraries of
Medicine: Retail Value Calculator
• http://nnlm.gov/mcr/evaluation/calculator.html
• How much would it cost to replace your
library services on the retail market?
• Calculate what it would cost to buy library
services - at a book store, through pay per
view for articles, from an information broker -
if you and your library weren't there.
Center for Information and Communication Studies
26. National Network of Libraries of
Medicine: Library value calculator
• http://nnlm.gov/mcr/evaluation/roi.html
• How much benefit does your institution, your
user, receive for every dollar spent by the
library?
• Value of benefits and costs for each service
• Total value of your library use
Center for Information and Communication Studies
27. Measuring the ROI of Today‟s
Libraries: About the study:
•Part of a grant to assess the
value and ROI of academic
library services (Lib-value)
•Assesses the
use, value, and ROI of 77
academic library services
•This data is from Bryant
University
•Similar project underway at
Drexel University (May
2012).
Center for Information and Communication Studies
28. Readings from the Library
Total Number of Readings from Library per year
No. of readings from library
Center for Information and Communication Studies
29. Purchase Value of Some
Services:
• The hourly rate of users:
– Faculty: $56.20 per hour
– Staff: $36.00 per hour
– Students: $34.60 per hour
Center for Information and Communication Studies
30. Contingent Value
• The cost of not having access to library-
provided articles.
• Faculty-only (students not asked)
• Total cost: $408,600
• Cost per Faculty: $1,200
• Cost per reading: $27
Center for Information and Communication Studies
31. Return on investment is also…
…values of all types and outcomes
that come to stakeholders and the
institution from use of the library‟s
collections, services, and
contribution to its communities.
Center for Information and Communication Studies
32. Academics praise the library
for its long-term outcomes
Electronic access to the
university library system from off-
site is crucial for swift access to
The journal
articles to support my teaching
collection at
and research activities.
my institution
is excellent
and
scholarship
is all the
richer for the Library resources have
contribution been essential to my
for easy work for the past 20
access to years.
journals and
print
publications.
Center for Information and Communication Studies
33. What a „successful‟ academic
looks like:
•Has won an award in the last two years.
•Publishes four or more items per year.
•Reads more of every type of material.
•Spends more time per book and other publication
readings.
•Uses the library for articles
•More often buys books and obtains other
publications from the Internet.
•Occasionally participates and creates social
media content.
Center for Information and Communication Studies
34. Persona: “Akrum Patel”
• Key Facts:
I would like my library • Associate professor in physics.
to have subscriptions
to more journals and • Reads 30 articles, 2 books, and 11 other
for longer periods of publications per month.
time.
• What he needs:
• Current issues of articles.
• Off-site access to collections.
• Access to search engines and online resources
without a distinction between library and non-library
resources.
• Factors:
• Reads seminal books.
• Wants to see trends over time.
• Has not visited a physical library for many years.
Center for Information and Communication Studies
35. Persona: “Sally Fitzgerald”
My research and teaching • Key Facts:
cannot exist without – Spends majority of time on research
[library‟s e-collections]…and and teaching.
not finding them right away
is heavily disruptive on my – Reads 30 articles and 15 books per
work. month.
• What she needs:
– Older articles in addition to new
publications.
– Frustrated when can only find abstracts
and not full-text.
• Factors:
– Library doesn‟t always have the books
she needs.
– Needs wide range of material.
Center for Information and Communication Studies
36. Value is demonstrated by time
invested, by value to purpose, by
outcomes of use and by ROI.
Multiple methods should be
used to measure value.
Center for Information and Communication Studies
37. Some final thoughts on measuring value
• Tie what you measure to the
mission
• Quantitative data can show ROI
and trends
• Qualitative data tell a story
• No one method stands alone
• Measure outcomes, not inputs
• The further downstream the
value, the more challenging to
measure, but perhaps the
most important.
Center for Information and Communication Studies
38. More details and searchable Lib-Value
bibliographic database available on the
project website:
http://libvalue.cci.utk.edu
Center for Information and Communication Studies
At the same time circulation of print materials remains flat at best and declining for some.Total circulation across libraries: (Kyrillidou, M. & Morris, S. (Editors). ARL Statistics 2008-2009. Association of Research Libraries: Washington D. C. 2011, p. 8)2001: 467,2772002: 462,2232003: 479,7332004: 496,3692005: 473,216 (start of decline)2006: 466,4032007: 456,5972008: 429,6262009: 414,482
Excludes outliers and humanities. 298 in the UK, 283 in the UIUC to date
Implied values do not show purpose, satisfaction, or outcomes of use, however (or whether what is downloaded was actually read). Because of that and, because libraries have always been good about gathering implicit value information, today I am going to focus on going beyond implicit measures and talk about methods to measure value beyond usage.
Measure purpose, outcome, and value from scholarly reading by focusing on critical incident of last readingInclude all reading (from library and not)Exchange (time spent) and use value (outcomes)Open ended questions provide another dimensionMethod can be used for other services
“Add that all together and you get a huge investment in time”One definition of value is:(use value) the favorable consequences derived from reading and using the information. Or “purchase or exchange value: what one is willing to pay for information in money and/or time” (Machlup)-What is illustrated in the slideYearly total of 56 eight-hour days or or nearly 3 work-months…Monthly Totals:Article reading (total)=21 hours (14 hours from library resources)Book reading (total)=16 hours (4 hours from library resources)Other reading (total)=9 hours (an hour and a quarter from library resources)
We defined repository as “institutional repository” in the US surveys and “subject or institutional repository” in the UK.UK:Other—4.5%-Dept/school subscription; 2.2% repository; 2.4% other (email, conference, publisher, office)UIUC:--8.6% dept/school subscription (included with library); 2.7% repository; 5.9% other (email, conference, publisher, office)
Question: Did you obtain your article through a print or electronic source? UK: 94% electronic; 6% print (n=775)UIUC, March 15, 2012: 91% electronic, 9% print (9 of 106)Location of reading (library-provided articles only)UK: 62% office/lab, 26% home, 10% travel, 2% libraryUIUC: 73% office/lab, 21% home, 3% library, 3% other (includes travel, coffee shop, etc.)
Value of virtual visits to the library.
Library-provided articles are more likely to be more important to principal purpose (statistically significant, F=4.911, p=.027). Library articles/other sources: 14%(library)/11% (other) absolutely essential, 27%(lib)/22% (other) very important, 28%(lib)/32%(other) important, 31% (lib)/33% (other) somewhat important, 0% (lib)/2% (other) not important.The “average” level of importance is: 3.24 for library (between important to very important), 3.00 for persona source and 3.05 for other source (It is 3.07 when personal and other are combined). It is statically significant but I don’t think it will look very interesting on a slide…)Not shown, but library articles also more likely to be cited (statistically significant-p=-.086) Articles from library: 56% cited, 33% maybe cited, (89% already or may be cited) and 11% not cited Articles not from library: 46% cited, 39% may be cited, (85% already or may be) and 16% not cited
Outcomes from reading are another important explicit value that go beyond just sheer amount of reading. From nearly 900 respondents in the United States (Tenopir et al, 2009a), readings were found to have many important outcomes. Faculty members told us that their Readings have profound outcomes. They:• Inspire new thinking (55%/54 of readings)• Improve results (40%/38 of readings)• Narrow, broaden, or change the focus (27%/28 of readings)• Resolve technical problems (12%10 of readings)• Save time (12%/10 of readings)• Lead to faster completion of the task at hand (7% /5 of readings)• Foster collaborations (6%/4 of readings)• Waste the time of the reader (<1%/<1 of readings)
Valuing Library Services Calculator What is your library worth to your institution? How much would it cost to replace your library services on the retail market? Calculate what it would cost to buy library services - at a book store, through pay per view for articles, from an information broker - if you and your library weren't there. Select your state and your library type, enter annual library budget and the number of months of statistics you are using. Consumer Health Libraries may be sponsored by a hospital, ,public library or other organization.Enter in the left hand column the number of times your library's services are used for those months.Either use the estimated retail cost - not your library cost - provided for each service or change it to reflect your area or library. Click here to see how to determine the cost of your services.In the three bottom lines enter any other services or resources you would like to include in your calculation along with the number of uses and the retail value for each service or resource.As you enter data the retail value of each service will be calculated on the right and the return for every dollar spent will be displayed.The total value of your library services is shown at the bottom of the worksheet.
Library Value Calculator What is your library worth? How much benefit does your institution, your user, receive for every dollar spent by the library?Enter a user's salary (no commas), enter the total library salary budget.Tab to each box and enter appropriate numbers for one year.Value of Benefits and Costs for each service will be calculated on the right.Total value of your library use is shown at the bottom of the worksheet. Tip: Tab between entries and do not use commas. Click here for information about each data point.Also a calculator for databases ROIAnd lots of evaluation toolshttp://nnlm.gov/mcr/evaluation/tools.html
Bryant university is a small, private university in Rhode Island. It has two colleges—business and Arts & Sciences.Its library currently offers 77 services, including Access to articles, books, and other publications, Reference searches, Formal instruction by librarians, 81 work stations, Leisure/workspacesIt services 3 types of users—faculty (164 full, 94 part), staff (465), and students (3216 undergrad, 256 graduate). Employs 17 librarians.
No staff readings from the library were observed; so results from faculty and students. If we take all of the reported readings from the library and extrapolate to the entire year and entire population, these self-reported numbers are quite consistent with usage figures.
The return component of ROI is determined by the value gained from reading or using services2 types of value:Purchase value: what readers or users pay for information in their time and/or money to process and read publications or process services.Use value: favorable consequences from reading or using services.Faculty and Staff rates determined by salary and fringe benefits divided by 2,200 hours. 2,200 hours assumes faculty and staff often work more than 40 hours a week. Student rates determined by tuition divided by number of hours involved with the university. Student’s number of hours include classes and other activities
Not sure if this is cost per year.
Personas are: Fictional character built on actual user data, Tool to represent target audience, Put a ‘face’ on the customer.How we created the personas:Separate respondents into groups based on similar demographic characteristics.Analyze the data by group and note patterns in accessing, obtaining, and reading.Further divide groups by similar patterns, which become the main basis of the personas.Populate each persona with real world examples from open-ended comments.Akrum=His factors are based on frequencies and averages of the survey questions along with the open-ended comments. We received many comments by successful scientists about referring to books that are “standard teaching texts” and books they have “known about for a long time”. Coupled with the fact that we received many comments about already having the book on their shelf or they got it from a colleague (answers to where they obtained the book), we determined that they are reading books that are the building blocks/seminal in their fields and using articles for new information (up-to-date research, etc.).“Trends over time” is from the year of article publications. We saw a wide range of publication dates, and while they are reading a lot of current articles they are also reading a lot of older articles. We also received many comments, such as the one on the slide.