Libraries have long sought to demonstrate the value of their collections through a variety of usage statistics. Traditionally, a strong emphasis is placed on high usage statistics when evaluating journals in collection development discussions. However, as budget pressures persist, administrators are increasingly concerned with looking beyond traditional usage metrics to determine the real impact of library services and collections. By examining journal usage in the context of scholarly communication, we hope to gain a more holistic understanding of the use and impact of our library’s resources. In this session, we begin by outlining our methodology for gathering comprehensive publication and citation data for authors affiliated with Northwestern University’s Feinberg School of Medicine, utilizing Web of Science as our primary data source and leveraging a custom Python script to manage the data. Using this data we discuss various potential metrics that could be employed to measure and evaluate journals in institutional and field-specific contexts, including but not limited to: number of publications and references per journal, co-citation networks, percentage of references per journal, and increases or decreases of references over time per title. We then consider the development of normalized benchmarks and criteria for creating field-specific core journal lists. We also discuss a process for establishing usage thresholds to evaluate existing journal subscriptions and to highlight potential gaps in the collection. Finally, we apply and compare these metrics to traditional collection development tools like COUNTER usage reports, cost-per-use analysis, Inter-Library Loan statistics and turnaway reports, to determine what correlations or discrepancies might exist. We finish by highlighting some use-cases which demonstrate the value of considering publication and citation metrics, and provide suggestions for incorporating these metrics into library collection development practices. Speakers: Joelen Pastva and Jonathan Shank, Northwestern University Project GitHub page: https://goo.gl/2C2Pcy

1. Capturing and Analyzing Publication,
Citation and Usage Data for
Contextual Collection Development
Presenters:
Joelen Pastva, Metadata Librarian
Jonathan Shank, Acquisitions & E-Resources Librarian
Project Team:
Ramune Kubilius, Collection Development, Special Projects Librarian
Karen Gutzman, Impact and Evaluation Librarian
Madhuri Kaul, Ph.D., Data Consultant
NASIG 2017, Indianapolis, IN

2. About us:
Galter Health Sciences Library
Northwestern University
Feinberg School of Medicine
Chicago, Illinois

3. Galter Health Sciences Library
• Serves Northwestern University’s Feinberg School of Medicine (FSM) in Chicago, Ill.
• Approx. 3,349 students, residents, and fellows
• Approx. 4,000 in the medical school’s faculty roster
• Staff (professional, research, support, etc.)
• Administratively separate from Northwestern University Library in Evanston
• Cost sharing with Evanston on big deal agreements and other large packages
• NU enterprise-wide system– Alma; custom front-end – Primo
• Separate standalone subscriptions and a medical-specific collection
• Centralized budget and selection model
• Cooperate with affiliated hospital libraries on some clinical medical resources
• Currently in transitional phase for handling of COUNTER
• No ERMS or usage client, efforts currently focused on JR1 stats
• Usage functionality coming to Alma in Summer of 2017
3

4. Project background
• COUNTER Overview
• Collection Development Motivations

5. COUNTER overview
• Standard format and “consistency” across vendors (Wical and Vandenbark 2014)
• Ease of utilizing for critical CPU analysis (Rathemacher 2010; Bordeaux, Kramer, and Sullenger 2005)
• Increasing compliance among vendors
• Growing interoperability
• Iterative improvements with each new release
• Active and engaged community of librarians, publishers and vendors
• Previous studies show COUNTER correlates significantly with other usage data
metrics like proxy logs, link resolver stats, web analytics, etc (De Groote, Blecic, and Martin
2013; Gao 2016)
Whatworks well
5

6. COUNTER limitations
• Merging multiple providers and platforms without a client (Luther 2002)
• Manual retrieval of reports and management of login credentials (Rathemacher 2010)
• Issues with accuracy and consistency with title changes, splits and merges
• Not all vendors are compliant or consistent with reports (Noonan 2007; Welker 2012)
• Interface & platform design can inflate stats (Davis and Price 2006)
• Usage is a relatively poor indicator of impact and value (Conger 2007; Noonan 2007)
• Conflicting studies on correlations with citation metrics, research activity & JIF
(Bollen and Van de Sompel 2008; De Groote, Blecic, and Martin 2013; Duy and Vaugh 2006; Gao 2016; Ralston et al. 2008)
• Incorrect IP information can distort figures
- 58% of IPs held by publishers to authenticate libraries are wrong (according to audit by PSI Ltd)
• Lack of distinction by location, school, campus, or department
Whatdoesn’t worksowell
6

7. COUNTER limitations
GHSL
Licenses
EMBASE ClinicalKey
Accesses
NUL
Licenses
ScienceDirect
Scopus
Cell Press
Accesses
NMH
Accesses
LCH
Licenses
ClinicalKey
Nursing
Accesses
Overview ofNU’sElsevier landscape
7

8. Project background
• COUNTER Overview
• Collection Development Motivations

9. Challenges for collection development
What resources best meet the needs of our users/institution?
Budget strain due to rising journal costs:
• What are essential titles, and what can we cut?
• How to demonstrate the value of library collections?
• How to demonstrate the impact of library collections?
Traditional usagedata
9
Idea
Preparation
ResearchWriting
DisseminationResource

10. Challenges for collection development
Traditional usagedata +citation analysis
What resources best meet the needs of our users/institution?
What are essential titles, and what can we cut?
 Most commonly cited journals
 Least commonly cited journals
 Number of citations per article
How to demonstrate the value of library collections?
How to demonstrate the impact of library collections?
 Citations show role journal title plays in generating and validating new scholarship
10
Idea
Preparation
ResearchWriting
DisseminationResource

11. Citation analysis: background
• Precedent – Citation studies date back to 1927, (Gross and Gross 1927) and have long been
recognized as a way to provide more context to supplement traditional usage data
- Methodology best practices (Hoffmann and Doucette 2012)
• Flexibility in choice of data source, scope, and tools
• General to local – citation patterns differ by field and institution, offering a more
localized view of the value of a resource (Belter and Kaske 2016, 420; Cusker 2012; Davis 2002, 157)
- Some studies contradict usage data (Gao 2016, 124; Ke and Bronicki 2015, 174)
- Some studies reinforce usage data (De Groote, Blecic, and Martin 2013, 117; Tsay 1998, 39)
11

12. Citation analysis: limitations
• Citing patterns potentially shaped by what library provides access to (Wilson and Tenopir
2008, 1395)
• Citations do not reflect overall use, such as what is used for instruction (De Groote, Blecic,
and Martin 2013, 111)
• Journals publishing more frequently tend to be cited more frequently (Blecic 1999, 21; Tsay,
35)
• Accuracy depends on source data for publications
• Time consuming process
12

13. Citation analysis
methodology

14. Citation analysis: data collection
• Web of Science (WoS)
- Northwestern University author affiliation
- Full publication record, including cited references
- 5 datasets
• 2007-2016, clinical, pre-clinical, and health (FSM)
• 2007-2016, dermatology
• 2016, clinical, pre-clinical, and health (FSM)
• 2016, dermatology
• 2016, all Northwestern University
- Subjects limited by WoS category
• Clinical, pre-clinical, and health represented by 45 total categories from Global
Institutional Profiles Project (GIPP) schema
14
Overall view
COUNTER comparison

15. Citation analysis: data wrangling
• WoS UI limits exports to 500 records at a time
- need to combine for analysis
• Parse cited references and year of publication columns
• Larger file sizes are more challenging
• Messy data
Translates to tedious (and possibly error-prone) work
15

16. Citation analysis: data wrangling
Solution - Python
Why Python?
• Simple, easy to learn
• Libraries developed for data munging and analysis
- NumPy, pandas, matplotlib, etc.
• Work can easily be replicated
• Eliminates potential for user error
• Faster (after initial time investment)
Drawbacks
• Larger files are slow to run
• Data inconsistencies require manual cleanup
16

17. Citation analysis: data wrangling
Steps for working with Python script:
1. Run script to clean and concatenate files output from WoS
- Cleans data for reading into pandas library
- Concatenates multiple files into one file for analysis
2. Run concatenated file through main script
- Creates dataframe from WoS data
- Parses cited references data
• Counts most cited journal titles
- Extracts original article’s publication year
- Generates figures based on data extracted and basic counting/comparison
- Option for additional views of data after processing
17

18. Citation analysis: data wrangling
Python script output:
• .csv file listing journal titles ordered by citation counts
• .csv files with processed data
• Figures
• Number of articles published per year
• Year of publication of cited articles
• Age of cited articles
• Number of citations per year
• Average number of citations per article, per year
Project GitHub page:
https://goo.gl/2C2Pcy
18

19. Citation analysis findings

20. 20
Cited Reference Analysis
of Feinberg School of Medicine’s publications from 2007 - 2016
Please note: Publication data (including cited references) exported from Web of Science in 5/2017. Analysis of
cited references was completed using custom Python script. Data visualized using Microsoft Excel. All publication
types from journals included in the Pre-Clinical, Clinical and Health GIPP schema were included in the analysis.
1,741
3,194
-
500
1,000
1,500
2,000
2,500
3,000
3,500
2007 2008 2009 2010 2011 2012 2013 2014 2015 2016
NumberofDocuments
Year Document Published
Number of Documents Published Per Year
Northwestern University Feinberg School of Medicine
2007 - 2016

21. 21
Please note: Publication data (including cited references) exported from Web of Science in
5/2017. Analysis of cited references was completed using custom Python script. Data
visualized using Microsoft Excel. All publication types from journals included in the Pre-
Clinical, Clinical and Health GIPP schema were included in the analysis.
of Feinberg School of Medicine’s publications from 2007 - 2016
Cited Reference Analysis continued…
48,576
journals were cited from
2007-2016
Journal Name
Number of Cited
References from Journal
NEW ENGL J MED 21227
CIRCULATION 14224
J CLIN ONCOL 13988
JAMA-J AM MED ASSOC 12978
BLOOD 9774
LANCET 9004
P NATL ACAD SCI USA 8938
J AM COLL CARDIOL 8663
J BIOL CHEM 7287
CANCER 6489
NATURE 6462
SCIENCE 5661
CANCER RES 5532
PEDIATRICS 5424
J ALLERGY CLIN IMMUN 5029
J CLIN ENDOCR METAB 4906
ANN INTERN MED 4726
NEUROLOGY 4660
GASTROENTEROLOGY 4397
ARCH INTERN MED 4337
Top 20 Most Cited Journals
80% of citations were
to top 2.69% of
journals

22. 22
of Feinberg School of Medicine’s publications from 2007 - 2016
Cited Reference Analysis continued…
0
200
400
600
800
1000
1200
1400
1600
2007 2008 2009 2010 2011 2012 2013 2014 2015 2016
Number of Citations for Top 20 titles, Per Journal Per Year
Feinberg School of Medicine, 2007-2016
NEW ENGL J MED CIRCULATION J CLIN ONCOL JAMA-J AM MED ASSOC
BLOOD LANCET P NATL ACAD SCI USA J AM COLL CARDIOL
J BIOL CHEM CANCER NATURE SCIENCE
CANCER RES PEDIATRICS J ALLERGY CLIN IMMUN J CLIN ENDOCR METAB
ANN INTERN MED NEUROLOGY GASTROENTEROLOGY ARCH INTERN MED

23. 23
of Feinberg School of Medicine’s publications from 2007 - 2016
Cited Reference Analysis continued…
2007, 35.78
2014, 41.37
35.00
36.00
37.00
38.00
39.00
40.00
41.00
42.00
2006 2008 2010 2012 2014 2016
Average#ofCitedReferencesPerDocument
Year of Document Publication
Average Number of Cited References, Per Document Per Year
Northwestern University Feinberg School of Medicine
2007 - 2016
Please note: Publication data (including cited references) exported from Web of Science in 5/2017. Analysis of
cited references was completed using custom Python script. Data visualized using Microsoft Excel. All publication
types from journals included in the Pre-Clinical, Clinical and Health GIPP schema were included in the analysis.

24. Cited Reference Analysis continued…
of Feinberg School of Medicine’s publications from 2007 - 2016
24
Please note: Publication data (including cited references) exported from Web of Science in 5/2017. Analysis of
cited references was completed using custom Python script. Data visualized using Microsoft Excel. All publication
types from journals included in the Pre-Clinical, Clinical and Health GIPP schema were included in the analysis.
0
10000
20000
30000
40000
50000
60000
70000
80000
90000
100000
0 10 20 30 40 50 60 70 80 90 100 110 120 130 140 150 160 174 190 216 247 331
NumberofCitedReferences
Age in Years of Cited Reference
Age of Cited Reference Compared to Age of Citing Document
Northwestern University Feinberg School of Medicine
2007-2016
Number of Cited References
2 years, 91,687

25. 25
Cited Reference Analysis continued…
of Feinberg School of Medicine’s publications from 2007 - 2016
Please note: Publication data (including cited references) exported from Web of Science in 5/2017. Analysis of
cited references was completed using custom Python script. Data visualized using VOSviewer. All publication types
from journals included in the Pre-Clinical, Clinical and Health GIPP schema were included in the analysis.
Cited References Journal Co-Citation Network
Northwestern University Feinberg School of Medicine
2007 - 2016
Circles
Size indicates number of cited references
Color and proximity indicates topical similarity
Lines
Thickness indicates number of times cited
together in same reference list
Color indicates topical similarity

26. 26
Cited Reference Analysis continued…
of Feinberg School of Medicine’s Dermatology publications from 2007 - 2016
Please note: Publication data (including cited references) exported from Web of
Science in 5/2017. Analysis of cited references was completed using custom
Python script. Data visualized using Microsoft Excel. All publication types from
journals included in Dermatology research area of the Web of Science schema
were included in the analysis.
3,346
journals were cited
in Dermatology
from 2007 - 2016
Top 20 Most Cited Journals in Dermatology, 2007-2016
Journal Name
J AM ACAD DERMATOL
Number of Cited
References from Journal
1699
BRIT J DERMATOL 1134
ARCH DERMATOL 1020
DERMATOL SURG 781
J INVEST DERMATOL 687
NEW ENGL J MED 311
INT J DERMATOL 296
J ALLERGY CLIN IMMUN 288
J EUR ACAD DERMATOL 277
J DRUGS DERMATOL 263
PEDIATR DERMATOL 252
PLAST RECONSTR SURG 219
CLIN EXP DERMATOL 205
ACTA DERM-VENEREOL 199
CANCER 196
DERMATOLOGY 190
LANCET 190
AM J SURG PATHOL 190
BLOOD 184
J CUTAN PATHOL 183
80% of citations were to
top 13.87% of journals

27. 27
0
20
40
60
80
100
120
140
2007 2008 2009 2010 2011 2012 2013 2014 2015 2016
Number of Citations for Top 20 Titles, Per Journal Per Year
Dermatology, 2007-2016
J AM ACAD DERMATOL BRIT J DERMATOL ARCH DERMATOL DERMATOL SURG
J INVEST DERMATOL NEW ENGL J MED INT J DERMATOL J ALLERGY CLIN IMMUN
J EUR ACAD DERMATOL J DRUGS DERMATOL PEDIATR DERMATOL PLAST RECONSTR SURG
CLIN EXP DERMATOL ACTA DERM-VENEREOL CANCER DERMATOLOGY
LANCET AM J SURG PATHOL BLOOD J CUTAN PATHOL
Cited Reference Analysis continued…
of Feinberg School of Medicine’s Dermatology publications from 2007 - 2016

28. 28
Cited Reference Analysis
of Feinberg School of Medicine’s Dermatology publications in 2016
Please note: Publication data (including cited references) exported from Web of
Science in 5/2017. Analysis of cited references was completed using custom
Python script. Data visualized using Microsoft Excel. All publication types from
journals included in Dermatology research area of the Web of Science schema
were included in the analysis.
1,079
journals were cited
in Dermatology
in 2016
Top 20 Most Cited Journals in Dermatology, 2016
Journal Name
Number of Cited
References from Journal
J AM ACAD DERMATOL 275
BRIT J DERMATOL 160
ARCH DERMATOL 135
DERMATOL SURG 119
J INVEST DERMATOL 93
J ALLERGY CLIN IMMUN 63
JAMA DERMATOL 63
J EUR ACAD DERMATOL 59
INT J DERMATOL 55
J DRUGS DERMATOL 55
NEW ENGL J MED 43
ACTA DERM-VENEREOL 38
CLIN EXP DERMATOL 35
DERMATOLOGY 35
PEDIATR DERMATOL 34
LASER SURG MED 32
CUTIS 31
J CUTAN PATHOL 30
LANCET 27
J DERMATOL 27
80% of citations were to
top 33.73% of journals

29. 29
Cited Reference Analysis
of Feinberg School of Medicine’s Dermatology publications in 2016
Please note: Publication data (including cited references) exported from Web of Science in 4/2017. Analysis of cited
references was completed using custom Python script. Data visualized using VOSviewer. All publication types from
journals included in Dermatology research area of the Web of Science schema were included in the analysis.
Cited References Journal Co-Citation Network
Northwestern University Feinberg School of Medicine
Dermatology, 2016
Circles
Size indicates number of cited references
Color and proximity indicates topical similarity
Lines
Thickness indicates number of times cited
together in same reference list
Color indicates topical similarity

30. Comparisons with COUNTER
usage data

31. COUNTER Methodology: Data Gathering
• JR1 Reports retrieved manually from publishers and shared library dashboard
- Reports retrieved for both HSL subscriptions and Main Library package deals
• Reports were merged into single Excel file with over 30,000 lines (OCFTRTA)
- Note: Process would have been much easier with COUNTER client
• Titles from cited reference reports then manually matched to COUNTER stats
- Issues with journal abbreviations from WoS, would need to overcome
before automating and/or looking at data in aggregate
- Titles with multiple providers were accounted for, collated and totaled,
although this was less of an issue than anticipated
31

32. COUNTER Methodology: Data Gathering
32

33. COUNTER Methodology: Data Comparisons
• Top 30 cited journals for all of NU in 2016 vs COUNTER usage
• Top 50 cited medical* journals in 2016 vs COUNTER usage
• Top 50 cited dermatology journals in 2016 vs COUNTER usage
*clinical, pre-clinical, and health
33

34. Top 30 Journals Cited by NU vs COUNTER, 2016
34
0
10000
20000
30000
40000
50000
60000
70000
80000
90000
0
500
1000
1500
2000
2500
3000
3500
4000
4500
5000
FulltextRetrievals
CitedReferences
Cited reference count COUNTER JR1 Total

35. Top 30 Journals Cited by NU vs COUNTER, 2016
35
• Pearson correlation coefficient, r = 0.46
• Spearman’s rho, ρ = 0.25
• Why so low? What’s going on here?
• Citation and usage patterns vary widely across disciplines

36. Top 30 20 Journals Cited by NU vs COUNTER, 2016
36
0
10000
20000
30000
40000
50000
60000
70000
80000
90000
0
500
1000
1500
2000
2500
3000
3500
4000
4500
FullTextRetrievals
CitedReferences
(Excludes Physics titles)
Cited reference count Aggregate COUNTER JR1
Spearman
ρ = 0.62
Pearson
r = 0.81

37. Top 50 Cited Medical Journals vs COUNTER, 2016
37
0
10000
20000
30000
40000
50000
60000
70000
80000
90000
0
500
1000
1500
2000
2500
3000
NEWENGLJMED
CIRCULATION
JAMA-JAMMEDASSOC
JCLINONCOL
LANCET
JAMCOLLCARDIOL
PNATLACADSCIUSA
BLOOD
PEDIATRICS
NATURE
JALLERGYCLINIMMUN
JBIOLCHEM
CANCERRES
SCIENCE
CANCER
JAMAInternMed
GASTROENTEROLOGY
NEUROLOGY
OBSTETGYNECOL
SPINE
ANNINTERNMED
STROKE
AMJOBSTETGYNECOL
CLINCANCERRES
DIABETESCARE
AMJTRANSPLANT
ARCHPHYSMEDREHAB
RADIOLOGY
JCLININVEST*
AMJRESPCRITCARE
CLININFECTDIS
CELL
AMJEPIDEMIOL
COCHRANEDBSYSTREV
AMJGASTROENTEROL
JAMACADDERMATOL
ANNSURG
ANNTHORACSURG
EURHEARTJ
AMJCARDIOL
NATGENET
CHEST
JNEUROSCI
LARYNGOSCOPE
AMJPUBLICHEALTH
BRITMEDJ
JUROLOGY
JIMMUNOL
FullTextretrievals
CitedReferences
Cited reference count Aggregate COUNTER JR1

38. Top 50 Cited Medical Journals vs COUNTER, 2016
• Obvious outliers with inflated COUNTER stats for multi-disciplinary titles
(Nature, Science, Cell, etc.)
• No “low” use titles in top 50
- Lowest journal still had 1906 full text retrievals (excl. Cochrane)
• No gaps (titles without current access) in top 50
• Slight statistical correlation, lower than other studies
Brief Analysis
38
Spearman
ρ = 0.54
Pearson
r = 0.52

39. Top 50 Cited Dermatology Journals vs COUNTER, 2016
39
0
10000
20000
30000
40000
50000
60000
70000
80000
90000
0
50
100
150
200
250
300
FullTextRetrievals
CitedReferences
(excludes OA, titles with missing data, and titles w/o current access)
Cited reference count Aggregate COUNTER JR1

40. Top 50 Cited Dermatology Journals vs COUNTER, 2016
• Obvious outliers with large COUNTER stats for multi-disciplinary titles (NEJM,
Lancet, JAMA, etc.)
• 3 “low” use titles with less than 100 full text retrievals make an appearance:
- Photodermatol Photo (74), Acta Derm-Venereol (57), & J Cutan Med Surg (25)
• 5 gap titles (journals w/o current full text) make an appearance:
- Dermatology, J Dermatol Treat, Eur J Dermatol, Am J Clin Dermatol, & Dermatologica
• No overall statistical correlation, which is to be expected
Brief Analysis
40
Spearman
ρ = 0.13
Pearson
r = -.08

41. Just for “Fun”: Cost Per Use vs Cost Per Cited Reference
WhereCPUisgreater than$5
41
$0.00 $50.00 $100.00 $150.00 $200.00 $250.00
BMJ Quality & Safety
Journal of Clinical Pathology
Allergy and Asthma Proceedings
British Journal of Sports Medicine
British Journal of Ophthalmology
Journal of Medical Genetics
Thorax
Antioxidants & Redox Signaling
Teaching and Learning in Medicine
Diabetes
Gut
Journal of Neurology Neurosurgery & Psychiatry
American Journal of Rhinology & Allergy
Archives of Disease in Childhood
Heart
Journal of Neurotrauma
AIDS Research and Human Retroviruses
Diabetes Care
Cost Per Cited Reference Cost Per Use

42. Collection Development
Applications

43. Collection Development Applications
• Prevent undervaluing of a title when other stats are questionable
- Cited reference count can easily be consulted before making decision
- Especially useful in instances of low usage, or high CPU based on reporting
issues
• Also useful for evaluating OA titles, or titles without COUNTER
Before making a painful cut, all possible data points should be consulted and
documented in order to back up or defend the decision.
Contextualizing Usage Statistics withCited Reference Counts
Full Title COUNTER JR1 Total Med Cited Reference Count
The journal of clinical endocrinology & metabolism 0 556
43

44. Collection Development Applications
• Use cited reference counts to identify and
rank high impact titles outside of collection
- Check against other metrics like
turnaways, ILL’s etc., to inform CD
• With more automation, this could be done
in aggregate
• Another data point to use in evaluating
“wish list” or bubble titles
Identifying Gaps
44

45. Collection Development Applications
• Cited reference counts are perhaps more compelling than ILL requests
- Illustrate need for highly requested titles, or
- Demonstrate low research impact of highly requested titles to defend not
subscribing
Top 4 Most Requested Titles Through ILL
Contextualizing orSupplementing ILLData withCited Reference Counts
Title ILL Requests Cited References
Brain Inj
(Brain Injury) 21 54
Disabil Rehabil
(Disability and rehabilitation) 20 11
Curr Pharm Des
(Current pharmaceutical design) 17 20
Psychol Med
(Psychological medicine) 16 83
45

46. Collection Development Applications
GapAnalysis: Users arefinding access outside oftraditional library services
46
0
5
10
15
20
25
30
35
40
Dermatology J Dermatol Treat Eur J Dermatol Am J Clin Dermatol Dermatlogica
Highly Cited Dermatology titles without current subscriptions
Cited References Cited references outside of access entitlements ILL Requests

47. Collection Development Applications
• Librarians often need to evaluate usage and impact for a specific context
• Most standard metrics (web analytics, link resolver stats, COUNTER, JIF) are at
much higher levels
- Impact factor has limited utility for school or library specific evaluation
- No significant correlation found between impact factor and # cited
references for FSM publications
• Citation analysis by school or research area, layered on top of broader usage
statistics, can provide a more holistic and contextualized understanding of
usage and impact within specific environments
• Outliers from any metric can be checked against other data points and
evaluated with more context
Contextualizing usagebyschool ordiscipline
47

48. Final Thoughts
• Outside of some commercial services, no automated solution for scaling up
• Cited reference data on it’s own is not that useful for collection development
- However, when used in conjunction with other metrics, meaningful
information surfaces quite easily
• Citation figures also useful for troubleshooting, sanity checks or substitutes,
when other stats are unavailable
• With more automation, might be possible to use citation data for broader
collection development and assessment activities without making as many
comparisons, we’re almost there but not quite yet
• Overall it’s a worthwhile tool to have for collection development, yay!
Collection Development Implications
48

49. References
• Belter, Christopher W., and Neal K. Kaske. "Using Bibliometrics to Demonstrate the Value of Library Journal Collections." College and Research Libraries 77, no. 4 (2016): 410-22.
• Blecic, D. D. "Measurements of Journal Use: An Analysis of the Correlations between Three Methods." Bulletin of the Medical Library Association 87, no. 1 (1999): 20-25.
• Bollen, Johan, and Herbert Van de Sompel. "Usage impact factor: the effects of sample characteristics on usage‐based impact metrics." Journal of the American Society for Information Science and
technology 59, no. 1 (2008): 136-149.
• Bordeaux, Abigail, Alfred B. Kraemer, and Paula Sullenger. "Making the most of your usage statistics." The Serials Librarian 48, no. 3-4 (2005): 295-299.
• Cusker, Jeremy. "Using Isi Web of Science to Compare Top-Ranked Journals to the Citation Habits of a "Real World" Academic Department." Issues in Science and Technology Librarianship, Summer
(2012).
• Davis, Philip M. "Where to Spend Our E-Journal Money? Defining a University Library's Core Collection through Citation Analysis." portal: Libraries and the Academy 2, no. 1 (2002): 155-66.
• De Groote, Sandra L., Deborah D. Blecic, and Kristin Martin. "Measures of Health Sciences Journal Use: A Comparison of Vendor, Link-Resolver, and Local Citation Statistics." Journal of the Medical
Library Association : JMLA 101, no. 2 (2013): 110-19.
• Gao, Wenli. "Beyond Journal Impact and Usage Statistics: Using Citation Analysis for Collection Development." The Serials Librarian 70, no. 1/4 (2016): 121-27.
• Gross, P.L.K., and E.M. Gross. "College Libraries and Chemical Education." Science N.S. 66, no. 1713 (1927): 385-89.
• Hoffmann, Kristin, and Lise Doucette. "A Review of Citation Analysis Methodologies for Collection Management." 2012 73, no. 4 (2012): 15.
• Ke, Irene, and Jackie Bronicki. "Using Scopus to Study Researchers’ Citing Behavior for Local Collection Decisions: A Focus on Psychology." Journal of Library Administration 55, no. 3 (2015): 165-78.
• Kraemer, Alfred. "Ensuring consistent usage statistics, part 2: working with use data for electronic journals." The Serials Librarian 50, no. 1-2 (2006): 163-172.
• Luther, Judy. "White paper on electronic journal usage statistics." The Serials Librarian 41, no. 2 (2002): 119-148.
• Noonan, Christine F., and Melissa K. McBurney. "Application of electronic serial usage statistics in a national laboratory." In Usage statistics of e-serials, ed. David C. Fowler, 151-60. Binghamton, NY:
Haworth Information Press, 2007.
• Ralston, Rick, Carole Gall, and Frances A. Brahmi. "Do local citation patterns support use of the impact factor for collection development?." Journal of the Medical Library Association: JMLA 96, no. 4
(2008): 374.
• Rathemacher, Andrée J. “E-Journal Usage Statistics in Collection Management Decisions: A Literature Review.” In Library Data: Empowering Practice and Persuasion, ed. Darby Orcutt, 71-89. Santa
Barbara, Calif.: Libraries Unlimited, 2010.
• Tsay, M. Y. "The Relationship between Journal Use in a Medical Library and Citation Use." Bulletin of the Medical Library Association 86, no. 1 (1998): 31-39.
• Wical, Stephanie H., and R. Todd Vandenbark. "Notes on Operations: Combining Citation Studies and Usage Statistics to Build a Stronger Collection." Library Resources & Technical Services 59, no. 1
(2015): 33-42.
• Welker, Josh. "Counting on COUNTER: The Current State of E-Resource Usage Data in Libraries." Computers in Libraries 32, no. 9 (2012): 6-11.
• Wilson, Concepción S., and Carol Tenopir. "Local Citation Analysis, Publishing and Reading Patterns: Using Multiple Methods to Evaluate Faculty Use of an Academic Library's Research Collection."
Journal of the American Society for Information Science and Technology 59, no. 9 (2008): 1393-408.

50. Questions?
joelen.pastva@northwestern.edu
@jolophon
j-shank@northwestern.edu
@ShankLib

51. Thank you!