Much of the demand today to add new journals comes from new multidisciplinary courses and programs. The usefulness of citation analysis as a tool for developing such collections is shown by the importance often placed on published impact factors. But not all libraries can afford access to the Web of Science’s Journal Citation Reports or Scopus, and many researchers have described weaknesses in those sources’ data. Google Scholar is helpful but it, too, has weaknesses. Critiques of these sources of impact factors will be briefly reviewed and a bibliography provided. But the most important reason to do an independent citation analysis is that impact factors reflect citations to journals from all disciplines. To determine the most-cited journals within a multidisciplinary field one should analyze only citations published within the topic area. Independently gathering citation data used to be a tedious process. But the inclusion of works cited in databases such as PsycINFO and ScienceDirect has made gathering data considerably easier, and citation management programs like RefWorks and Zotero speed the process of compiling and organizing citations. How to use these tools to relatively quickly assemble a sample of works cited in a specific topic area will be demonstrated. I’ll also discuss various methods of gathering a valid sample and issues to consider when determining sample size. I’ll conclude with a discussion of the strengths and weaknesses of using do-it-yourself citation analysis for collection development or supporting new program development and offer suggestions for anyone interested in doing a project for the purpose of publishing the results. Presenter: Steve Black, College of Saint Rose

1. Practical Applications
of Do-it-Yourself
Citation Analysis
Steve Black
The College of Saint Rose
June 10, 2012
Accompanying handout: http://www.slideshare.net/NASIG/blackpractical-applications-of-citation-analysis

2. Citation analysis is
the study of
patterns and
frequencies of
citations

3. Why
• Objective, quantitative measurement of
impact (relative frequency of citation) of
– journals
– authors
– institutions
• Citation analysis can help answer
“What are the best journals for ____?”

4. Critiques of relying on impact factors
• data errors
• Matthew effect
• gaming the system
• impact vs. quality vs. importance
• global vs. local impact
(See bibliography)

5. Never forget,
& remind others when needed,
impact factor is but one of many
considerations
for collection development,
paper submissions,
tenure & promotion.

6. Do-It-Yourself Citation Analysis
• tailored to specific needs not
covered elsewhere
• can be quick-and-
dirty, exhaustive, or in between
• saves expense of Scopus or Web
of Knowledge

7. Basic DIY Approach
• Select a target population
– journals in a sub-discipline (e.g. forensic psychology)
– researchers at an institution, school, department, or
even one faculty member (e.g. all the papers published in last 10
years by faculty of Dept. X of Big U.)
– subjects or keywords (e.g. phrases on hot research topics)
• Select a sample that represents the target
population
• Compile works cited in the chosen sample
• Sort and count the works cited

8. Google Scholar & Zotero
• Google Scholar is good for identifying articles
on a topic, BUT
– no authority control (which S Black?)
– works cited displayed in many formats, if available
at all

9. Freeware can help compile . . .

10. . . .but Zotero’s not designed for DIY
citation analysis
• emphasis on full text searching
• no authority control
• inconsistent data structure
• extracting and downloading the right data to a
spreadsheet not straightforward

11. http://kimmonsdesign.com/node/24

12. Case study in DIY citation analysis:
forensic psychology
Published as
“Frequently Cited Journals in Forensic Psychology,”
Psychological Reports, v.110, no.1, (2012), p.276-
282.

13. Goal: create ranked list of most-cited
journals in forensic psychology
• General Method
– identify leading journals specific to the field to use
as sources of citations
– gather citations from recent volumes of those
journals
– compile cites and count times cited
– rank cited journals by frequency of citation

14. Source Journals Sample
• For published article:
– 2007, 2008, and 2009 volumes of 6 widely held
journals specific to forensic psychology
– n=16,518 works cited
• For this presentation:
– 2011 volume of Law and Human Behavior
– n=1378 works cited

15. Tools used
• WorldCat (to identify most widely held journals)
• PsycINFO via EBSCOhost (to identify works cited)
• RefWorks (to organize works cited)
• Excel (to count and rank cited journals)

16. Retrieve records for articles in
source journals, one issue at a time

17. Add cited references to folder

18. Export references . . .

19. . . . to RefWorks

20. RefWorks creates a temporary folder
of the downloads

21. Create a folder for each issue’s works cited
(keep discrete units to allow statistical analysis of reliability)

22. Move the downloaded cites to the
appropriate folder

23. Export references in tab delimited
format

24. Save as a text document

25. Open the text file in Excel . . .

26. . . .by going through 3 steps to convert
from text

27. Work with cited journal titles
shown as “Periodical Full”
Note that
some
tagged
reference
types are
imprecise,
don’t
count the
“slop.”

28. Sort the title column alphabetically

29. For accurate counts, review titles and
remove discrepancies

30. Count citation frequency
(choose a cutoff point equal to about 0.5% of sample)

31. Record frequency counts

32. Sort by largest to smallest
to create ranked list

33. Bradford distribution 1:n:n2:n3 roughly
holds in this case as 1:3:9:27...
Law and Human Behavior 154
Criminal Justice and Behavior 45
Journal of Applied Psychology 45
Behavioral Sciences and the Law 40 130
Journal of Personality and Social Psychology 35
Psychological Assessment 32
Applied Cognitive Psychology 29
American Psychologist 26
Psychiatric Services 26
Journal of Abnormal Psychology 21
Psychology, Public Policy, and Law 21
Personality and Individual Differences 15
Psychological Bulletin 15 220
Assessment 14
Psychological Review 14
Journal of Applied Social Psychology 13
Journal of Experimental Psychology: Applied 13
Psychological Science 13
Psychology, Crime and Law 12
International journal of law and psychiatry 11
Legal and Criminological Psychology 11
Criminology 10
Sex Roles 9
Aggressive Behavior 8
American Journal of Public Health 8
International Journal of Forensic Mental Health 8
International Journal of Offender Therapy and Comparative Criminology 8
Journal of Criminal Law and Criminology 7
Journal of personality assessment 7
Journal of the American Academy of Psychiatry and the Law 7
Aggression and Violent Behavior 6
American Journal of Psychiatry 6
Archives of General Psychiatry 6
Basic and Applied Social Psychology 6
British Journal of Psychiatry 6
Journal of personality disorders 6
Professional Psychology: Research and Practice 6
Virginia Law Review 6
Annual Review of Psychology 5
British Journal of Criminology 5 231

34. ...even though LHB cites a mix of
forensic and general psych journals
Law and Human Behavior 154
Criminal Justice and Behavior 45
Journal of Applied Psychology 45
Behavioral Sciences and the Law 40 130
Journal of Personality and Social Psychology 35
Psychological Assessment 32
Applied Cognitive Psychology 29
American Psychologist 26
Psychiatric Services 26
Journal of Abnormal Psychology 21
Psychology, Public Policy, and Law 21
Personality and Individual Differences 15
Psychological Bulletin 15 220

35. How should we take this neatly ranked
list of forensic psych journals?

36. It depends on the reliability of our
rankings
• Measure reliability 2 ways:
– Macro: variability in ranked lists
– Micro: variability in individual titles’ times cited

37. Macro: How well do ranked lists correlate?
Feb 2011 April 2011 June 2011
Law and human behavior 28 Law and human behavior 30 Law and human behavior 17
Journal of Applied Psychology 13 Psychiatric Services 14 Criminal Justice and Behavior 14
Psychological assessment 11 Behavioral Sciences and the Law 12 Journal of personality and social psychology 11
Applied Cognitive Psychology 7 Journal of Applied Psychology 9 Personality and Individual Differences 8
Journal of abnormal psychology 6 Psychology, Public Policy, and Law 9 Behavioral Sciences and the Law 6
Assessment 5 Criminal Justice and Behavior 7 Criminal Justice Policy Review 6
Virginia Law Review 5 Journal of personality and social psychology 6 International Journal of Offender Therapy and Comparative Criminology 6
Criminal Justice and Behavior 4 Sex Roles 6 Assessment 5
Criminology 4 American Psychologist 5 Journal of consulting and clinical psychology 5
Wisconsin Law Review 4 International Journal of Forensic Mental Health 5 Journal of Applied Social Psychology 4
?
Micro: How variable are frequencies of citation
to LHB, J App Psych, etc.?

38. Macro method:
Test consistency (reliability) with
Spearman’s rho rank correlation
coefficient

39. To calculate rs, first assign ranks
• Choose a cutoff so number of ranks in each
group is equal.
• Handle ties by averaging, e.g. tied for 2 & 3
are each ranked 2.5.
• Assign bottom rank to titles with no match,
e.g. if n=22, unmatched titles are ranked 22.

40. Ranks of journals cited in 2011 volume of
Law and Human Behavior
(blanks will be filled with rank = 22)
Title cited in LHB 2011 frequency all
Allrank FebRank AprilRank JunRank AugRank OctRank DecRank
Law and Human Behavior 154 1 1 1 1 1 1 1
Criminal Justice and Behavior 45 2.5 9.5 6 2 2 9.5
Journal of Applied Psychology 45 2.5 2 4.5 3 4.5 4
Behavioral Sciences and the Law 40 4 3 6 4 2 16
Journal of Personality and Social Psychology 35 5 16.5 7.5 3 8.5 9.5 11.5
Psychological Assessment 32 6 3 13 8.5 11 4
Applied Cognitive Psychology 29 7 4 17.5 13 12 13.5 2
American Psychologist 26 8.5 16.5 9.5 8.5 9.5 8.5
Psychiatric Services 26 8.5 2 16.5 6
Journal of Abnormal Psychology 21 10.5 5 13 9.5 11.5
Psychology, Public Policy, and Law 21 10.5 4.5 4.5 16
Journal of Consulting and Clinical Psychology 18 12 8.5 5.5 21
Personality and Individual Differences 15 13.5 4 16
Psychological Bulletin 15 13.5 11.5 13 8.5
Assessment 14 15.5 6.5 8.5 21
Psychological Review 14 15.5 3 21
Journal of Applied Social Psychology 13 18 16.5 17.5 10
Journal of Experimental Psychology: Applied 13 18 16.5 8.5
Psychological Science 13 18 16.5 11.5
Psychology, Crime and Law 12 20 4
International journal of law and psychiatry 11 21.5 16.5 13.5
Legal and Criminological Psychology 11 21.5 17.5 13 16

41. Rank correlations among cites in single
issues of Law and Human Behavior are
weak and inconsistent
2011 Feb Apr June Aug Oct Dec
2011 1
Feb 0.24 1
Apr 0.47 -0.34 1
June 0.26 -0.05 -0.2 1
Aug 0.63 0.05 0.4 0.15 1
Oct 0.59 -0.05 0.46 -0.47 0.11 1
Dec 0.23 0.25 -0.66 -0.42 -0.03 -0.02 1

42. How does LHB (2011) correlate with
the published list?
"Frequently Cited Journals in Forensic Psychology," Psychological Reports v.110, no. 1, Feb. 2012: 276–282
Title Percent of citations times cited (n=16,518) rank
Law and Human Behavior 5.71% 1243 1
Behavioral Sciences and the Law 2.46% 536 2
Criminal Justice and Behavior 1.79% 390 3
British Journal of Psychiatry 1.63% 354 4
Journal of Applied Psychology 1.62% 352 5
Journal of Personality and Social Psychology 1.50% 326 6
American Journal of Psychiatry 1.34% 291 7
Journal of Forensic Psychiatry and Psychology 1.30% 283 8
Applied Cognitive Psychology 1.20% 261 9
Psychology, Public Policy, and Law 1.15% 250 10
Psychology, Crime, and Law 1.08% 236 11
Psychological Assessment 1.03% 224 12
Personality and Individual Differences 1.01% 221 13
American Psychologist 0.99% 215 14
Legal and Criminological Psychology 0.96% 210 15
Journal of Consulting and Clinical Psychology 0.94% 204 16
Journal of the American Academy of Psychiatry and the
Law 0.92% 201 17.5
Sexual Abuse: A Journal of Research and Treatment 0.92% 201 17.5
Archives of General Psychiatry 0.91% 198 19
Psychiatric Services 0.85% 185 20
Journal of Applied Social Psychology 0.84% 183 21
Psychological Bulletin 0.83% 180 22
Journal of Abnormal Psychology 0.70% 153 23

43. Spearman’s rho rank correlation for one volume
of LHB vs. the larger sample is rs=.63
LHB Psych
2011 Reports D D2
title rank rank
Law and Human Behavior 1 1 0 0
Criminal Justice and Behavior 2.5 3 -0.5 0.25
Journal of Applied Psychology 2.5 5 -2.5 6.25
Behavioral Sciences and the Law 4 2 2 4
Journal of Personality and Social Psychology 5 6 -1 1
Psychological Assessment 6 12 -6 36
Applied Cognitive Psychology 7 9 -2 4
American Psychologist 8.5 14 -5.5 30.25
rs =1-(6*684)/22*(222-1)
Psychiatric Services 8.5 20 -11.5 132.25 = 1 – (4104/10626)
Journal of Abnormal Psychology 10.5 22 -11.5 132.25
Psychology, Public Policy, and Law 10.5 10 0.5 0.25 = 1- .37
Journal of Consulting and Clinical
Psychology
12 16 -4 16 rs = .63
Personality and Individual Differences 13.5 13 0.5 0.25
Psychological Bulletin 13.5 22 -8.5 72.25
Assessment 15.5 22 -6.5 42.25
Psychological Review 15.5 22 -6.5 42.25
Journal of Applied Social Psychology 18 21 -3 9
Journal of Experimental Psychology: Applied 18 22 -4 16
Psychological Science 18 22 -4 16
Psychology, Crime and Law 20 11 9 81
International journal of law and psychiatry 21.5 22 -0.5 0.25
Legal and Criminological Psychology 21.5 15 6.5 42.25
sum of D squared 684

44. Sample size and expected reliability
Sample sample size expected correlation
(works cited) with population
one issue of journal n=250 very low
one volume of journal n=1500 moderate
several volumes of 3-5 n>10,000 strong
journals

45. Micro Method
• Measure coefficients of variation for individual
titles’ frequency of times cited
Coefficient of variation =
standard deviation ÷ mean

46. Examples for LHB 2011
Times Cited in 2011 Law and Human Behavior
Feb Apr June Aug Oct Dec σX mean Coeff. Variation
Law and Human Behavior 28 30 17 25 27 27 4.55 25.67 18%
Journal of Applied Psychology 13 9 0 8 8 7 4.23 7.50 56%
Psychology, Public Policy, and Law 1 9 0 0 8 3 4.04 3.50 115%
Greater Coefficient of Variation = lower reliability
• Top journal pretty stable
• Titles lower down the ranked list are more variable

47. Assuming the sampled journals
validly represent the topic,
a sample of
• n<1,000 works cited can indicate the top
journal (or tell you if there is none)
• n>1,000 can generate a rough indication of
leading journals
• n>10,000 can create a useful ranked list

48. • Even with n>10,000, different samples will
yield different ranks, especially further down
the list,
• So a journal’s rank must ALWAYS be taken as
an approximation of its “true” ranking.

49. Strengths of DIY Citation Analysis
• Can target a precise area
• Yields data not otherwise available
• Objective, quantitative collection development
tool
• Results may be of fairly broad interest
(publishable!)
• If project is well organized, data can be gathered
& sorted by research assistant(s)
• Interdisciplinary--can team with faculty outside
the library

50. Weaknesses of DIY Citation Analysis
• Very time consuming to do well
• A really robust ranking requires n>20,000, and
even that’s not definitive
• Works cited may not be readily available (few
databases make it as easy as PsycINFO)
• It’s recreating the (very expensive) wheel –
better to use Scopus or Web of Knowledge if
you can

51. Tips for publication
• Don’t be too parochial or narrow—editors must think their readers
will be interested.
• BUT choose something not already in Journal Citation Reports
• Be very thorough with your lit review before gathering data (search
disciplinary databases as well as the library literature)
• Gather citations and test sample before solidifying your method
• Carefully select and describe the sample
– representative
– clearly defined
– robust
– good face validity
• Group data and run statistics by journal volume
• Consider submitting to a journal in the topic area

52. Tips for Collection Development
• Use citation analysis to assess impact of
specialized journals
• Don’t be surprised to find surprises
• Judge new titles by other criteria
• Use ranked lists for assessment evidence

53. Practical Applications
• What’s the top journal on this topic?
– Compile cites from 10-20 articles on the topic to identify it
(or find there’s not one)
• What does the library need to support a proposed new
major?
– Compile cites from 1-2 volumes of 2-4 journals to flag titles
not already available
• What’s that herd of cats in Dept. X REALLY use?
– Analyze publications & dissertations from last decade
• OMG I’m up for tenure soon!
– Identify a topic and do a thorough analysis of 2-3 volumes
of 4-6 journals; include reliability tests

54. Steve Black
blacks@strose.edu
(518) 458-5494