1. REVIEWING THE
LITERATURE
Ashley Juhl, M.S.P.H
Indira B. Gujral, M.S., Ph.D.
Epidemiology, Planning, and Evaluation
Branch

2. Agenda
What is Evidence-Based Public Health? (Indira Gujral)
 Brief overview of why it matters
Conducting the Search (Ashley Juhl)
 Identifying the search question
 Searching the databases
 Utilizing the CDPHE Digital Library
Critiquing the Literature (Indira Gujral)
 Abstract pertinent information from each document
 Organize documents for review
 Summarize the scientific literature review
 Rating evidence

3. Objective
What will I learn from this training?
 How to search for and obtain articles using
the CDPHE Digital Library
 How to critique and summarize articles
 How to use tools to apply these new skills
 Critique Literature
 Literature Library
 Levels of Evidence to Rate Strategy

4. Using Scientific Literature
Step 1
Step 7 Step 2
Step 6
Step 3
Step 5
Step 4
Materials Located: I:E P EPortfolio Project

5. WHAT IS EVIDENCE-
BASED PUBLIC HEALTH?

6. What is “Evidence?”
Objective
 Systematic reviews (Meta-analysis)
 Literature from one or more journal articles
 Public health surveillance data
 Program evaluations
 Qualitative data
 Community members
 Other stakeholders
 Media/marketing data
 Word of mouth
 Personal experience
Slide modified from presentation by Ross Brownson, PhD (2011) Subjective

7. What is “Evidence-based Public Health”?
 Institute of Medicine (1988) – public
health is driven by ―crises, hot issues,
and concerns of organized interest
groups‖
 Barriers: lack of political will, deficits in research,
resources, leadership, competencies, and deficits in
relevant and timely research
 Goal – to improve population health
through widespread adoption of
evidence-based strategies

8. Why use Evidence-Based Strategies?
 Higher likelihood of a successful
program
 Reduces costs
 Impacts the population
 Contribute to the body of knowledge

9. Types of Public Health Research Studies
Characteristic Descriptive Research Intervention Research Contextual Conditions
Typical Data/ Size and strength of Relative effectiveness of Information on the adaptation and
Relationship preventable risk—disease public health intervention translation of an effective
relationship (measures of intervention
burden, etiologic research)
Common setting Clinic or controlled community Socially intact groups or Socially intact groups or
setting community-wide community-wide
Example Smoking causes lung cancer Price increases with a Understanding the political
targeted media campaign challenges of price increases
reduce smoking rates
Quantity More (study = 78.5%) Less (study = 14.9%) Even Less
Action Something should be done. This particular intervention How an intervention should be
should be implemented implemented
Slide modified from presentation by Ross Brownson, PhD (2011)

10. CONDUCTING THE SEARCH

11. Objectives
 Learn how to formulate an answerable question for
the literature review (PICO)
 Identify search databases
 Gain knowledge of resources for grey literature
 Learn about the Digital Library project and how to
access full-text journal articles

12. SEARCH QUESTION

13. PICO
 The acronym PICO can be used to describe this
―answerable‖ question, so that
 ―P‖= patient problem or population
 ―I‖ = intervention
 ―C‖ = comparison intervention
 ―O‖ = outcome

14. PICO Examples
Problem/ Intervention Comparison Outcome
Population
Unintended School-based sex Abstinence Avoidance of
pregnancy among education education unintended
teens pregnancy
Education in family Consistent
planning clinic contraceptive use
Education in school- Delayed initiation
based health clinics of sexual
intercourse
Community-based All three outcomes
programs

15. Sample Search Questions
 For teenagers, will school-based sex education
programs as compared to school-based abstinence
programs decrease unintended pregnancy rates?
 Among teenagers, will education in family planning
clinics increase consistent contraceptive use?
 Among teenagers, will education in school-based health
clinics delay the initiation of sexual intercourse?
 Among teenagers, will education in community-based
health clinics decrease unintended pregnancy rates,
increase consistent contraceptive use, and delay the
initiation of sexual intercourse?

16. SEARCHING THE DATABASES

17. What is “Evidence?”
Objective
 Systematic reviews (Meta-analysis)
 Literature from one or more journal
articles
 Public health surveillance data
 Program evaluations
 Qualitative data
 Community members
 Other stakeholders
 Media/marketing data
 Word of mouth
 Personal experience
Subjective
Slide modified from presentation by Ross Brownson, PhD (2011)

18. Systematic Reviews
 PubMed
 http://www.ncbi.nlm.nih.gov/pubmed?holding=codphlib
 searches for systematic reviews in biomedical literature
 The Cochrane Library
 http://www.thecochranelibrary.com/view/0/index.ht
 systematic reviews of primary research in human health care and health
policy
 free full text from CDPHE computer (through 4/2012)
 The Campbell Library
 http://www.campbellcollaboration.org/library.php
 prepare, maintain and disseminate systematic reviews in education, crime
and justice, and social welfare
 free full text

19. What is “Evidence?”
Objective
 Systematic reviews (Meta-analysis)
 Literature from one or more journal
articles
 Public health surveillance data
 Program evaluations
 Qualitative data
 Community members
 Other stakeholders
 Media/marketing data
 Word of mouth
 Personal experience
Subjective
Slide modified from presentation by Ross Brownson, PhD (2011)

20. PubMed
 Search citations for biomedical literature from MEDLINE, life
science journals, and online books
 Clinical Queries will locate clinical studies, systematic
reviews, and medical genetics citations
 MeSH Database will help pinpoint concepts
 ClinicalTrials.gov will locate clinical trial information and
results
 http://www.ncbi.nlm.nih.gov/pubmed?holding=codphlib

21. My NCBI
 Save searches and records
 Automatic email alerts
 Filter search results by subject
 Free
 http://www.ncbi.nlm.nih.gov/sites/myncbi/

22. PubMed Search
 For teenagers, will school-based sex education
programs as compared to school-based abstinence
programs decrease unintended pregnancy rates?
 teenagers
 school-based sex education
 school-based abstinence education
 unintended pregnancy

23. What is “Evidence?”
Objective
 Systematic reviews (Meta-analysis)
 Literature from one or more journal
articles
 Public health surveillance data
 Program evaluations
 Qualitative data
 Community members
 Other stakeholders
 Media/marketing data
 Word of mouth
 Personal experience
Subjective
Slide modified from presentation by Ross Brownson, PhD (2011)

24. Types of Grey Literature
 conference papers/posters/abstracts
 official documents and white papers
 reports and proceedings
 theses and dissertations
 official documents
 informal communications
 research in progress reports
 registered clinical trials

25. Grey Literature
 NOT controlled by commercial publishers
 Often includes information vital to research –
especially public health
 NOT indexed in major databases, including
PubMed
 More current than published material

26. Grey Literature Sources
 Specialized search engines
 New York Academy of Medicine Grey Literature Report
 Science.gov
 Scirus
 Google Scholar
 Promising Practices
 Institutional repositories, scientists websites

27. CDPHE DIGITAL LIBRARY

28. Problem
 State and local public health lack sufficient access
to databases, full-text journal articles, and books
 Individual/program subscriptions
 University access through students/faculty
 Impedes ability to find best available evidence to
improve health

29. Digital Library Project
 Purpose: To better understand the usefulness of
providing public health departments with access to
selected licensed e-resources (databases, books,
journal articles)
 Funder: National Network of Libraries of Medicine,
New England Region
 CDPHE Lead: Dr. Lisa Miller (Director, DCEED)
 Funding through April 2012
 CDPHE Digital Library launched in January 2011

30. Project Objectives
 Promote selected e-journals and two databases
that contain evidence-based public health resources
(in addition to PubMed):
 Stat!Ref– compilation of full-text E-books
 Global Health—contains public health information
back to 1912
 License EndNote for multiple users
 Provide subsidized Interlibrary Loan to those who
work in the selected public health departments

31. Where is the Digital Library?

32. Digital Library Contents
 Colorado specific PubMed link
 Access to Cochrane Library, STAT!Ref and Global
Health
 Links to 55 e-journals with free full-text
 Free journal article delivery service
 Poudre Valley Health System Medical Library
 Other databases, academic, and government
websites

33. Access to Journal Article Delivery Service
 Access to LoansomeDoc will be provided by Poudre
Valley Health System
 Library of first resort
 Registration to LoansomeDoc to monitor usage
https://docline.gov/loansome/login.cfm
 Free service, easy to use
 Utilize University of Massachusetts Lamar Soutter
Library
 Library of last resort

34. Requesting Journal Articles for Delivery
 PubMed
 Select send to – order - order articles
 Sign in to LoansomeDoc
 Request article even if not in holdings (ignore cost box)
 Other sources
 Email Jerry Carlson directly gmc@pvhs.org
 Include all article information
 Articles will be e-mailed to requestor directly at no
charge

35. Requesting Journal Articles for Delivery
Continued
 Before submitting request, make sure the journal is
not listed in the ―E-Journals‖ section of the Digital
Library page
 Immediate retrieval of article (access to free full-text
from a CDPHE computer)

36. Digital Library Questions
 Please Contact:
 Karen Dahlen (external project coordinator)
 kdahlen@comcast.net
 Ashley Juhl (internal CDPHE contact)
 ashley.juhl@state.co.us
Happy Searching!

37. QUESTIONS?
Stretching session

38. HOW TO CRITIQUE AN
ARTICLE

39. Abstracting Information
Critique an Article
 Critique: ―an act of criticizing‖
Merriam Webster
 Key Elements:
 Balanced
 Includes strengths and weaknesses
 Constructive
 Providing suggestions for how the study might be
improved

40. What will I be Critiquing?
Objective
 Systematic reviews (Meta-analysis)
 Literature from one or more journal articles
 Public health surveillance data
 Program evaluations
 Qualitative data
 Community members
 Other stakeholders
 Media/marketing data
 Word of mouth
 Personal experience
Subjective
Slide modified from presentation by Ross Brownson, PhD (2011)

41. How to Critique an Article (Handout)
1. Source
* Is the report from a peer reviewed publication?
* Is the research current?
2. Research Problem
* Is the problem identified clearly?
* Is it significant--does the researcher provide a good
argument for significance?

42. How to Critique an Article
3. Literature Review
* Is it convincing that the author reviewed a sufficient
amount of literature? (More than one source?)
* Is it balanced, presenting literature that supports
and that differs from the researcher's position?
* Is the review written critically (giving strengths and
weaknesses of previous work)?
* Are references current, or a combination of current
and classic?

43. How to Critique an Article
4. Theoretical Framework
* Is a theoretical framework specified?
* Does the framework "fit" the problem? Include all
relevant variables?
* Are results interpreted in reference to the
theoretical framework?
--Note: qualitative research, descriptive studies, and
physiologic studies typically will not have a
theoretical framework.

44. How to Critique an Article
5. Variables
* Are the variables in the study appropriate to the problem?
* Are the means to measure the variables appropriate?
6. Hypotheses
* Are hypotheses stated? If not, does the researcher provide
sufficient information to determine what the hypotheses
were?
* If stated, are the hypotheses clear? Specific? Testable?
--Note: qualitative research and many types of descriptive
studies do not involve hypotheses. In a qualitative study the
researcher may suggest hypotheses as a result of the
research.

45. How to Critique an Article
7. Design (overall)
* Is the design specified correctly?
* What design was used?
* Is the design appropriate to answer the research
question?
* Did the researcher attempt to control for threats to
internal and external validity?

46. How to Critique an Article
8. Sample
* Is the sample size adequate?
* Is the sample likely to be similar to members of the
appropriate population overall?
* Are the criteria for including and/or excluding
people or items from the sample clear and
appropriate?

47. How to Critique an Article
9. Data Collection
* Are the instruments or other means for data
collection described sufficiently?
* Are reliability and validity of instruments
addressed? Are these adequate?
* Are data collection methods described clearly?
* Are the data collection methods appropriate?
* Could the researcher have affected the results of
the study in some way related to the collection of
data?

48. How to Critique an Article
10. Data Analysis
* Is the process used to analyze data clear?
* Were the processes for data analysis appropriate to answer the research
question?
* Do the results provide an answer to the research question?
* If tables are provided, are these clear and understandable?
For qualitative studies, some additional items for critique include:
* Does the researcher provide sufficient examples of the data (for
example, passages or quotations from interviews) to support the
identified results?
* Does the researcher describe processes that were used to avoid biasing
or influencing the data obtained and the analysis procedures used? (Ex.
Peer review, logs, memos, "member check―)

49. How to Critique an Article
11. Discussion and Interpretation of Findings
* Does the discussion "fit" with the data? Is it logical based on
the data and results presented?
* Does the researcher discuss the findings in regard to previous
research?
* Does the researcher discuss the findings in regard to the
theoretical framework?
* Does the researcher identify limitations of the study? How do
these affect the quality of the study?
* Does the researcher discuss implications for practice? Are
these appropriate?

50. How to Critique an Article
12. Ethical Considerations
* Does the researcher indicate that approval was obtained
from appropriate review boards?
* Were the rights of human subjects protected (confidentiality)?
13. Application to Public Health
* How similar are the conditions of the study (setting, sample,
interventions, etc.) to our setting?
* How feasible would it be to make a change in practice based
on this research?

51. HOW TO ORGANIZE MY
ARTICLES

52. Organizing Documents for Review
TIPS FOR KEEPING YOURSELF
SANE….AND ORGANIZED!
 Organize by publication type (review articles,
government/agency guidelines, original research)
 Enter information into reference management
database (EndNote, Reference Manager, ProCite)
 Store articles in a file drawer in alphabetical order
by author’s name
 Keep your literature summaries organized
electronically (Xcel, Access, Word)

53. Literature Library
Literature Library
• Xcel Sheet with all of the pertinent information
• More information than an abstract
• Critical for sustaining work over time (i.e.
childhood obesity)
• Materials Located: I:E P EPortfolio Project

54. Literature Library
 Title: Managed Care and School-Based Health Centers
 Authors: Kaplan, DW Archives of Pediatric Medicine 1998
 Study Design: Retrospective cohort
 Study Population: Adolescent members of Kaiser Permanente 342 adequate Records Review
Chi-Squares presented
 Results: Adolescents with access to SBHCs were more than 10 times more likely to make a mental health or substance
abuse visit (98% of these visits were made at the SBHC) (P,.001). Adolescents with SBHC access had an after-hours
(emergent or urgent) care visit rate of 0.33 to 0.52 visits per year less (38%-55% fewer visits) than adolescents without
SBHC access, and, overall, made almost 1 additional medical visit per year. A greater percentage,
80.2%, of adolescents with access to SBHCs had at least 1 comprehensive health supervision visit compared with 68.8%
of adolescents without access (P=.04). In addition, the adolescents with access were screened for high-risk behaviors at a
higher rate. School-based health centers seem to have a synergistic effect for adolescents enrolled in managed care in
providing comprehensive health supervision and primary health and mental health care and in reducing after-hours
(emergent or urgent) visits.
 Conclusion: School-based health centers are particularly successful in improving access to and
treatment for mental health problems and substance abuse.
 Notes: old article but everyone cites this Managed Care and School-Based Health Centers

55. HOW TO RATE THE
LITERATURE (HANDOUT)

56. SUMMARY OF EVIDENCE

57. Public Health Evidence Rating (Handout)
How Established Considerations for Level of Scientific Data Source Examples
Evidence
Proven Peer review via systematic or Based on study design and execution Community Guide
narrative review External validity Cochrane reviews
Potential side benefits or harms Narrative reviews based on published
Costs and cost-effectiveness literature
Likely Effective Peer Review Based on study design and execution Articles in the scientific literature
External validity Research-tested intervention programs
Potential side benefits or harms Technical reports with peer review
Costs and cost-effectiveness
Promising Written program evaluation Summative evidence of effectiveness State or federal government reports
without formal peer review Formative evaluation data (without peer review)
Theory-consistent, plausible, potentially Conference presentations
high-reach, low-cost, replicable
Emerging Ongoing work, practice-based Formative evaluation data Evaluability assessments*
summaries, or evaluation works in Theory-consistent, plausible, potentially Pilot studies
progress high-reaching, low-cost, replicable National Institute of Health (NIH)
Face validity research
(RePORT database)
Projects funded by health foundations
Not Recommended Varies. Evidence of effectiveness is conflicting Varies.
and/or of poor quality.
Weak theoretical foundation
Balance of benefit and harm cannot be
established or evidence demonstrates that
harm outweighs the benefits.
* A pre-evaluation activity that involves an assessment to establish whether or program or policy can be evaluated, what the
barriers to its evaluation might be.
Source: Adapted from Healthy People 2020 and Brownson RC, Fielding JE, Maylahn CM. Evidence-based Public Health: A
Fundamental Concept for Public Health Practices . Annual Review of Public Health. Vol. 30: 175-201

58. FEASIBILITY OF
IMPLEMENTING THE
STRATEGY