This document provides an outline and overview of a speech on what makes a systematic review systematic from an epidemiological perspective. It discusses key aspects of systematic reviews including how they differ from non-systematic reviews, guidelines for conducting systematic reviews, inclusion of randomized and observational data, assessing risk of bias, and summarizing results. The speaker aims to explain the value added by systematic reviews over traditional literature reviews in providing objective and replicable summaries of evidence on a topic.
Call Girls Whitefield Just Call 7001305949 Top Class Call Girl Service Available
What makes a systematic review systematic
1. What makes a systematic review
systematic? Epidemiologic perspectives
ANNA SIDORCHUK, MD, PHD, RESEARCH COORDINATOR
KAROLINSKA INSTITUTET
DEPARTMENT OF PUBLIC HEALTH SCIENCES
ANNA.SIDORCHUK@KI.SE
THE EPIDEMIOLOGY PHD PROGRAM SEMINAR
APRIL 29, 2013
2. Outline of the speech
Non-systematic vs. systematic review
Why do systematic reviews? What is the added value?
PRISMA/MOOSE/AMSTAR guidelines and checklists
Randomized vs. observational data
Process (criteria, optimize search, determine data to be
abstracted, quality score)
Anna Sidorchuk 2
3. 30 april 2013Anna Sidorchuk
Primary data versus secondary data
Traditionally, researchers collect and analyze their own data
(referred to as primary data)
Secondary data analysis is based on data collected by someone
else (or reanalysis of your own published data)
Systematic review is one of the ways to analyze secondary data
Systematic review - systematic, qualitative review of published
research in a particular field
Adapted from: ESRC Workshop, Researcher Development Initiative, Department of Education, University of Oxford
4. 30 april 2013Anna Sidorchuk
What The Lancet thinks about
systematic reviews
In 2005, the editors wrote:
“. . . we will require authors of clinical trials submitted to The
Lancet to include a clear summary of previous research findings,
and to explain how their trial’s findings affect this summary. The
relation between existing and new evidence should be illustrated
by direct reference to an existing systematic review and meta-
analysis…”
5. Stages of waste in the production and reporting
of research evidence relevant to clinicians and
patients by Chalmers and Glasziou (Lancet, vol. 374, 2009)
Anna Sidorchuk 5
6. 30 april 2013Anna Sidorchuk
Why do we do systematic reviews?
To help busy clinicians to summarize current knowledge in
relation to the area of interest
To provide high-quality research evidence to guide clinical
practice
To support clinical decision-making
To support research proposals
7. 30 april 2013Anna Sidorchuk
How systematic review can help busy
clinicians?
By systematically:
identifying,
appraising,
synthesizing,
and, if appropriate, statistically combining studies on a
specific topic
8. 30 april 2013Anna Sidorchuk
In classifications of levels of evidence,
systematic reviews are included in the highest level of
evidence
9. Hierarchy of evidence: a framework for ranking
evidence evaluating healthcare interventions
Journal of Clinical Nursing
Volume 12, Issue 1, pages 77-84, 20 DEC 2002 DOI: 10.1046/j.1365-2702.2003.00662.x
http://onlinelibrary.wiley.com/doi/10.1046/j.1365-2702.2003.00662.x/full#f1
10. 30 april 2013Anna Sidorchuk
Meta-analysis
A statistical method to combine findings across studies
Should be considered within the framework of systematic reviews
review needs to use a systematic approach to minimize bias, address
the issues of the completeness of the evidence, quality of studies
and combinability of studies
Meta-analysis - the statistical pooling of the results of
studies that are part of a systematic review
11. When systematic review is required?
Before undertaking a systematic review it is necessary to
check whether there are already existing or ongoing reviews,
and whether a new review is justified
Search:
the Database of Abstracts of Reviews of Effects (DARE)
the Cochrane Database of Systematic Reviews (CDSR)
30 april 2013Anna Sidorchuk
Adapted from: Systematic Reviews. CRD’s guidance for undertaking reviews in health care (2009). Centre for Reviews and
Dissemination, University of York. ISBN 978-1-900640-47-3.
12. When is meta-analysis appropriate?
There exists a critical mass of comparable studies designed to
address a common research question
Data are presented in a form that allows the meta-analyst to
compute an effect size for each study
Characteristics of each study are described in sufficient detail
to allow meta-analysts to compare characteristics of different
studies and to judge the quality of each study
30 april 2013Anna Sidorchuk
13. Systematic reviews vs. non-systematic
Compared to traditional literature reviews:
there is a definite methodology employed in the research analysis
(more like that used in primary research); and
the results of the included studies are quantified to a standard metric
thus allowing for statistical techniques for further analysis.
Therefore process of reviewing research literature is more
objective, transparent, and replicable; less biased and
idiosyncratic to the whims of a particular researcher
30 april 2013Anna Sidorchuk
14. Systematic reviews vs. non-systematic
Feature Systematic review Narrative review
Question Often a focused one question Often broad in scope
Sources and search Comprehensive sources and
explicit search strategy
Not usually specified,
potentially biased
Selection Criterion-based selection,
uniformly applied
Not usually specified,
potentially biased
Appraisal Rigorous critical appraisal Variable
Synthesis
Qualitative summary that
includes statistical synthesis
(meta-analysis)
Often a qualitative summary
Inferences Usually evidence-based Sometimes evidence-based
Anna Sidorchuk 14
Adapted from: Y Yuan and R H Hunt, Am J Gastroenterol 2009; 104:1086–1092; doi:10.1038/ajg.2009.118
15. 18 September, 2009Anna Sidorchuk 15
Increase in the number of published
meta-analyses (PubMed references)
Publication year
16. • The essence of good science is replicable and generalisable results
Do we get the same answer to research questions when we run the study again?
• The primary aims of meta-analysis is to test the generalisability of results
across a set of studies designed to answer the same research question
Are the results consistent? If not, what are the differences in the studies that explain
the lack of consistency?
Why are systematic review and meta-
analysis important?
30 april 2013Anna Sidorchuk
17. When there is systematic variation in outcomes from different studies,
meta-analysis tries to explain these differences in terms of study
characteristics:
measures used
study design
participant characteristics
controls for potential bias
etc.
If the results of individual studies are
inconclusive or controversial…
30 april 2013Anna Sidorchuk
18. “… doing a meta-analysis is easy,
doing one well is hard.”
Ingram Olkin, Stanford University
19. Benefits of systematic reviews, incl. meta-
analysis
Increased power: by combining information from many
individual studies, the meta-analyst is able to detect systematic
trends not obvious in the individual studies
Conclusions based on the set of studies are likely to be more
accurate than any one study
30 april 2013Anna Sidorchuk
20. Benefits of systematic reviews, incl. meta-
analysis
Improved precision: based on information from many studies,
the meta-analyst can provide a more precise estimate of the
population effect size (and a confidence interval)
Provides potential corrections for potential biases,
measurement error and other possible artefacts
Identifies directions for further primary studies to address
unresolved issues
30 april 2013Anna Sidorchuk
21. Benefits of systematic reviews, incl.
meta-analysis
Typically there is study-to-study variation in results. When
this is the case, the meta-analyst can explore what
characteristics of the studies explain these differences (e.g.,
study design) in ways not easy to do in individual studies
Easy to interpret summary statistics (useful if communicating
findings to a non-academic audience)
30 april 2013Anna Sidorchuk
22. Limitations of systematic reviews, incl.
meta-analysis
Comparing apples and oranges
Quality of the studies included in the meta-analysis
What to do when studies don’t report sufficient information
(e.g., “non-significant” findings)?
Including multiple outcomes in the analysis (e.g., different
achievement scores)
Publication bias
22
23. Limitations of systematic reviews, incl.
meta-analysis
Meta-analysis conclusions may still differ if different studies
are sampled or excluded for different reasons
Need to be explicit
Quality standards
Inappropriate handling of data can lead to wrong conclusions
Sample size consideration
23
24. Publication bias
Studies that are published are more likely to report
statistically significant findings. This is a source of potential
bias
30 april 2013Anna Sidorchuk
25. Publication bias
The debate about using only published studies:
peer-reviewed studies are presumably of a higher quality
VERSUS
significant findings are more likely to be published than non-
significant findings
There is no agreed upon solution. However, one should retrieve all studies
that meet the eligibility criteria, and be explicit with how they dealt with
publication bias. Some methods for dealing with publication bias have
been developed (e.g., Fail-safe N, Trim and Fill method)
30 april 2013Anna Sidorchuk
26. RCT vs. observational studies
Both are fine as long as all the methodological issues are
addressed
If your focus is effect of therapy or prevention – use RCTs
If you are interested in prognosis, etiology, adverse
effects, association between risk factors and outcomes –
use observational studies
Fine to use both types of studies in the same review
Anna Sidorchuk 26
27. Steps in a meta-analysis
Establish
research
question
Define
relevant
studies
Develop
code
materials
Locate and
collate
studies
Pilot coding;
coding
Data entry
and effect
size
calculation
Main
analyses
Supplementar
y analyses
30 april 2013Anna Sidorchuk Adapted from: ESRC Workshop, Researcher Development Initiative, Department of Education, University of Oxford
28. Practical tips
Define all key elements (PICO model) a priory
Formulate the research question as clear as possible
Make an agreement with all your co-authors on the key elements
If needed ask the experts in the area of outcome or/and exposure for
advice
Keep everything in the protocol
When writing an article include the elements in the aim
30 april 2013Anna Sidorchuk
29. Practical tips
Define all inclusion/exclusion criteria and components of search a priory
Make criteria and the components as clear as possible, but feel free to
make some changes during the actual literature search (usually due to
availability of data, e.g. if only retrospective studies are available on the
issue, or no info on ethnicity, etc)
Make an agreement with all co-authors on abovementioned
Again, keep everything in the review protocol
30 april 2013Anna Sidorchuk
30. Practical tips
Define coding and form for data extraction a priory
Make an agreement with all co-authors on abovementioned
If use Excel for data extraction and, therefore, for coding, keep the legend
for each coding element within the same file on the separate sheet
As an alternative – use Access
Keep everything in the review protocol!
30 april 2013Anna Sidorchuk
31. Practical tips
Define as detailed as possible by reading other meta-analysis and
systematic reviews on the similar outcome(s) and exposure(s):
All key words
MESH terms
Define the electronic sources to be used for your search
Ask librarians for advice on search strategy and technic
Define if grey literature will be included
Keep everything in the protocol!!!
30 april 2013Anna Sidorchuk
33. Quality Standards in Systematic
reviews and Meta-Analysis
Several organizations work on this
Cochrane collaboration
http://www.cochrane.org
Campbell collaboration
http://www.campbellcollaboration.org
Consort: Consolidated Standards of Reporting Trials, includes
Quorum: Quality of Reporting of Meta-analyses group
http://www.consort-statement.org/mod_product/uploads/QUOROM
Statement 1999.pdf
PRISMA (Preferred Reporting Items for Systematic reviews and Meta-
Analyses), AMSTAR (A measurement tool to assess systematic reviews)
30 april 2013Anna Sidorchuk
34. 30 april 2013Anna Sidorchuk
PRISMA: Checklist of items to include when reporting a systematic review or meta-
analysis.
Section/Topic # Checklist Item
Reported
on Page #
TITLE
Title 1 Identify the report as a systematic review, meta-analysis, or both.
ABSTRACT
Structured
summary
2
Provide a structured summary including, as applicable:
background; objectives; data sources; study eligibility criteria,
participants, and interventions; study appraisal and synthesis
methods; results; limitations; conclusions and implications of key
findings; systematic review registration number.
INTRODUCTION
Rationale 3
Describe the rationale for the review in the context of what is
already known.
Objectives 4
Provide an explicit statement of questions being addressed with
reference to participants, interventions, comparisons, outcomes,
and study design (PICOS).
35. 30 april 2013Anna Sidorchuk
PRISMA: Checklist of items to include when reporting a systematic review or meta-
analysis.
Section/Topic # Checklist Item
Reported
on Page #
METHODS
Protocol and
registration
5
Indicate if a review protocol exists, if and where it can be
accessed (e.g., Web address), and, if available, provide
registration information including registration number.
Eligibility
criteria
6
Specify study characteristics (e.g., PICOS, length of follow-
up) and report characteristics (e.g., years considered,
language, publication status) used as criteria for eligibility,
giving rationale.
Information
sources
7
Describe all information sources (e.g., databases with dates of
coverage, contact with study authors to identify additional
studies) in the search and date last searched.
Search 8
Present full electronic search strategy for at least one
database, including any limits used, such that it could be
repeated.
Study selection 9
State the process for selecting studies (i.e., screening,
eligibility, included in systematic review, and, if applicable,
included in the meta-analysis).
Data collection
process
10
Describe method of data extraction from reports (e.g., piloted
forms, independently, in duplicate) and any processes for
obtaining and confirming data from investigators.
Data items 11
List and define all variables for which data were sought (e.g.,
PICOS, funding sources) and any assumptions and
simplifications made.
36. 30 april 2013Anna Sidorchuk
PRISMA: Checklist of items to include when reporting a systematic review or meta-
analysis.
Section/Topic # Checklist Item
Reported
on Page #
METHODS
Risk of bias in
individual studies
12
Describe methods used for assessing risk of bias of
individual studies (including specification of whether this
was done at the study or outcome level), and how this
information is to be used in any data synthesis.
Summary
measures
13
State the principal summary measures (e.g., risk ratio,
difference in means).
Synthesis of
results
14
Describe the methods of handling data and combining
results of studies, if done, including measures of consistency
(e.g., I2
) for each meta-analysis.
Risk of bias
across studies
15
Specify any assessment of risk of bias that may affect the
cumulative evidence (e.g., publication bias, selective
reporting within studies).
Additional
analyses
16
Describe methods of additional analyses (e.g., sensitivity or
subgroup analyses, meta-regression), if done, indicating
which were pre-specified.
37. 30 april 2013Anna Sidorchuk
PRISMA: Checklist of items to include when reporting a systematic review or meta-
analysis.
Section/Topic # Checklist Item
Reported
on Page #
RESULTS
Study selection 17
Give numbers of studies screened, assessed for eligibility,
and included in the review, with reasons for exclusions at
each stage, ideally with a flow diagram.
Study
characteristics
18
For each study, present characteristics for which data were
extracted (e.g., study size, PICOS, follow-up period) and
provide the citations.
Risk of bias
within studies
19
Present data on risk of bias of each study and, if available,
any outcome-level assessment (see Item 12).
Results of
individual studies
20
For all outcomes considered (benefits or harms), present, for
each study: (a) simple summary data for each intervention
group and (b) effect estimates and confidence intervals,
ideally with a forest plot.
Synthesis of
results
21
Present results of each meta-analysis done, including
confidence intervals and measures of consistency.
Risk of bias
across studies
22
Present results of any assessment of risk of bias across
studies (see Item 15).
Additional
analysis
23
Give results of additional analyses, if done (e.g., sensitivity
or subgroup analyses, meta-regression [see Item 16]).
38. 30 april 2013Anna Sidorchuk
PRISMA: Checklist of items to include when reporting a systematic review or meta-
analysis.
Section/Topic # Checklist Item
Reported
on Page #
DISCUSSION
Summary of
evidence
24
Summarize the main findings including the strength of
evidence for each main outcome; consider their relevance to
key groups (e.g., health care providers, users, and policy
makers).
Limitations 25
Discuss limitations at study and outcome level (e.g., risk of
bias), and at review level (e.g., incomplete retrieval of
identified research, reporting bias).
Conclusions 26
Provide a general interpretation of the results in the context of
other evidence, and implications for future research.
FUNDING
Funding 27
Describe sources of funding for the systematic review and
other support (e.g., supply of data); role of funders for the
systematic review.
39. Quality Assessment of Primary Studies
A common practice is to assign a quality score to primary
studies
Preferably based on an explicit checklist usually
The judgment to include or exclude a study must be stated
and justified explicitly
This judgment is mostly based on quantitative criteria, but
involves some subjectivity
30 april 2013Anna Sidorchuk
40. NEWCASTLE - OTTAWA QUALITY ASSESSMENT SCALE
CASE CONTROL STUDIES
Note: A study can be awarded a maximum of one star for each numbered item within the Selection and Exposure
categories. A maximum of two stars can be given for Comparability.
Selection
1) Is the case definition adequate?
a) yes, with independent validation ¯
b) yes, eg record linkage or based on self reports
c) no description
2) Representativeness of the cases
a) consecutive or obviously representative series of cases ¯
b) potential for selection biases or not stated
3) Selection of Controls
a) community controls ¯
b) hospital controls
c) no description
4) Definition of Controls
a) no history of disease (endpoint)
b) no description of source
30 april 2013Anna Sidorchuk
41. NEWCASTLE - OTTAWA QUALITY ASSESSMENT SCALE
CASE CONTROL STUDIES
Comparability
1) Comparability of cases and controls on the basis of the design or analysis
a) study controls for _______________ (Select the most important factor.) ¯
b) study controls for any additional factor ¯ (This criteria could be modified to indicate specific control for a
second important factor.)
Exposure
1) Ascertainment of exposure
a) secure record (eg surgical records) ¯
b) structured interview where blind to case/control status ¯
c) interview not blinded to case/control status
d) written self report or medical record only
e) no description
2) Same method of ascertainment for cases and controls
a) yes ¯
b) no
3) Non-Response rate
a) same rate for both groups ¯
b) non respondents described
c) rate different and no designation
30 april 2013Anna Sidorchuk
42. “Ugly face” of systematic reviews and
meta-analyses
Can be misleading
Can be misused
30 april 2013Anna Sidorchuk