Introduction to Systematic Reviews

1. Aims of this session….
• Outline what a systematic review is
• To discuss scope and the formulation of a review question
• The purpose and format of a protocol
• Systematic searching/screening of studies
• Data extraction/quality appraisal and intro to evidence
synthesis

2. What is a
systematic review?
SYSTEMATIC: Done or acting according to a fixed plan
or system: methodical
REVIEW: A critical appraisal of a book, play or other
work

3. What is a
systematic review?
“A systematic review is a review in which there is a comprehensive search for
relevant studies on a specific topic, and those identified are then appraised
and synthesized according to a predetermined and explicit method.”* (*Klassen
et al. Guides for reading and interpreting systematic reviews. Arch Pediatr Adolesc Med 1998;152:700-704.)
A systematic review attempts to collate all empirical evidence that fits pre-
specified eligibility criteria in order to answer a specific research question. It
uses explicit, systematic methods that are selected with a view to minimizing
bias, thus providing more reliable findings from which conclusions can be
drawn and decisions made (Antman 1992, Oxman 1993)

4. Why we need systematic reviews
• Minimise the impact of bias/errors
• Can help to end confusion
• Highlight where there is not sufficient evidence
• Combining findings from different studies can
highlight new findings
• Can mitigate the need for further trials

5. Why we need systematic reviews
• Facilitate rational decision making
• Health care providers, researchers and policy makers
are inundated with unmanageable amounts of
information
– Over 20 million citations in PubMed
– Approx. 75 to 100 RCTs published daily
– Usually impossible to consider all relevant individual primary research
studies in a decision making context
• Enable practitioners to keep up to date with evidence
accumulating in field and to practice evidence-based
medicine

6. Why not traditional reviews
• ‘Unscientific’ rarely pre-specify or make methods explicit
• Rarely transparent or reproducible
• Usually qualitative, subjective, opinions of individual
• Often incomplete, filing cabinet or MEDLINE review
• Difficult to make sense across groups of studies,
especially when conflicting based on qualitative reading
alone

7. Hierarchy of evidence

8. Who undertakes systematic
reviews?
• Cochrane/Campbell Collaboration
• NICE/Regulatory bodies
• Health Technology Assessment
• Academics/researchers/Clinicians
• MSc/PhD students

9. Who undertakes systematic
reviews?
• Multidisciplinary teams
– Clinicians
– Health services researchers
– Information scientists
– Statisticians
– Health Economists
– Patient and public involvement – particularly
for guidelines

10. Key Stages in a Systematic Review- the
process
Define research/review question
In consultation/collaboration with
the clinical community,
commissioners and patient/public
representatives
Identify relevant studies
Develop a comprehensive search
strategy and undertake systematic
searches of the literature
Assess eligibility
Select those studies which meet
the pre-defined inclusion criteria
Data extraction /checking
Develop data extraction from into which
study information and outcome data can
be extracted, checked & verified
Synthesis
Narratively and/or statistically
summarise/describe the data, exploring
similarities and differences between
studies.
Develop review protocol
Pre-specify the type of studies to
be included, the methods of
collating, appraising and analysing
data
Knowledge translation
Review details and results are
disseminated to relevant target audiences
using appropriate formats
Study assessment/appraisal
Assess the quality and validity of the
included studies using the pre-defined
method.

11. Define research/review question
• Questions may be broad or narrow
• Well-formulated questions will guide many
aspects of the review process
– Searching strategy
– Inclusion/exclusion criteria
– Data extraction
– Choice of synthesis method
– Presentation/dissemination of findings

12. Quick Activity
Discuss a very broad question and how you might narrow it? (10
mins)
Discuss the potential limitations of your review questions
If time and resource were not a limitation – consider how useful
would the answer to your review question be?

13. Current guidance
• a clear and concise statement of a
review's objectives (or questions) is critical
and should begin with a precise statement
of the primary objective, including the
interventions reviewed and the targeted
problem; ideally, this would be presented
in a single sentence
Cochrane & Prisma Statment

14. Current guidance
“To assess the effects of [intervention or
comparison] for [health problem] in [types of
people, disease or problem, and setting if
specified].”
• Several criteria/frameworks proposed to
help guide question development

15. Question formulation
• Determining the scope is a decision dependent upon
multiple factors:
– Perspectives regarding a question’s relevance and
potential impact;
– Supporting theoretical, biologic and epidemiological
information;
– The potential generalizability and validity of answers to the
questions;
– Available resources;
– The wider literature base – has a recent high-quality SR
been conducted?

16. Question formulation
• Advantages and disadvantages to both broad and narrow questions
• The validity of very broad question may be criticized for ‘mixing
apples and pears’; but advantages might include
– Comprehensive summary of the evidence
– Generalizability of findings
• Most obvious advantage of narrow focus is clarity of objectives and
ease of reading; but disadvantages might include
– Sparse evidence may limit findings/usefulness
– Generalizability of findings

17. Question formulation
• Often dealing with complex interventions
• Might be a need to develop working definitions of the
intervention of interest
– Several options on how to do this (pragmatic real world v
theoretical, logic models, etc.)
– Use content experts outside the review team to ensure
that the resulting definitions are likely to be robust and
meaningful

18. Protocol Development
• A protocol is an essential component of
the systematic review process
• Helps to ensure careful a priori planning
– Consistency
– Transparency
– Integrity
• Integral part of the process for leading
organisations/publication process

19. Protocol Development
• One of the features that distinguish a
systematic review from a narrative review is
the pre-specification of criteria
– Inclusion
– Exclusion
– Methods
– Outcomes to be synthesised
– Etc.

20. PROSPERO – CRD initiative
• Search for existing
current reviews
• Register their planned
review online
• Publish protocol online
• Update record on
Prospero website as the
review progresses
• Avoids duplication of
reviews

21. Searching for Information
• Types of
– Studies (RCTs, non-RCTs, cohort/case-controlled)
– Population and setting
– Interventions
– Outcome measures
• Cochrane Handbook and CRD Guidelines
– Both provide explanations re the difference study designs, likely
biases and issues to consider when including them
www.york.ac.uk/inst/crd/pdf/Systematic_Reviews.pdf
http://handbook.cochrane.org/

22. Searching for Information
• MESH terms and key words/synonyms
– Medical Subject Heading – controlled vocabulary
thesaurus used for indexing articles
– young; adoles*; teen*; child*...................
*end of the ‘stem’ of the word it will automatically search
for all the endings for that word stem
• Child* will also return children, childbearing, childbirth
and so on…

23. Searching for Information
• Word variants
– AIDS
• acquired immunodeficiency syndrome
• acquired immuno-deficiency syndrome
• acquired immune deficiency syndrome
• acquired immune-deficiency syndrome
• Synonyms e.g. Newborn: infant, toddler, baby, etc.
• Plurals e.g. child : children OR teenager : teenagers
• Spelling variants (UK vs US) e.g. randomise/randomize

24. Searching for Information
Where to search
– Electronic databases: Medline, Embase, Cochrane, PsycInfo, etc.
– Grey literature, dissertations, theses, conference proceedings, national
bodies (NICE, HTA), clinical trial database (www.clincialtrails.gov/)
– Look at the databases own guidance for searching they vary!

25. Boolean operators

26. Searches in medline
2006-2010 database
Alcohol – subject (all subheadings) 5565
Alcohol – subject or keyword 12052
([young or adoles* or teen*] [review or
system*]
19180
Rows above combined with or 6791

27. More specific – get help!

28. Selection of Studies
• Reference manager software package
– Endnote – RefMan – ProCite – Mendeley
• Import results and screen
– Assess titles/abstracts against your predetermined criteria
– If in doubt include
– Retrieve full text articles of initial selections
• Assess full text for inclusion
– Requires judgement (>1 reviewer)
– Check reviewer agreement (3rd review to resolve)
– Use a selection form to ensure consistency and record decisions

29. Data Extraction/Quality Appraisal

30. Data Extraction
• Be clear what information you want from the studies:
– Study details
– Data for your analysis
• Information will need to be collected relating to:
– Methodology
– Population
– Interventions being compared
– Outcomes evaluated

31. Give consideration to….
• What effect measures you are you going to calculate
– What data do you need to do this?
• How are you planning to group studies for the analysis?
– By intervention?
– By study design?
• What information do you need to extract to enable you to organise
and analyse the way you want?
• REMEMBER YOUR PROTOCOL – IT IS YOUR ROADMAP, FOLLOW IT!

32. How much to extract??
• Level of judgement is required
– Sufficient to describe studies
– Sufficient to allow you to undertake the planned
analysis
– Sufficient so you do not need to return to the full
text papers
• However
– You need to limit unnecessary detail

33. Data extraction software?
• There is a wide selection of software to choose from
• Selection depends on a number of factors
• Main considerations are probably
• What are you are familiar with?
• What package best suits your data?
• How many included studies do you have?

34. Which software?
• Word
• Excel
• Access
• EPPI reviewer
• COEVIDENCE
• REVMAN
• ????

35. Consistency/Standardisation
• We all have to be doing the same thing
• Essential >one reviewer is extracting data
• Data must be interpreted in the same way by all reviewers
• Independent piloting of data extraction forms – always one
standardised form
• Regular discussion of progress/disagreements
• Regular comparison of data extraction – don’t wait till the end

36. Efficient data extraction
• Once data extraction is complete you may need to:
• Sort/search your data
• Filter data
• Calculate frequencies
• Transform data (e.g. SE to SD)
• Categorising/coding data will make these tasks easier
• Needs to be implemented with consistency by the whole team
• A database can be designed to have this functionality

37. Things to consider
• Are you including more than one study design?
• You may need separate forms for each study design
• However, you are still answering the same question, so make sure
the core information extracted is the same
• Have one or a few studies reported data differently from the
others?
• Will the data still be useful?
• Should you include it?
• Make sure the core information extracted is the same
• You may need to update the form, or have more than one form
• Any changes need to be agreed and made consistently

38. Stay on track……
• Be careful about collecting ‘extra’ data
• It is very tempting to collect data that are not directly relevant
to the review question
• The data needed to answer the review question should have
already been decided (REMEMBER YOUR PROTOCOL)
• Collect data for good reasons – stay focused and don’t get
side-tracked
• Time and effort to collect, only to find it is not useful

39. Quality Assessment & Critical Appraisal
• Why bother????
• What are we trying to achieve?
• Not all published and unpublished literature is
rigorous!
– being in a journal doesn’t mean it is good
• Quality may be used as an explanation for
differences in study results or to guide
interpretation of findings, strength of inferences

40. Quality Assessment & Critical Appraisal
• Quantitative studies
– Internal validity
– Bias: selection; performance; detection; attrition; reporting
– External validity
• Move away from checklists/numerical scores to domain based
assessment
– Cochrane Risk of Bias - RCTs
– QUADAS 2 – diagnostic accuracy
– ROBIS for systematic reviews

41. Quality Assessment & Critical Appraisal
• Qualitative studies
• Three broad categories
– Rigour: has a thorough and appropriate approach
been applied to key research methods in the
study?
– Credibility: are the findings well presented and
meaningful?
– Relevance: how useful are the findings to you and
your organisation?

42. CASP appraisal checklist
1. Clear aims of research (goals, why it is important, relevance)
2. Appropriate methodology
3. Sampling strategy
4. Data collection
5. Relationship between researcher and participants
6. Ethical issues
7. Data analysis
8. Findings
9. Value of research (context dependent)

43. Data Synthesis
• Building up; putting together; making a whole out of the
parts; the combination of separate elements of thought
into a whole; reasoning from principles to a conclusion

44. Data Synthesis
• Results from different studies need to be
synthesised
• Are studies and results similar enough to be
combined into a single numerical result?
– NO – qualitative descriptive/narrative summary
– YES – quantitative meta-analysis
• Heterogeneity
– Difference in results can arise due to differences in study design,
population, selection, intervention delivery
– How similar is similar? Results from heterogeneous studies
should not be pooled

45. Narrative synthesis
• Instead of/alongside
meta-analysis
• Potential bias in
presentation
• Lack of a take home
message

46. Tools for narrative synthesis
• Partly informed by methodological work in qualitative
synthesis
– Tabulation
– Groupings and clusters
– Vote counting as a descriptive tool
– Examination of moderator variables (elements of e.g. setting,
population)
• Rodgers et al Evaluation 2009 15 49-72

47. Meta-analysis/forest Plot

48. Most important thing:
Be organised!!!

49. Gantt charts

50. Reference management
• Use a reference manger to sift and store
• Keep all citations retrieved
– Add in those you can’t download
• Use to de-duplicate results
• Sift citations for inclusion/exclusion
– Can use codes/notes

51. Version control
• Dates – YYYYMMDD
• Version numbering
– v0.1 = first draft
– v1.0 = final version
– v1.1 = minor amendments to final version
– v2.0 = major revision
• Avoid using draft, draft 1, final final, etc.
• Clear naming convention
– E.g. Date_project_title
• 20/11_Autism HTA_resultsv1.2.doc

52. Other help available
• FMS Systematic review group
– Informal monthly session where methods are
discussed and issues can be raised
– fiona.beyer@ncl.ac.uk or jenni.hislop@ncl.ac.uk
– alternatively you have my contact details
• MSc in Public Health and Health Services
Research (~October 2016)
– 10 credit module ‘Introduction to systematic reviewing and
critical appraisal’
– pghealth@newcastle.ac.uk