1. Systematic review and
Meta-analysis
By: Mehdi Ehtesham
MPH/ Epidemiologist,
Avicenna Research Consulting
London, UK

2. Concept of Meta-analysis
Meta-analysis is defined as the statistical synthesis of the data from separate but similar (comparable) studies,
leading to a quantifiable summary of the pooled results Meta-analysis differs from most medical and
epidemiological studies in that no new data are collected. Instead, results from previous studies are combined.
Steps in carrying out meta-analysis include:
• formulating the problem and study design;
• identifying relevant studies;
• excluding poorly conducted studies or those with major methodological flaws;
• measuring, combining and interpreting the results.
Which studies are identified and whether they are included or excluded from the meta-analysis are crucial
factors. Another important step is measuring the results of the studies on a single scale. This allows
comparisons to be made between studies even if they used different measures of outcome. Meta-analysis is a
relatively new scientific method; research into the best techniques to use is still ongoing and expanding into
new areas. It is not yet as well-accepted as other statistical techniques that have a longer tradition of use.
The use of meta-analysis in medicine and epidemiology has increased in recent years for :
ethical reasons, cost issues, and the need to have an overall idea of effects of a particular intervention in
different population groups.
This is particularly true in the area of clinical trials, where the sample size of individual trials is often too small to
permit conclusions to be drawn from any one trial, although conclusions can be drawn from aggregated results.
For example, meta-analysis showed a significant effect in preventing a second heart attack or stroke, even
though no single study had convincingly shown this.

3. Introduction and learning objectives
Does a large number of studies help us to arrive at a definitive
answer, or do more studies lead to more confusion and lack of
clarity? And what about all the unpublished studies investigating
the same question?
reviews of all available evidence on a articular question can be
extremely useful. However, in order to assess the validity of the
various studies, integrate appropriate information, and arrive at an
overall result, we need a rigorous, systematic approach. That is, we
need to carry out a systematic review of all the evidence, both
published and unpublished.
A systematic review is a review of the
methods and results of all individual
studies designed to answer the same
research question and that conform to
set criteria.

4. As part of this process we may want to obtain a
quantitative summary (for example, of a treatment effect
or an exposure risk) across comparable studies. This may
be (most commonly) through combining the results of the
individual studies, or by analyzing the raw data from the
studies if they are available. The statistical methods
required to carry out such analyses are known as meta-
analysis.
Meta-analysis
is the statistical analysis of the data/results from studies
included in a systematic review to produce an overall, pooled
result.
Introduction and learning objectives

5. Meta-analysis
is commonly used to combine data from a number of randomized,
controlled trials (RCTs) of therapies or interventions.
However, the techniques may also be used for observational
epidemiological studies of risk factors.
The results of any one study may have too much random error to show
any clear effect. That is, the study may not be powerful enough to
demonstrate significant differences (type II error).
In combining data from several studies, we increase the sample
size and so increase power and obtain more precise estimates.
Systematic reviews and associated meta-analyses are playing an
increasingly important role in both research and practice.
Introduction and learning objectives

6. Many studies have reported the results of interventions to reduce illness through
improvements in drinking water, sanitation facilities, and hygiene practices in
less developed countries.
There has, however, been no formal systematic review and meta-analysis
comparing
the evidence of the relative effectiveness of these interventions. We
developed a comprehensive search strategy designed to identify all peer-
reviewed articles, in any language, that presented water, sanitation, or
hygiene interventions. We examined only those articles with specific
measurement of diarrhoea morbidity as a health outcome in non-outbreak
conditions. We screened the titles and, where necessary, the abstracts of 2120
publications.
46 studies were judged to contain relevant evidence and were reviewed in
detail. Data were extracted from these studies and pooled by meta-analysis to
provide summary estimates of the effectiveness of each type of intervention.
All of the interventions studied were found to reduce significantly the risks of
diarrhoeal illness. Most of the interventions had a similar degree of impact on
diarrhoeal illness, with the relative risk estimates from the overall meta-analyses
ranging between 0.63 and 0.75. The results generally agree with those from
previous reviews, but water quality interventions ( point-of-use water treatment)
were found to be more effective than previously thought, and multiple
interventions (consisting of combined water, sanitation, and hygiene measures)
were not more effective than interventions with a single focus. There is some
evidence of publication bias in the findings from the hygiene and water
treatment interventions.
Example:

7. Why is it important that reviews be
systematic?
Conducting a systematic review involves a great deal of work,
although it is generally quicker
and less costly than carrying out a new study.
Clearly, it is important to establish why all of this activity is so
important.

8. Method of systematic review – overview
We require systematic methods in order to establish whether
research findings are
consistent and generalisable. The process of carrying out a
systematic review comprises
the following steps:
1. Decide on the objectives of the review.
2. Define criteria for including/excluding studies in the review.
3. Find / locate studies.
4. Select studies according to the eligibility criteria.
5. Assess the methodological quality of the studies.
6. Extract data.
7. Describe and compile the results.
8. Report the results of the review.

9. Defining criteria for inclusion and exclusion of
studies
for a systematic review, it is important to specify the criteria
studies need to meet before
being eligible for inclusion. Failure to do this can result in bias
in terms of which studies
are selected. The inclusion criteria should relate to the
required study populations,
treatments (which may be interventions or risk factors in
observational studies), study outcomes, length of follow-up
and aspects of methodological quality.

10. Methods
Initial selection criteria and data
extraction
Two selection criteria were used to identify articles:
(1) description of specific water, sanitation,
or hygiene interventions, or some combination of such
interventions
(2) Diarrhoea morbidity reported as the health outcome,
measured under endemic
(non-outbreak) conditions.
In addition, only published studies were used, to maintain
quality (via peer review)
and transparency.
No study was excluded from the review or meta-analysis on
the basis of quality criteria
alone.

11. Identifying relevant studies Publication bias
When carrying out a systematic review it is important to include relevant
unpublished studies, as well as published, peer-reviewed studies. Thus a
simple literature search using electronic databases is not sufficient. One
of the most important reasons for this is that studies reporting statistically
significant results are more likely to be published than those with non-
significant results.
This selective publication, or publication bias, means that we may reach
over-optimistic or misleading conclusions if we include only published
studies in our systematic review. This is particularly the case with small
studies: we have seen that any study might occasionally produce a
significant effect when no such effect really exists (type 1 error), but
since small studies are more difficult to publish than large ones, there is a
tendency for those with significant results to be offered (and accepted)
for publication more frequently than small studies without ‘interesting’
results.
Statistical significance does not guarantee the quality, validity or clinical
significance of the research and good studies which have conclusively
demonstrated a lack of treatment effect or lack of association may
never be published.
In the future such registries will make it easier to identify studies for a
systematic review in order to reduce the risk of publication bias.

12. How should we measure methodological
quality?
Poor quality could be among the criteria for excluding studies
from a review. Given the somewhat subjective nature of the
decision process as to whether studies meet
minimum inclusion criteria in relation to quality, it is good
practice for two reviewers independently to check the eligibility
of candidate studies, with disagreements being resolved
through discussion with a third reviewer.
Methodological quality can be quantified by scoring the quality
of studies on a pre-existing
scale, such as that developed for randomised trials by Jadad et
al. (1996). This scale has five items:
two relate to blinding, two to randomisation, and one to the
description of withdrawals/dropouts.
When using the Jadad scale, each of the five items receives a
‘yes’ or a ‘no’, resulting in
an overall/composite quality score that can range from 0 to 5;
higher scores reflect better
methodological quality.

13. Assessment of methodological quality
What is methodological quality?
The concept of methodological quality is hard to define but
typically it is used to describe the
design, procedures and conduct of a study, how the analysis has
been carried out, the relevance of the study to policy and
practice, and/or the quality of the reporting.
For example, ideally, we might set the following inclusion criteria
for a systematic review of clinical trials:
• placebo controls (if possible)
• evidence of effective randomization
• blinding used, ideally at least double blinding; that is, of
study participants and research staff assessing the
outcome
• near complete follow-up of subjects
• analysis by intention-to-treat
Complete information is not always available however, even in
published journal articles, and it is sometimes necessary to define
the methodological quality of studies in terms of only basic
criteria.
Alternatively, it may be necessary to contact the authors to
obtain the missing information.

14. Why measure methodological quality?
Even after we have excluded studies with poor methodological quality, it is
likely that the
remaining studies will still be of variable quality. It has been demonstrated
empirically that studies with poor quality can distort the results from
systematic reviews and meta-analyses. For example, a study of 250 trials from
33 meta-analyses of a range of interventions relating to pregnancy and
childbirth found that non-random treatment allocation and lack of double
blinding of controlled intervention trials were associated with larger treatment
effects. The main findings of this study are presented in Table 9.1.2,
comparing the effect estimates of trials defined as having
inadequate/unclear methodology with those from trials defined as having
adequate methodology (an OR ratio of less than 1 indicates an exaggerated
treatment effect).

15. Extracting data and describing the results
The process of data extraction should be carried out with as much care as
was taken for assessing the methodological quality of studies. Again, it is
important that two independent observers extract the data to ensure that
errors are minimized.
Data extraction requires that a (data extraction) form be used for all the
studies selected for the review, and this should be carefully designed,
piloted and revised if required. Typically information required for
randomized trials will comprise
(i) the reference,
(ii) the setting and sample of people being studied,
(iii) the intervention or treatment and how this is measured,
(iv) the health outcome and how this is measured and
(v) a measure of the size of effect (e.g. OR or relative risk) with associated
CIs.
The descriptive presentation of results from a systematic review normally
involves three stages.
First, it is important to state clearly the numbers of studies included and
rejected from the review.
Second, the results of the articles included in the systematic review are
tabulated presenting a
summary of the main attributes of the studies.
The third stage is to outline descriptively the main results of the review.
The emphasis on presenting the results from a systematic review is placed on
a descriptive narrative summary of the findings. the value of a systematic
review in providing a quantitative summary of results is through the pooling
of data in the form of a meta-analysis.

16. Results of the meta-analyses: fixed-effect estimate of relative risk (RR) 0.75
(95% CI 0.72–0.78); heterogeneity p < 001; random-effects estimate of RR
0.63 (95% CI
0.52–0.77); Begg’s test p = 019. †Calculated. ‡Result used for the overall
meta-analysis, which provided a pooled estimate of relative risk.

17. Method of meta-analysis – overview
Essentially there are four main steps in carrying out a meta-
analysis. These are:
• An assessment of publication bias using a funnel plot (or a
statistical analogue of the funnel
plot) to look for asymmetry.
• A statistical test for heterogeneity (difference) of the
intervention effect between the selected
studies.
• A pooled estimate (e.g. RR or OR) and 95% CI for the
intervention effect after combining all
the trials, the statistical approach used depending on whether
or not statistical heterogeneity has been identified between
the selected studies.
• An hypothesis test for whether the intervention effect is
statistically significant or not.

18. Presentation of results: forest plot
As we saw in Table 9.1.3 (Table 1 of paper A) for the hygiene
interventions, the results of a
meta-analysis can be displayed in tables showing the results for
each individual study, the pooled summary estimate and CI, and
the results of the tests for heterogeneity and publication bias.
The estimates and CIs for each study and the pooled results should
also be illustrated pictorially in a forest plot. Figure 9.2.2 (Figure 2
from paper A) is a forest plot illustrating the results of the meta-
analysis of studies investigating the effect of household treatment
water quality interventions on diarrhoea. We will now look in more
detail at the information provided, and how this should be
interpreted.

19. Summary
• A systematic review is a review of the methods and results of all
individual studies designed to answer the same research question
and that conform to a set of pre-agreed criteria.
• Systematic reviews can provide essential information for the early
introduction of effective practice.
• It is important that reviews employ systematic methods in order to
establish whether research findings are consistent and
generalizable.
• The process of carrying out a systematic review comprises the
following stages:
(i) decide on the objectives of the review, which should be clear
and explicit;
(ii) define the inclusion and exclusion criteria, which should also be
clear and explicit; (iii) identify relevant studies, by carrying out an
effective and wide-ranging literature search that will minimize the
likelihood of publication bias;
(iv) assess the methodological quality of studies to be included in
the review using independent assessors;
(v) extract data using independent reviewers;
(vi) present results from the review in the form of a descriptive
summary of the selected
studies in a table, with text commentary.