The document discusses tools for assessing risk of bias in systematic reviews. It outlines the development of the Cochrane Risk of Bias tool, which evaluates randomized controlled trials across 7 domains related to selection bias, performance bias, detection bias, attrition bias, and reporting bias. The tool aims to provide a standardized approach for assessing bias in a way that is transparent and focuses on biases rather than other quality issues. It has informed the risk of bias assessment approach used in Cochrane systematic reviews.

1. Tools for Assessing Risk of Bias
Doug Altman
Centre for Statistics in Medicine
University of Oxford

2. Acknowledgement
 Cochrane Collaboration Bias Methods Group

3. Systematic review
 A systematic review attempts to collate all empirical
evidence that fits pre-specified eligibility criteria to
answer a specific research question
 It uses explicit, systematic methods that are
selected with a view to minimising bias, thus
providing reliable findings from which conclusions
can be drawn and decisions made
3

4. Key characteristics of a systematic
review
 A clearly stated set of objectives with an explicit,
reproducible methodology
 A systematic search that attempts to identify all
studies that would meet the eligibility criteria
 An assessment of the validity of the findings of the
included studies, such as through the assessment of
risk of bias
 Systematic presentation and synthesis of the
characteristics and findings of the included studies
[Liberati et al, BMJ 2007]
4

5. Taking account of study quality
 Methodological quality of the primary studies may
affect their results
 The influence of study quality should routinely be
examined
– How?
 Quality is difficult to define
– design, conduct and analysis of a trial?
– clinical relevance?
– quality of reporting?
– all of these?
Focus on Randomised Controlled Trials (RCTs)
5

6. Quality scales and checklists
 Many exist (>40)
– vary hugely in complexity and scope
 Many combine trial quality with reporting quality
and other aspects (e.g. sample size)
 Very popular: Jadad et al 1996
– Simple score from 0 to 5
– Serious flaws: combines methodology and reporting
 Comparison of high vs low quality trials depends
– Results depend strongly on which scale is used
[Jüni et al, JAMA 1999]
6

7. The Cochrane Collaboration aims to help people
make well-informed decisions about healthcare by
preparing, maintaining and promoting the
accessibility of systematic reviews of the effects of
healthcare interventions
(and diagnostic test accuracy) 7

8. Cochrane library
8

9. Rethinking the approach to assessing
study quality
 2005: Cochrane Collaboration initiative
 Rationale: quality scores are not helpful
 Focus should be on aspects with potential for bias,
such as
– concealment of treatment allocation
– blinding of outcome assessment
– handling of patient attrition
9

10. Cochrane Risk of Bias (RoB) tool
 The Risk of Bias (RoB) tool was developed over
several years as a way to address concerns
about the existing method of assessing study
“quality” in Cochrane reviews
 Based on empirical evidence where possible
 It is intended to lead to uniform approach
across reviews from all Cochrane review groups
10

11. Developing a Risk of bias tool
 In May 2005, 16 statisticians, epidemiologists, and
review authors attended a three day meeting to
develop the new tool
 Identified items that were truly potential biases
rather than sources of heterogeneity or imprecision
 Seven areas were discussed leading to a proposed set
of criteria for assessing protection from bias
– as adequate, inadequate, or unclear
11

12. The Risk of Bias Tool
12
http://www.cochrane-handbook.org/

13. 13
BMJ 2011;343:d5928

14. Risk of bias
General principle of the RoB tool
 Avoidance of bias
Assessment of risk of specific types of bias

 Selection bias
 Performance bias
 Detection bias
 Attrition bias
Risk of bias summary
14

15. 15
Participants
Treatment A Comparator B
Outcome
Assessment
Outcome
Assessment
Detection bias
Selection bias
Attrition bias
Bias
Performance bias

16. 16
Population
Treatment A Comparator B
Outcome
Assessment
Outcome
Assessment
Detection bias
Selection bias
Attrition bias
Bias
Performance bias
Randomisation
Blinding of
participants and
personnel
Blinding of
outcome assessors

17. Domains of the Cochrane
“Risk of Bias” tool
Review authors’ judgement:
 Was the allocation sequence adequately generated?
 Was allocation adequately concealed?
 Was knowledge of the allocated intervention
adequately prevented during the study?
– Participants
– Trial personnel
 Were incomplete outcome data adequately addressed?
 Are reports of the study free of suggestion of selective
outcome reporting?
 Was the study apparently free of other problems that
could put it at a high risk of bias?
17

18. Domains of the Cochrane
“Risk of Bias” tool
Review authors’ judgement:
 Was the allocation sequence adequately generated?
 Was allocation adequately concealed?
 Was knowledge of the allocated intervention
adequately prevented during the study?
– Participants
– Trial personnel
 Were incomplete outcome data adequately addressed?
 Are reports of the study free of suggestion of selective
outcome reporting?
 Was the study apparently free of other problems that
could put it at a high risk of bias?
18
Separate assessment for each
outcome

19. The “Risk of Bias tool” (RoB)
General principles
19
 2 steps
 What was reported
 Extracted text: what was reported in the published report /
protocol/ contact with authors
 Commentary
 Judgment relating to the risk of bias
 Low risk of bias
 High risk of bias
 Unclear (judgment is impossible)

20. The “Risk of Bias tool” (RoB)
General principles. What was reported?
Sequence generation. Low
risk
Quote: “patients were randomly allocated”.
Comment: Probably done, since earlier reports
from the same investigators clearly describe
use of random sequences (Cartwright 1980).
Blinding of
participants and
personnel
(performance bias)
Low
risk
Quote: “double blind, double dummy”; “High
and low dose tablets or capsules were
indistinguishable in all aspects of their
outward appearance. For each drug an
identically matched placebo was available”.
Comment: Probably done
Blinding of outcome
assessment
(detection bias)
(Mortality)
Low
risk
Quote: “Obtained from medical records”
Comment: review authors do not believe this
will introduce bias.
etc
20

21. The “Risk of Bias tool” (RoB)
General principles. Judgment
 High risk of bias
 Bias of sufficient magnitude to have a notable
impact on the results
 Unclear risk of bias
 Insufficient details reported
 Appropriate reporting, but the risk of bias is
unknown
21

22. Attrition bias: Considerations
 How much data is missing from each group?
 Why is data missing in each group?
 How were data analysed?
 Handling of incomplete outcome data
22

23. Attrition bias
Low risk of bias
• No missing outcome data
• Reasons for missing data not related to outcome
• Missing data balanced across groups, with similar
reasons
• Missing data not enough to have a clinically relevant
impact on the intervention effect estimate
• Missing data have been imputed using appropriate
methods
23

24. Attrition bias
High risk of bias
• Reason for missing data related to outcome, with
either imbalance in numbers or reasons
• Missing data enough to induce clinically relevant bias
in intervention effect estimate
• ‘As-treated’ analysis with substantial departure of
the intervention received from that assigned at
randomization
• Inappropriate use of imputation
24

25. The “Risk of Bias tool” (RoB)
General principles
 Reviewers specifically trained
 Independent duplicate assessment with consensus
 Decisions need to be pre-specified in the protocol
 Classification of outcomes (subjective / objective)
 Blinding: successful blinding procedure
 Missing data
 Other risk of bias
 Contact authors for missing information
25

26. Selective outcome reporting
 Concern is that there is a tendency to selectively
report outcomes that are statistically significant
– Empirical evidence [Kirkham et al, BMJ 2010]
 Challenges in detecting such behaviour
 Ideally need study protocol
 May be internal evidence from publications or trials
register
 Usually have to assess this item as “Unclear”
26

27. Risk of bias summary – each study
27

28. Risk of bias summary – all studies
28

29. Summary assessment of the risk of bias
across trials
Classification
Low risk of bias Most information is from trials at low risk of bias
Unclear risk of bias Most information is from trials at low or unclear risk
of bias
High risk of bias The proportion of information from trials at high risk
of bias is sufficient to affect the interpretation of
results
29

30. Risk of bias for non-randomised studies
Nonrandomised studies of interventions:
 Many issues are the same
 Main concern is selection bias – how were treatment
chosen?
 High risk of non-comparable groups
 Some tools exist – e.g. “Newcastle-Ottawa”
 Cochrane Collaboration is planning to develop a RoB
tool
Other observational studies – e.g. prognosis,
aetiology:
 Parallel developments are needed30

31. Conclusions
 Assessing the risk of bias is an essential step for an
appropriate interpretation of systematic reviews and
meta-analysis
 Cochrane Risk of Bias tool includes 7 items to be
evaluated for RCTs
 Need for transparency
 Training and use of the handbook recommendations
 Similar approach should be adopted for other study types
31

32. Closing comments
 Good systematic reviews are a major source of
reliable information on the benefits and harms of
health care interventions
 A key aspect is assessment of risk of bias of the
primary studies
 Cochrane Risk of Bias tool includes 7 items to be
evaluated for RCTs
– A similar approach is needed for other study types
 Should take account of risk of bias in the analysis
and interpretation
32

33. 33