An Introduction to the NIHR Complex Reviews Support Unit (CRSU) Professor Keith Abrams

1. NIHR Complex Reviews Support Unit
(CRSU) – An Introduction
Keith R Abrams, PhD CStat
Department of Health Sciences,
University of Leicester, UK
on behalf of CRSU
Department of Health Disclaimer: The views and opinions expressed herein are those of the
authors and do not necessarily reflect those of NIHR, NHS or the Department of Health.

2. Outline
• Background & Objectives of CRSU
• First 18 months of CRSU
– Working with Cochrane Review Groups
– Workshops & Training events
– Development Work & Issues
• Meta-analysis of DTA data
• Network meta-analysis (NMA)

3. Background – 1
• Why are reviews increasingly complex?
– Increasingly complex clinical and important policy
questions
– More interest in complex interventions
– Existing evidence is often limited and
heterogeneous
– Multiple treatment/intervention options with no (or
limited) head-to-head evidence
– Outcomes of interest have complex data structure

4. Background – 2
• However, the lack of methodological expertise and
researcher capacity in this area is a recognised major
barrier to completing such complex reviews.
• In response to this, the National Institute for Health
Research Complex Reviews Support Unit (NIHR CRSU)
was set up in July 2015 …
• to support and encourage successful delivery of complex
reviews of importance to the UK National Health Service
(NHS) and …
• to contribute to building capacity and capability within the
research community.

5. Objectives – 1
• The primary objective of the unit is to build a
successful working relationship with NIHR in
supporting the UK NHS in delivering clinically
and cost-effective services that are evidence-
based.
• The CRSU focuses on providing timely and
appropriate support for the delivery of
complex reviews that are funded and/or
supported by NIHR.

6. Objectives – 2
• These include Cochrane reviews, reviews
funded by the Systematic Review
Programme and other NIHR programmes,
and other NHS and NHS supported sources.
• The unit will also work closely with NIHR to
support scoping and prioritising of future
complex reviews.

7. CRSU members …
• University of Glasgow
– Olivia Wu (Director)
– Neil Hawkins (Deputy
Director)
– Moira Aitken (Project
Manager)
– David Stott
– Hilary Thomson
– Mhairi Mackenzie
– Peter Langhorne
– Terence Quinn
• London School of Hygiene &
Tropical Medicine (LSHTM)
– Richard Grieve
• University of Leicester
– Nicola Cooper (Deputy
Director)
– Alex Sutton
– Keith Abrams
– Suzanne Freeman
– Rhiannon Owen
• Advisory/Governance
Committee
– Ken Stein (Chair)
– Steve Palmer
– Nicky Welton
– David Tovey
– Sally Bailey

8. Expertise
Within CRSU, the key areas of expertise include:
• Diagnostic Test Accuracy (DTA) reviews
• Network Meta-Analysis (NMA)
• Individual Participant Data (IPD) meta-analysis
• Economic evaluation
• Realist synthesis
• Narrative synthesis of quantitative & qualitative data
• Use of routine data and/or non-randomised studies
• Prevalence & Prognostic reviews
• Time-to-event/survival outcomes

9. First 18 months …
• Working with Cochrane Review Groups
• Workshops & Training events
• Development work

10. Cochrane Reviews – 1
The CRSU has successfully supported the
following Cochrane Review Groups so far:
• Dementia Group – building on an existing
and recent Cochrane review on diagnosing
dementia in stroke patients, the CRSU is
supporting the development of a proposal to
demonstrate the potential added-value of the
more complex approaches to diagnostic test
evaluation, beyond those currently
considered by the Cochrane Collaboration
(see later).

11. Cochrane Reviews – 2
• Gynaecological, neuro-oncological and
orphan cancers Group – provided advice on
network meta-analysis of treatments for
metastatic brain tumours.
• Programme of 30 reviews on upper
digestive disorder at University College
London – provided advice on network meta-
analysis for multiple reviews.
• Heart Group and Airways Group – provided
comments and advice on protocols

12. Cochrane Reviews – 3
CRSU is currently providing support to
Cochrane Review Groups applying for awards
from the NIHR Incentive Scheme and
Programme Grants, including;
• Cochrane Injuries
• Cochrane Tobacco Addiction
• Cochrane ENT
• Cochrane Eyes and Vision
• Cochrane Gynaecological, Neuro-oncology & Orphan
Cancers (GNOC)
• Cochrane Oral Health
• Cochrane Airways

13. Workshops & Training
• Workshops at Cochrane UK & Ireland Symposium 2016:
– Methodological challenges in complex reviews
– NIHR Systematic Reviews Programme: opportunities for
greater impact
• Workshops at Cochrane UK & Ireland Symposium 2017:
– Can Cochrane Reviews take a more active role in
informing the design of future trials?
– Examples of collaborations with Cochrane Review
Groups: assessment of complex interventions, test
accuracy and network meta-analyses.
• NICE Centre for Guidelines:
– 1-day workshop on Introduction to Systematic Reviews
& Meta-Analysis of Diagnostic Test Accuracy (DTA) data

14. Development Work & Issues
• DTA Reviews
– Raise awareness of the challenges of conducting
diagnostic test accuracy (DTA) reviews and offer
potential (simple to more complex) solutions to some
but not all of the challenges (e.g. multiple thresholds)
– Provoke discussion regarding how to ensure reviews
of diagnostic tests answer clinically-relevant
questions

15. Sensitivity vs. Specificity
pdf
Diagnostic variable, D
Group 0
(Healthy)
Group 1
(Diseased)
TP
TN
Group 1
Diseased
Group 0
Healthy
Test + TP FP
Test - FN TN
DT
Test +Test -
Threshold
Sensitivity = number of true positives/total with disease
Specificity = number of true negatives/total without disease

16. Receiver Operating Characteristic (ROC)
Curve: Selecting the Threshold
T
1 - specificity (False positive rate)
Sensitivity(Truepositiverate)
45o line = random guess
Perfect
classification
Lower
threshold
Higher
threshold
Group 1
Diseased
Group 0
Healthy
Test + TP FP
Test - FN TN
0
1
0
1
Point T gives Max. accuracy
threshold BUT
ignores relative opportunity
costs of FP and FN results

17. Challenges of meta-analysing DTA data
• More complex than for effectiveness data due to:
– Two dependent outcomes – sensitivity and
specificity
– Variable & multiple test threshold levels (either
explicit or implicit) between & within studies
– Different reference tests (imperfect gold standard)
• Other issues include:
– Different populations and/or study conduct
(leading to between-study heterogeneity)
– Data quality & risk of bias

18. Naïve analysis – pooling sensitivity &
specificity separately

19. 0
.1.2.3.4.5.6.7.8.9
1
Sensitivity
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Specificity
Two dependent outcomes
- Sensitivity and Specificity
• Requires a meta-analysis model that models sensitivity,
specificity and their correlation simultaneously
Hierarchical sROC:
sROC curve, 95% credible &
95% prediction region
0
.1.2.3.4.5.6.7.8.9
1
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Specificity
Bivariate: Point estimate,
95% credible & 95%
prediction region for
sensitivity and specificity
• Statistical models are equivalent although
presentation of results are different

20. Challenges of meta-analysing DTA data
• More complex than for effectiveness data due to:
– Two dependent outcomes – sensitivity and
specificity 
– Variable & multiple test threshold levels (either
explicit or implicit) between & within studies 
– BUT data on test threshold in primary studies (if
known) often ignored
– Different reference tests (imperfect gold standard) 
• Other issues include:
– Different populations and/or study conduct (leading
to between-study heterogeneity) 
– Data quality & risk of bias 
– Limited by the data & #studies

21. Discussion points
• Whilst solutions exist, meta-analysis of DTA data is more
complex and requires an understanding of the principles
underlying the different approaches/methods …
– http://methods.cochrane.org/sdt/handbook-dta-reviews
– 1-day workshop on Introduction to Systematic Reviews & Meta-
Analysis of Diagnostic Test Accuracy (DTA) data
• … But also raises other (even more complex) issues for
example when there are multiple candidate tests and NHS
needs to make a decision as to which tests should be
used/funded.
• From a NICE perspective, could argue that only makes
sense to consider tests & interventions, e.g. many oncology
drugs now come with a companion diagnostic (test)

22. Development Work & Issues
• DTA Reviews
– Raise awareness of the challenges of conducting
diagnostic test accuracy (DTA) reviews and offer
potential (simple to more complex) solutions to some
but not all of the challenges (e.g. multiple thresholds)
– Provoke discussion regarding how to ensure reviews
of diagnostic tests answer clinically-relevant
questions
• Network meta-analysis (NMA)

23. Early thrombolysis for AMI (Caldwell &
Higgins BMJ 2005):
NMA considers the
network of all
relevant evidence
to a decision
problem
NMA methods allow
comparison/effect
estimates for all
interventions &
associated
uncertainty

24. Assumptions in NMA
• Similarity
– Trials are clinically and methodologically similar and
comparable
• Exchangeability
– If patients in one trial were substituted in another, the
observed treatment estimates would be expected to be
the same (allowing for random variation)
• Transitivity
– dAB = dAC − dBC & dAC = dAB – dCB
• Consistency
– Indirect and direct estimates are consistent

25. Discussion points
• Can we add value to existing reviews using network
meta-analysis?
• Can this be readily incorporated in your current reviews?
• Is network meta-analysis within the remit of Cochrane
reviews?
• Cochrane Methods – Comparing Multiple
Interventions
– http://methods.cochrane.org/cmi/comparing-multiple-
interventions-cochrane-reviews
• But methods can be complex & have to be implemented
with care, and therefore there is a capacity & training
issue.

26. Development Work & Issues
• DTA Reviews
– Raise awareness of the challenges of conducting
diagnostic test accuracy (DTA) reviews and offer
potential (simple to more complex) solutions to some
but not all of the challenges (e.g. multiple thresholds)
– Provoke discussion regarding how to ensure reviews of
diagnostic tests answer clinically-relevant questions
• Network meta-analysis (NMA)
• Methods for synthesising time-to-event data to inform
estimates of comparative effectiveness and decision
models
• Complex Interventions

27. Getting in touch & resources …
www.nihrcrsu.org
Follow us at
@NIHRCRSU
Google:
“NIHR CRSU”

28. Thank you
Any Questions?