1. Introduction to critical appraisal of
quantitative research for healthcare
Alan Fricker
Library Liaison Manager

2. Objectives of session
• Understand the main concepts of critical appraisal and its role
in evidence based practice
• Understand the different levels of quantitative evidence
• Know about resources available to help critically appraise
research

3. Daily Mail exercise
• Would you treat a patient based on this
article? Why?
• Validity
• Reliability
• Transferability to practice
(by Laurie Pink)

4. What is evidence based practice?
Evidence-based practice is the integration of
• individual clinical expertise
with the
• best available external clinical evidence
from systematic research
and
• patient’s values and expectations

5. The evidence-based practice (EBP) process
• Decision or question arising from a patient’s care
• Formulate a focused question
• Search for the best evidence
• Appraise the evidence
• Apply the evidence

6. EBP in practice
http://clinicalevidence.com/x/set/static/cms/efficacy-categorisations.html 21 Feb 2014

7. Why does evidence from research fail to
get into practice?
• 75% cannot understand the statistics
• 70% cannot critically appraise a research paper
Using research for Practice: a UK experience of the
barriers scale. Dunn, V. et al.

8. What is critical appraisal?
• “To weigh up the evidence critically to assess its validity
(closeness to the truth) and usefulness (clinical applicability)
“[Sackett and Haynes, 1995]
• “Critical appraisal is concerned
with the acquisition of necessary
skills with which to
discern clinical research
papers accurately”
[Ajentunmobi, 2002]

9. Why critical appraisal?
• “It usually comes as a surprise to students
to learn that some...published articles
belong in the bin and should not be used
to inform practice” (Greenhalgh, 2006)
• Not all papers are equal: some are good,
some are bad, most have strengths and
weaknesses

10. True or False? Critical appraisal is...
• An objective assessment of research
process and results
• Tearing research apart
• Assessment of a paper based on it’s
results
• A balanced evaluation of benefits and
strengths of the research against its flaws
and weaknesses
• Trying to prove the authors “wrong”
• A process that can only be undertaken by
experts and statisticians
• True
• False
• False
• True
• False
• False
10

11. How do I appraise?
• Step 1: Identify the research method used
– Quantitative or qualitative?
• Step 2: Identify the research methodology
and whether it is appropriate
– Systematic review? Clinical trial?
• Step 3: Checklists keep you focused
(CASP)
http://www.casp-uk.net/

12. CASP website
12

13. Appraising original research
• Rigour: are the results valid?
– Is the research question focused?
– Was the method appropriate?
– How was it conducted, e.g. randomisation,
blinding, recruitment and follow up?
• Results: what did the research find?
– How was data collected and analysed?
– Are they significant?
• Relevance: Will the results help my work with
patients?
13

14. Components of a research paper: IMRAD
• Introduction
– Why?
• Methods
– How?
• Results and
– What?
• Discussion
– Meaning

15. Step 1: Identify the research method
Quantitative Qualitative
Uses numbers to describe
and analyse
Uses words to describe and
analyse
Useful for finding precise
answers to defined
questions
Useful for finding detailed
information about people’s
attitudes / perceptions
Objective process Subjective process
Deductive reasoning Inductive reasoning
Statistical sampling Theoretical sampling

16. Quant or qual?
• How effective is outpatient pulmonary
rehabilitation in patients with COPD?
• Why might patients with COPD choose
not to receive outpatient pulmonary
rehabilitation?
• What is the optimum model for
transitions to palliative care in
hospitals?
• How are transitions to palliative care
perceived by hospital staff?
• Qualitative
 Qualitative
 Quantitative
 Quantitative

17. Step 2: Identify the methodology
• Clinical trial
– Whether one intervention is better than another
– Systematic review / meta-analysis measures overall
effect of several clinical trials
• Cohort study
– Measures effectiveness of intervention over time
• Case-controlled study
– Comparative: what makes some populations different
from others
• Survey / Case study
– How things are now (Crombie, 1996)

18. All studies are not created equal!
(but...it often depends on the question being asked)
Systematic reviews
Meta-analysis
Randomised
controlled trials
Cohort studies
Case-controlled studies
Surveys and case reports

19. Interventional studies
Systematic reviews, RCTs
• Pros:
– Provide compelling evidence
– Overcome confounding factors
• Cons:
– Difficult / time consuming to recruit
– Expensive
– Not appropriate for all conditions
19

20. Observational studies
• Cohort, case-control and case studies / series
• Pros:
– Easier to recruit
– Good for addressing questions of harm
• Cons:
– Confounding factors
• Age, existing diseases, body mass index, alcohol/tobacco
intake, high blood pressure, socio-economic status etc etc...
– Can provide circumstantial evidence, not
definitive proof of the causes of a disease
20

21. Step 3: Checklist – screening questions
• Question 1: Did the study ask a clearly
focused question? (PICO)
– Who is the population under study?
– What is the intervention / exposure?
– What is the outcome(s)
• Question 2: Was the study design
appropriate?
– Is it the most suitable one for addressing the
study question?

22. Randomisation
• Question 3: Randomisation
– Patients should be randomly allocated to
intervention / control groups
– Is the method described?
– Are the groups well-balanced?
• Baseline table or chart
– If the groups are not balanced is this
acknowledged and what steps have been taken to
overcome the problem?

23. Example of baseline table
From:
Farion, K J et al (2008) The
effect of vapocoolant spray
on pain due to intravenous
cannulation in children: a
randomized controlled trial
CMAJ vol 179(1) p31-6

24. Blinding
• Question 4: Blinding
– Preventing those involved in a trial from knowing to which
comparison group, i.e. experimental or control, a particular
participant belongs
• The risk of bias is minimised
• Participants, caregivers, outcome assessors and analysts
can all be blinded
– Single and double blinding are in common use
• Blinding of certain groups is not always possible
– e.g. Surgeons in surgical trials
• Placebo – inactive version of treatment

25. Follow-up, intention-to-treat
• Questions 5/6: Were all participants followed up and data
collected in the same way?
• Were any participants lost to follow-up?
– Flow diagram
• Intention-to-treat analyses
– Analysing people, at the end of the trial, in the groups to which
they were randomised, even if they did not receive the intended
intervention
– Prevents attrition bias: caused by patients withdrawing from a
trial
– Paper should specify if ITT was or was not used

26. Example of flow diagram
From:
Farion, K J et al (2008) The
effect of vapocoolant spray
on pain due to intravenous
cannulation in children: a
randomized controlled trial
CMAJ vol 179(1) p31-6

27. Sample size
• Question 7: Sample size
• Based on primary outcome measure
• Power calculation:
– Used to calculate the sample size necessary to detect a true
difference between outcomes in the control and intervention
groups
– Allows the researchers to work out how large a sample they will
need to use
– Power of 80-90% is standard
• For an example of how to calculate see
http://www.statisticalsolutions.net/pss_calc.php

28. Presentation of results
• Question 8: How the results are presented
• Results usually expressed in terms of likely harms or
benefits – can be relative or absolute
– Risk: proportion of people experiencing an outcome
– Measurement: mean or median difference
– Survival curves / hazards
• How large or meaningful is this result?
– Odds ratios, mean differences, absolute risk reduction (ARR),
relative risk reduction (RRR), number needed to treat (NNT)

29. A few facts about...risk
• When talking about the chance of something happening
e.g. death, hip fracture, we can talk about: risk and
relative risk
• Risk is the chances of a particular outcome being
observed in an individual
• Risk can be good or bad
– Risk of someone dying
– Risk of someone getting better
• Risk is a proportion and usually compares risks in an
experimental and control group

30. Expressing comparisons
of effectiveness: an example
You have responsibility for your Trust
budget and are looking at introducing a new
analgesic. Which one would you fund?
A: Reduces the rate of pain by 62.5%
B: Absolute risk reduction of pain of 50%
C: Decreases patients’ pain rate from 80% to 30%
D: For every 2 patients treated with the new drug, one
extra patient would expect a reduction in pain

31. Answer: They are all the same!
• 200 patients enrolled in study
• We give the new analgesic agent to 100 people and
find that 30 still have pain within two hours
• We give a placebo tablet containing no active drug to
another 100 people and observe 80 still have pain
• From this we can calculate the Experiment Event
Ratio (EER) and the Control Event Ratio (CER) – from
which we can calculate ARR, RRR and NNT

32. EER and CER
• Experimental Event Rate (EER) is the
proportion of patients in the experimental
group in whom an event (pain) is observed
EER = 30/100 = 30% (0.3)
(30% of people who received the drug still experienced pain)
• Control Event Rate (CER) is the proportion of
patients in the control group in whom an
event (pain) is observed.
CER = 80/100 = 80% (0.8)
(80% of people who received a placebo still experienced pain)

33. ARR & RRR
• Absolute Risk Reduction (ARR) is the
difference between the Control Event Rate
(CER) and the Experimental Event Rate
– ARR = CER – EER ARR = 80% – 30% = 50%
(can also be expressed as 0.5)
– The analgesic will prevent 50% of patients experiencing pain
that would have otherwise occurred
• Relative Risk Reduction (RRR) = ARR/CER
– RRR = 50/80 (62.5%) (or 0.625)
– Analgesic reduces risk of pain in 62.5% of the experimental
group compared to the control group
ARR!

34. NNT• The number of people you would need to
treat to see one additional occurrence of a
specific beneficial outcome
• Number needed to treat (NNT) is 100/ARR (if
ARR is a %)
 NNT = 100/50 = 2
 For every 2 patients treated with the new analgesia, one extra
patient would be expected to benefit
• Best NNT=1

35. Summary
• The same data can be presented in different
ways using very simple calculations
• How it is presented often depends upon
what the author is trying to convey
• Relative risk reduction (RRR) will be a bigger
number (more impressive!) than an
absolute risk reduction – use with caution
• ARR or NNT often give a better idea of how
likely a patient is to derive benefit from a
treatment

36. Two by Two tables
36

37. Calculations
37

38. Your go
(by marsmet552)

39. Precision / significance of results
• Question 9: How precise are the results?
• Are the results precise enough to make a
decision
– Did they take into account odds and the play of
chance?
– Are the results significant?
• Did they use confidence intervals?
• Did they use p-values?

40. Odds • Odds is the probability of an event occurring
compared to the probability of it not
occurring
• The odds of an event occurring in the
experimental group is:
– Number of people experiencing the outcome
event / Number of people not experiencing it
• Odds ratio
– Odds of people experiencing the outcome event /
Odds of people experiencing the control event
– Odds ratio of 1= no difference between groups

41. Essential statistics: P-value
• Measures probability
• Represents the probability that the result could have
occurred by chance if the null hypothesis was true
• Null hypothesis: that there are no differences between
groups being compared or no relationship/association
between variables in the relevant populations
• P-value of 0.05 or less = “statistically significant”
(likelihood of results being due to chance less than1 in 20)

42. 42

43. Essential statistics: confidence intervals
• Assesses significance of a given sample
• The range in which we are 95% (or 99%)
confident that the ‘real’ result of the study lies
when is extrapolated to the whole of the
population sampled in the study
• 95% confidence interval (CI) = “clinically
significant”
– Narrow CI (1.07,1.24) = more confident
– Wide CI (0.15, 59.89) = less confident

44. CIs and statistical significance
• When quoted alongside an absolute difference a CI that
includes zero is statistically non-significant
• When quoted alongside a ratio or relative difference a CI
that includes one is statistically non-significant
• IF statistically significant:
• Less than zero (or one) = less of the outcome in the
treatment group.
• More than zero (or one) = more of the outcome.
• BUT - Is the outcome good or bad?

45. Cardiac deaths - less = good

46. Smoking cessation – more = good

47. Does statistical significance
matter?
• Statistical significance does not necessarily equal clinical
significance
• Type I error: Seeing effects that are not real (P<0.05)
• Type 2 error: Seeing no effect when there is one (power >
80%)
• Research papers should not contain too many statistical
comparisons – increases chance of spurious findings
• Don’t get bogged down with stats – they are only one
part of the paper!
http://www.medicine.ox.ac.uk/bandolier/painres/download/whatis/what_are_conf_inter.pdf

48. Transferability
• Question 10: Transferability
– Can I apply these results to my own practice?
• Group under study: are they the same as your
patients?
– Socio-cultural origin, gender, age etc.
– Location of research: country, setting
• Does the paper consider the results from the
perspective of difference stakeholders?
• Do the benefits outweigh the harms / costs?
• Should practice / policy change?

49. Be aware of potential bias
• Statistical bias
– Absence of baseline
– Lack of randomisation (selection bias)
– Poor blinding (observer bias)
• Population bias
– Study focuses on one particular group
• Loss to follow-up
http://www.medicine.ox.ac.uk/bandolier/Extraforbando/Bias.pdf

50. Ethics
• Has the research received ethical approval via an ethics
committee?
• Who carried out / funded the research?
– Any conflicting interests must be declared
• Is the data anonymised? Did patients given written
consent to treatment?
• Evidence of attention to ethical issues (Dawes, 2005)

51. The good news!
• Some resources have already been critically
appraised for you.
• An increasing number of guidelines and
summaries of appraised evidence are
available on the internet.

52. Summary
• Search for resources that have already been appraised first, e.g.
Guidelines, Cochrane systematic reviews.
• Search down through levels of evidence, e.g. systematic reviews,
RCTs.
• Use checklists to appraise research.
• Look out for the following:
– Research design
– Choice of study
– Statistics explained and displayed clearly
– Bias / Ethics considered
– Transferable results

53. Tips for success
• Group work
• Read the paper
• Start with easy questions
• Keep calm and carry on
• Review, feedback
“Undertaking a critical appraisal is really using
your everyday skills, and applying them in a more
structured and systematic way” - Dawes (2005)

54. References
• Ajetunmobi, O (2001) Making sense of critical appraisal
Arnold, London
• Bowers, D et al (2001) Understanding clinical papers John
Wiley & Sons, Chichester
• Bowling, A (2009) Research methods in health (3rd ed)
Open University Press, Buckingham
• Chalmers, I and Altman, D G (1995) Systematic reviews
BMJ Publishing, London
• Crombie, I K (1996) The pocket guide to critical appraisal
BMJ Books, London
• Dawes, M (2005) Evidence-based practice: a primer for
healthcare professionals (2nd ed) Elsevier, Edinburgh

55. References
• Everitt, B S (2003) Medical statistics from A-Z Cambridge University
Press, Cambridge
• Greenhalgh, T (2014) How to read a paper: the basics of evidence-
based medicine (5th ed) BMJ Books, London
• Hart, A (2001) Making sense of statistics in healthcare Radcliffe
Medical Press, Abingdon
• Khan, K S et al (2003) Systematic reviews to support evidence-based
medicine RSM Press, London
• Pereira-Maxwell, F (2005) A-Z of medical statistics: a companion for
critical appraisal Arnold, London
• Stewart, A (2002) Basic statistics and epidemiology Radcliffe Medical
Press, Abingdon

56. Useful websites
• Bandolier http://www.medicine.ox.ac.uk/bandolier/
• Centre for Evidence-based Medicine http://www.cebm.net
• Cochrane Library http://www.thecochranelibrary.com
• Critical Appraisal Skills Programme (CASP) http://www.casp-
uk.net/
• Users Guide to the Medical Literature (JAMA) See list of articles
http://bit.ly/txAejQ