1. How to conduct health technology assessment
using Gradepro
Workshop at HTAi, 2017, Rome, Italy
Arindam Basu
University of Canterbury School of Health sciences,
arindam.basu@canterbury.ac.nz

2. Objectives
• Mix the concepts of GRADE and Health Technology Assessments
• Introduce Gradepro
• Rework an example of HTA through Gradepro
1

3. Concepts of Health Technology Assessment
• HTA is Policy focused
• Identify and appraise technology and their impacts
• HTA studies Individual, as well as population based outcomes
• HTA is about both desired and adverse outcomes
• Synthesise information to enable policy and regulatory approaches
2

4. How to do HTA: Synthesize Evidence
• Primary Studies
• Secondary Data Analysis
• Meta Analysis and Systematic Reviews
3

5. Primary Studies
• Case series
• Cross sectional surveys
• Case control and Cohort Studies
• Randomised Controlled Trials
• Other types of Trials
4

6. Secondary Data Analysis
• Analysis of Registry based data
• Analysis of other databases
5

7. Meta Analyses and Systematic Reviews
• Synthesis of primary studies
• Studies are pooled together
• Emphasis on studies across outcomes
• Primary analysis units are studies
6

8. Steps of Meta analysis and SRs
• Frame a question
• Search and retrieve relevant literature
• Identify whether the studies are homogeneous
• Pool the results of the studies
• Conduct subgroup analyses and meta regression
• Test for publication bias
• Arrive at conclusive evidence
7

9. What is GRADE and GRADEpro?
• GRADE is a tool that helps to manage evidence
• Evidence Portfolio = Quality of Evidence + Summary of Findings
• Use the Evidence Portolio to develop Recommendations
8

10. Conceptual Diagram of GRADE
Figure 1: Concept of GRADE, GRADEpro implements this
9

11. Features of GRADE way of doing things
• We work on outcomes across studies
• Quality of evidence is for pairs of outcomes and interventions
• We also use an intervention and an alternative (placebo?)
10

12. How to do GRADE: Start with the outcome
• Select the outcome
• Assign a score to the outcome from 0-9
• 0-3 = not important, 4-6=important, 7-9 = critical
• Build a consensus among the team to assign importance scores
• For each outcome, assimilate studies
11

13. Type of studies to consider
• Intervention studies
• Diagnostic studies
• Controlled Trials
• Cohort Studies
• Studies with an intervention and a control group
• Other types of studies (Observational Studies)
12

14. How do we decide if we should recommend this?
• What is the overall quality?
• How much is the effect?
• What is the importance of the outcome?
• Consider desired and adverse outcomes
13

15. Steps of constructing the Guideline with Grade
• Set up an evidence profile
• Evidence Profile = Appraise Quality of Evidence + Summary of
Findings
• Assess the Summary of Findings for the body of evidence
• Assign an importance score
• Assimilate these information for recommendation
14

16. Appraisal of the Quality of Evidence
• Star rating system
• 4 pluses = Very high,3 pluses = High, 2 pluses = Moderate, 1 plus
= Low
• Very high = We have very high confidence that the results are true
effects and additional studies will add little
• High = We are confident that the results are true effects and further
studies
• Moderate = Additional studies may be needed
• Low = Our confidence in the evidence is low and we need further
evidence
15

17. Steps of Awarding quality scores to evidence
• We start with the study design
• If RCT or Experimental Study designs, we assign 4 +
• If Observational Study designs, we assign 3 + to start with
• We assign penalties (take out one or two +s) on 4 counts
• We award points (add one or two +s) on 3 counts
• If it is a meta analysis of RCTs, then we start with 4+s
• If it is a systematic review or meta analysis of Observational study
designs, we start with 3 +s
16

18. Take out one or two +s if:
• Findings are Inconsistent
• Measurement of the intervention or outcome is Indirect
• Measurement of Effect Estimate is Imprecise
• Detect risk of bias
17

19. Inconsistency of the findings
• Not an issue if you deal with single study
• If two or more studies pooled together:
• Check the I squared or Cochran’s Q
• Cochran’s Q is based on Fixed Effects Model and Chi-square test
• If the p-value for Cochran’s Q is < 0.05, suspect heterogeneity
• If the I-square value is less than 40%, suspect heterogeneity
• If you suspect heterogeneity, check original studies
18

20. Indirectness of measurement
• Proxy measures
• Biomarkers
• Surrogate Measures
• Ask: Are these reliable and accurate measures?
• Example: Pain - Visual Analog Scale
• Control of Diabetes: HbA1c
19

21. Imprecision
• What is the point estimate?
• What is the 95% CI around the point estimate?
• Does the 95% CI straddle the null value?
• Relative Risk versus Absolute Risk
• Is the study or the pooled studies underpowered?
20

22. Risk of Bias
• Were the compared groups similar at the beginning of the study?
• Did the authors use intention to treat analysis?
• How did the authors select participants for observational studies?
• How did the authors measure outcomes for observational studies?
• Did they report publication bias for meta analysis?
21

23. Publication Bias
• Large studies with effects in the desired directions
• Small studies with equivocal findings
• Fugitive literature
• File Drawer effect
• Funnel Plot in Meta analysis
22

24. Funnel plot
Figure 2: Funnel Plot, ES = Effect Size, Standard Error is an indication of the
sample size, the vertical line is the summary estimate line
23

25. Rules to add points
• If you find that the effect size is large enough
• If you find the authors adjusted for all plausible confounding variables
• If you find authors reported dose response effect
24

26. How large is large enough?
• RR or OR 3.0 or higher
• Absolute Risk Reduction large
• Small NNT
• For Diagnostic studies, check Diagnostic Odds Ratio
25

27. Did the authors adjust or control for confounding?
• Did they report results of multivariate analyses?
• Did they use matching or restriction to control for confounding?
• What did they do for random selection for RCTs?
26

28. How do we balance the evidence for recommendation?
• Magnitude of the effect
• Quality Appraisal Score
• Importance of the Outcome
• Is the outcome beneficial or harmful?
27

29. Scenarios for recommendation
• Large effect, very high quality, critical outcome, beneficial:
recommended
• Small effect, low quality of evidence, unimportant outcome, and
harmful: not recommended
• Also, large effect, high quality of evidence, important but harmful
outcome: not recommended
• Other factors need to be considered: feasibility, and resources
28

30. Gradepro step by step
• Get an account, go to http://gradepro.org
• If you use Google Chrome, use their app
• If you use other browsers, bookmark the site for offline use
(Command-d for Mac or Ctrl-d) for other computers
• Log in and name the project
• It will take a while when you log in
29

31. Gradepro step by step: screenshot
Figure 3: Screenshot of Gradepro Software or Webapp
30

32. What we will study
• We will study the technology “Radiofrequency denervation (RFD)”
• We will study the outcome “Low back pain”
• We will try to determine if we can recommend RFD for Low back
pain
31

33. Here is the technology of RFD
Figure 4: RFD image
32

34. Here is the study on which we will work
Figure 5: Screenshot of the article we will study
33

35. Download the full text of the article and save it
• Click Me to Download or view the Systematic Review for the HTA
• Reference information:
Piso B, Reinsperger I, Rosian K. Radiofrequency denervation
for sacroiliac and facet joint pain. Decision Support Document
No. 99; 2016. Vienna: Ludwig Boltzmann Institute for Health
Technology Assessment.
34

36. Step 1: Start a New Project
• We will click on the box to start a new project
• Type the name of the new project
• We will type: “Radiofrequency denervation for Low Back Pain”
• We will next select: “Evidence Profile” for the type of project
• Hit Apply and it will bring up the next window
35

37. Step 2: Activate the panels
• We will activate all the side panels
• Click on the “Cog like” structure on the top of the window
• Click all the boxes
36

38. Step 3: Understand the different modules
• On the side panel, you will see nine icons
• Evidence to Decision making templates: decide how you will apply
this evidence
• Add Tasks for your team or yourself (productivity)
• People icon: add team members as these things are team based
activity
• Write the scope of the work in details
• Bookmark looking icon: add or import references for the project
• Curved arrow: add a prognosis question if you want
• Balance icon: Comparison (this is where we will spend time today)
• Document icon: You can draft the document
• Another document icon: you can either create an app or publish it
to the web
37

39. Step 4: Fill in the scoping document
• Add the title
• Write the purpose of this assessment
• Write the target population, here our target population are
individuals with low back pain or facet pain and those who qualify
for RFD
• Healthcare setting: “Tertiary Care setting”
• Types of Interventions: “RFD”, if you want to write in details, you
can
• Key stakeholders and users: “doctors, patients, payers, health
technology professionals”
• Existing documents: “All documents that will contribute to this
technology assessment”
38

40. Step 5: Add Question
• Add either Management or Diagnostic Question
• This is confined to comparison of two approaches only
• If RFD versus Placebo, then that’s one
• RFD versus standard treatment should be another
• These would be two different management questions
• You can add two questions
• You can add more, depending on the outcomes as well
• Skip the outcomes module at the moment
39

41. Step 6: Add References
• Add manually
• If you have an Endnote XML format, you can import file
40

42. Step 7: Click Comparison to Add Management Question
• “Radiofrequency denervation” vs
• “Placebo” for
• “Low back pain”
• Click on the “save” icon to save the question
• You can add as many questions as you want
• If we had a team that approved our questions, then they would show
up here automatically, here we have to add them manually
• Setting: “Private Practice”
• If we have a list of articles to work from we add it here
• Then we click on the save icon to save the question
• You can drag the question to a Group so that if you have many
groups e.g., for low back pain relief or quality of life, you can add
many questions to these groups; depending on your concepts
41

43. Step 8: Add Outcomes
• Click on the question
• This will bring up the Quality Assessment and Summary of Findings
panels
• Click on Add Outcome to Add Outcome Manually
• If you have another GRADEpro project or if you use Revman5, the
Cochrane Collaboration Meta Analysis, you can import outcomes
from these sources
• You can add as many outcomes as you want, but Gradepro authors
insist at most adding seven outcomes (I do not know the reason
why)
42

44. Step 9: Fill in the Outcome details Part I
• Name the outcome: “Pain Relief”
• Short Name: “Pain Relief”, if you have longer name, this is the
place to make it short
• Assessed or Measured with: “Visual Analog Scale”
• Length of Follow up: this is optional
43

45. Filling the length of Follow Up
• Gradepro assumes that you will only include follow up studies, either
RCTs or Cohort Studies or Other follow up studies
• You can add a single study or you can add a systematic review
• You will work on the basis of outcomes, not on the basis of
individual studies or collected group of studies
• The last point is important
• To keep things simple, for our rapid meta-analysis based appraisal,
we leave this blank
44

46. Step 10: Fill in the type of outcome
• As this is for a single outcome, you have three choices - your
outcome can be dichotomous, that is binary, yes or no
• Dead/Alive, Recovered/Not, etc
• It can be continuous, measured on a scale (it does not get finer in
details than this)
• It can be narrative for those cases where you do not have a number
to report – qualitative
• For us, we tick: “continuous”
• Next six radio buttons: “pooled” if you deal with meta analysis,
“not pooled” if you deal with many studies and you decide you will
use Median or Mean or other measures yourself rather than
statistical pooling, and the rest of the buttons are self explanatory
• We select: “pooled” as we will work on the basis of this
meta-analysis
45

47. Step 11: Fill the quality assessment box first
• Number of studies: depends on the outcome you study as even
within a meta-analysis, you will see varying number of studies pooled
together for a particular outcome
• For us: “6”; refer to page 10 of the research paper
• Study Design: refers to the type of study on which you base your
evidence synthesis. Here, in the meta-analysis, we dealt with RCTs
• Risk of Bias: If you work with meta-analysis, you will need to check
whether the authors reported the risk of bias of individual studies on
which the report is based. If they have done, OK, else, you will have
to do them yourself
• for us: “Half of the trials had high risk of bias”, so we will say,
“serious”
• It will ask you to add an explanation: add explanations but restrict
to 1000 characters
46

48. Step 12: Quality Assessment: Inconsistency
• If a single study, “not serious”, if not:
• for meta-analysis, check for heterogeneity statistics
• Rule of thumb: heterogeneity measured by Q-statistics or I-square
• If I-square is 0% or less than 30%, inconsistency is not serious; if
not, read the original papers to decide for yourself
• If it is not reported, you can conduct your own statistical procedures,
or state, “serious”
47

49. Step 13: Judge whether the evidence is direct or indirect
• Decide whether the population of the study or studies is directly
relevant to the population you will apply to
48

50. Step 14: Decide if the evidence is imprecise
• Check the point estimate
• Check the 95% Confidence Interval around the point estimate
• If they traverse the null, then they are imprecise
49

51. Step 15: Decide if there are other issues
• Is there publication bias?
• Check if the authors have included a funnel plot
• If they haven’t, then you will need to create your own funnel plot
• Would confounding variables reduce the effect estimate?
50

52. Step 16: Fill in the Summary of Findings
• Total number of participants in each arm
• The effect estimate
• Quality Score is already filled in
• Fill in the level of importance of this particular outcome
• For us: we will tabulate results till the one month of follow up
51

53. Step 17: Summary of Findings, part II
• Number of patients in the treatment arm:
• Number of patients in the placebo arm:
• As the measures were continuous, hence the programme selects
“Absolute Measures of Risk”
• Estimate of the Effect: “MD” of -1.47
• 95%CI confidence limits from -2.28 to 0.67
52

54. Step 18: Decide the importance
• If team based, it will be your team’s decision
• If solo, you decide
• Count the votes, and there are other methods where you can decide
and give it a score
53

55. Step 19: Decide whether this outcome-intervention pairing can
be recommended
• Consider the effect size
• Consider the quality of evidence
• Consider the importance of this outcome
• Create a matrix of all these choices
54

56. Step 20: Export the table to be used in documents
• Click on the Export icon
• Select HTML as an export option
• Save the file and open it using Google Docs or other apps
55

57. Limitations of the GRADE approach
• It only allows for interventions or diagnostic tests
• It only allows for head to head comparisons of two interventions
• Lets you conduct part of a full HTA
• You have to use other tools for cost effectiveness analysis
56

58. Benefits of HTA in Gradepro
• You can have objective quality appraisal criteria
• You can mix and match as many studies as you want
• You can use different studies in the same scope
• You can have different sets of outcomes
57

59. Conclusion
• Brief tour of HTA in Gradepro
• Hands on examples of HTA in Gradepro from a reanalysis of an
existing HTA
58