Review of information retrieval – first experiences in Germany with manufacturer dossiers on new drugs

1. Review of information retrieval – first experiences in
Germany with manufacturer dossiers on new drugs
9th HTAi Annual Meeting BILBAO
26 June 2012
Dr. Siw Waffenschmidt
Information Management Unit, IQWiG

2. Content
1) Introduction to the early benefit assessment of new drugs
2) Requirements for information retrieval in the dossier template
3) Review of information retrieval by IQWiG
4) Examples
Informationsbeschaffung in Dossiers 2

3. Drug assessment according to AMNOG
Market
Submission of a dossier Price set by entry
by the manufacturer manufacturer
Assessment (by
IQWiG) Month 3
Decision on added
benefit by the Federal
Joint Committee Month 6
no Added benefit? yes
Reference Price negotiations
Permissible for
price yes no SHI and
reference price?
manufacturer
Price at no added
cost versus Discount on
Agreement? yes
comparator sales price Month 12
treatment no
Decision by arbitration
body Month 15
On request of SHI or
manufacturer: health
3 economic evaluation

4. Content
1) Introduction to the early benefit assessment of new drugs
2) Requirements for information retrieval in the dossier template
3) Review of information retrieval by IQWiG
4) Examples
Informationsbeschaffung in Dossiers 4

5. Requirements for information retrieval in dossier templates –
Module 4
 4.2.3 Methods of information retrieval
 Bibliographic literature search
 Search in trial registries
 Selection of relevant studies
 4.3.1 Results of information retrieval
 RCTs of the drug to be assessed
 Studies for indirect comparisons
 Indirect comparisons on the basis of RCTs
 Non-randomized comparative studies
 Further studies
 Appendix
 4-A: Search strategies – bibliographic literature search
 4-B: Search strategies – search in trial registries
 4-C: List of studies screened as full texts and excluded (with reason for
exclusion)
 Attachments
 RIS files
Informationsbeschaffung in Dossiers 5

6. Content
1) Introduction to the early benefit assessment of new drugs
2) Requirements for information retrieval in the dossier template
3) Review of information retrieval by IQWiG
4) Examples
Informationsbeschaffung in Dossiers 6

7. Process in Information Management Unit:
Check the search in bibliographic
Check the search in trial registries
databases
Evaluate information retrieval
Depending on results of evaluation:
(potentially) conduct own search in Depending on results of evaluation:
bibliographic databases and/or trial No further action
registries
Documentation
Informationsbeschaffung in Dossiers 7

8. Checking the search in bibliographic databases (formal check)
Methods of information retrieval
Check whether requirements for the search procedure and the presentation of
results have been fulfilled as specified in the dossier template
Screening
Was the procedure for study selection described?
Are deviations from the above requirements justified?
Result of the information retrieval
Check consistency of retrieved hits
Do the individual sections of the dossier contain consistent information on
inclusion and exclusion criteria?
 Documentation of the above steps in checklists developed for this purpose.
Informationsbeschaffung in Dossiers 8

9. Checking the search in bibliographic databases (content check)
 Is the number of hits plausible?
 Checking the search strategy presented in Appendix 4-A
 Were search filters used and correctly transferred?
 Syntax acc. to PRESS Checklist
 Was the study pool identified via the search strategy?
 Is the search sensitive enough?
Sampson M, McGowan J, Lefebvre C, Moher D, Grimshaw J.
PRESS: Peer Review of Electronic Search Strategies. Ottawa:
Canadian Agency for Drugs and Technologies in Health; 2008.
URL:
http://www.msmandhiv.org/documents/globaltopics/PRESS.pdf.
Informationsbeschaffung in Dossiers 9

10. Process in Information Management Unit:
Check the search in bibliographic
Check the search in trial registries
databases
Evaluate information retrieval
Depending on results of evaluation:
(potentially) conduct own search in Depending on results of evaluation:
bibliographic databases and/or trial No further action
registries
Documentation
Informationsbeschaffung in Dossiers 10

11. Potential literature searches by IQWiG
1. Search in public trial registries
- ClinicalTrials.gov
- ICTRP Search Portal and
- potentially in industry registries
2. Search in bibliographic databases
- Simple Structured Boolean Search (SSBS)
- combined with the first 20 Related Citations in PubMed
Sampson M. Updating searches for systematic reviews. Aberystwyth:
University; 2009. URL:
http://cadair.aber.ac.uk/dspace/bitstream/2160/3760/1/Sampson%20-
%20Updating%20Searches%20for%20Systematic%20Reviews%20-
%20PhD.pdf.
Waffenschmidt S, Janzen T, Hausner E, Kaiser T. Assessing the
completeness of systematic reviews via the “related articles” function and/or
a simple structured Boolean search in PubMed – a pilot study.2011.
(Scientific evidence for healthcare quality and patient safety: 19th Cochrane
Colloquium; 19.-22.10.2011; Madrid, Spanien).
Informationsbeschaffung in Dossiers 11

12. Process in Information Management Unit:
Check the search in bibliographic
Check the search in trial registries
databases
Evaluate information retrieval
Depending on results of evaluation:
(potentially) conduct own search in Depending on results of evaluation:
bibliographic databases and/or trial No further action
registries
Documentation
Informationsbeschaffung in Dossiers 12

13. Documentation
Informationsbeschaffung in Dossiers 13

14. Content
1) Introduction to the early benefit assessment of new drugs
2) Requirements for information retrieval in the dossier template
3) Review of information retrieval by IQWiG
4) Examples
Informationsbeschaffung in Dossiers 14

15. Examples of shortcomings potentially resulting in a check of the
completeness of the study pool by IQWIG
 Search strategy errors
 Lack of sensitivity
 Lack of plausibility of the number of hits
 Inconsistencies in search documentation
Informationsbeschaffung in Dossiers 15

16. Syntax errors → Search not sensitive enough
Informationsbeschaffung in Dossiers 16

17. EMBASE
Interface: Elsevier
Ovid Interface
results in 64.508
Informationsbeschaffung in Dossiers 17

18. Lack of plausibility of the number of hits
Informationsbeschaffung in Dossiers 18

19. Search checked on 08.07.2011: n= 61
New citations since 01.05.2011: n=3
Informationsbeschaffung in Dossiers 19

20. Experiences so far
 additional relevant studies have only been identified from the list of studies
provided by the manufacturer (e.g. study listed but excluded by
manufacturer), not by own searches
 Challenges in the checking and evaluation of information retrieval:
 Assessment of the sensitivity of the searches
 Indirect comparisons and further studies
Informationsbeschaffung in Dossiers 20

21. Conclusions
 Our approach to the review of information retrieval in manufacturer dossiers
seems to be workable.
 Besides checking the search strategies, peer review of information retrieval
should cover all aspects of a search.
 Ultimately, the in(completeness) can only be verified by doing a focused
search yourself .
Informationsbeschaffung in Dossiers 21

22. Institute for Quality and Efficiency in Health Care (IQWiG)
Dr. Siw Waffenschmidt
Potential Conflicts of Interest: NONE
 Im Mediapark 8
 50670 Cologne, Germany
 Tel: +49-221/3 56 85-0
 Fax: +49-221/3 56 85-1
 Siw.Waffenschmidt@iqwig.de
 www.iqwig.de
Informationsbeschaffung in Dossiers 22

23. Search in bibliographic databases
Choice of search techniques is based on the work by Sampson 2009 (“Updating
searches for systematic reviews” [PhD])
On the basis of Sampson’s approach, we conducted our own analysis based on
systematic reviews of drugs:
Main result:
The combination of the first 20 Related Citations and the SSBS seems to be the most
appropriate technique, with high completeness (sensitivity: 97.1%), good reliability
(sensitivity range: 80-100%) and an efficient use of resources (precision: 5.0%, NNR:
24). (manuscript submitted)
Informationsbeschaffung in Dossiers 23

24. Example – IQWiG’s search in PubMed
Pubmed: 13.01.2012
Total number of hits without duplicates: 70
Screening result: no relevant additional studies found
Items
Search Query
found
#1 Search Sativex*[tiab] or Nabiximol*[tiab] or Cannab*[tiab] or 20478
GW*1000*[tiab] or *THC*[tiab] or Marinol*[tiab] or
Tetrahydrocannabinol*[tiab] or Nabilon*[tiab]
Simple
Structered
#2 Search multiple sclerosis 51363
Boolean Search
#3 Search #1 and #2 355
#4 Search (Therapy/Narrow[filter]) AND (#3) 33
#5 Search 14615106[pmid] 1
The first 20 Related
#6 Related Citations for PubMed (Select 14615106) 106
Citations of the 5
#7 Select 19 document(s) 19 relevant citations
#8 …… ….
included in the dossier
Informationsbeschaffung in Dossiers 24