Quantitative methods of Signal detection on spontaneous reporting systems - Seminar Paris v

1. An agency of the European Union
Signal detection: le point de vue
de l’EMA (EudraVigilance, CIOMS,
nouvelle legislation)
Ne soyez pas dupes … je vais vous donner MON point de
vue
Presented by: François MAIGNEN
Principal scientific administrator (PhvRM)

2. Presentation title (to edit, click View > Header and Footer)2
Introduction & Disclaimers
- Background (main objective of seminar)
- Conflicts of interests & disclaimer
- Apologies for the lack of French
- Learning objectives:
- Fundamentals Disproportionality analysis
- Evaluation / Comparison of the methods (limitations, stats vs
clinical)
- Fundamental issues included in CIOMS VIII / EudraVigilance
guideline on the use of signal detection methods in EudraVigilance
DAS: DMEs/TMEs/Medical confirmation/Prioritisation/Impact
analysis
- PITFALLS +++

3. GENERAL PRINCIPLES
3

4. Before we start … Let’s bet on horse racing …
Presentation title (to edit, click View > Header and Footer)4

5. Signal detection = horse racing
• You might want to bet on the horse which will win the race.
• You might want to find the top three / five horses which will
win the race.
• You might want to read a specialised newspaper to find out
about each of the horse which will enter the race (pedigree,
jockey, owner, previous records, track, form, …).
• You will possibly use the odds to help you to decide (4:1 what
is % of bets backing a win of this horse?). An outsider might
win the race (more money).
• It is always easier to comment once the race is over than
finding the correct combination BEFORE the race starts.
5

6. Signal detection
6
PRIOR KNOWLEDGE/INFO
STATISTICS LUCK
GUESS (PROSPECTIVE) IS ALWAYS
MORE DIFFICULT

7. 7
7
Measures of disproportionate reporting
Most of the methods routinely used in pharmacovigilance
(spontaneous reporting systems) databases are based on
measures of disproportionate reporting (i.e. ROR, PRR, BCPNN,
MGPS, etc …).
Basically: “Observed vs Expected” analysis in a given database
i.e. % of reports involving a given reaction for a given medicine
compared to the % of reports involving this reaction on the
whole database

8. 8
8
A spontaneous reporting system database
SRS Drug 1 Drug 2 Drug 3 Drug 4 Drug 5 Drug 6 Drug 7 … Drug N
Event 1 n11 n12 n13 n14 n15 n16 n17 … n1N
Event 2 n21 n22 n23 n24 n25 n26 n27 … n2N
Event 3 n31 n32 … … … … … … n3N
Event 4 n41 n42 … … … … … … n4N
Event 5 n51 n52 … … … … … … n5N
Event 6 n61 n62 … … … … … … n6N
… … … … … … … … … …
Event P nP1 nP2 nP3 nP4 nP5 nP6 nP7 … nPN

9. Proportional Reporting Ratio
9
Drug 1 All other
medicinal
products
Total
Event 1 a c
All other
reaction
terms
b d
Total
N = a +
b + c + d
c + d
a + c
a + b

10. 1010
Proportional Reporting Ratio
PRR = a/(a+b) / c/(c+d) WHAT DOES THAT MEAN IN
PRACTICAL TERMS?
a/(a+b) = Proportion of reports involving a specific adverse
event among all the reports involving DRUG A
c/(c+d) = Proportion of reports involving THE SAME adverse
event among all the reports of your database but DRUG A

11. 1111
Proportional Reporting Ratio
If the rate of reporting of AE for drug 1 is similar to the rate of
reporting of this AE for all the other products of the database,
the PRR will be equal to 1 (same proportion of reports involving
the reaction for drug A than for the other drugs) …
BUT … If the reaction is proportionately MORE reported with
drug A than for the other products, the PRR will be increased
(typically > 1).
 DIS-PROPORTIONALITY of reporting

12. 1212
Disproportionality analysis (example)
. CNS drug for which the total No of reports is 400, of these 20
reports of diarrhoea
. All other products in the database (1 million reports excluding
reports involving drug A), of these 50,000 reports of diarrhoea.
PRR = [20/400] / [50,000/1,000,000] = 1 (no SDR)

13. 1313
Disproportionality analysis (example)
CNS drug for which the total No of reports is 400, of these 40
reports of drowsiness
. All other products in the database (1 million reports excluding
reports involving drug A), of these 25,000 reports of diarrhoea.
PRR = [40/400] / [25,000/1,000,000] = 4 (presence of a SDR)

14. Strong underlying assumptions
- Association between a true risk and reporting of this risk (not
always true i.e. notoriety bias)
- Similar under-reporting for products across the database (not
true)
- Role of the confounding (indication, underlying disease)
14

15. 15
Improvements of these methods
•Considering possible confounding factors:
stratification and log-linear models (ROR – see work
from E. Van Puijenbroek)
•Trying to circumvent low expected values or low
case counts: Bayesian models (A. Bate & W.
DuMouchel)
•Other regression methods: LASSO and Bayesian
logistic regressions (N. Noren, D. Madigan)
•Public Health relevance not always clear or
demonstrated
•Some methods can be computationally demanding

16. 16
Bayesian methods
BCPNN and MGPS rely on the same principle of conjugate prior
distributions:
•These methods will shrink the value of the measure of
disproportionality using a Bayesian approach (prior based on
existing dataset)
•BCPNN: cell counts ~ Binomial dist., conjugate prior = beta
•MGPS: cell counts ~ Poisson, conjugate prior = Gamma
(mixture of Gammas).
FUNDAMENTALLY SAME PRINCIPLE AS DA +++

17. 1717
Bayesian methods
Assume binomial y=7
successes, 20 trials.
Non informative prior = Beta
(2,4)

18. 1818
Thresholds - ARBITRARY
All these methods provide a ranking …
Thresholds = arbitrary
Trade-off between
•Reviewing too many drug-event pairs
(loss of operational benefit)
•Missing some signals
No ADR ADR

19. Limitations of the quantitative methods
19
The concept of threshold implies that not all the
reports will be reviewed and the quantitative
methods will not detect all the signals (for which
the data have been reported to the database on
which the DMA is used)
See Importance of reporting negative findings in
data mining – the example of exenatide and
pancreatitis Pharm Med 2008; 22(4): 215-219).

20. 2020
Comparison of the methods
Methodological difficulties
No gold standard / no standardised reference method (in many
instances “traditional methods of PhV”)
Imprecision of what constitutes a signal
Retrospective vs prospective evaluation
Importance of clinical judgement. The added value of clinical
evaluation is currently unknown (if any).

21. 2121
Comparison of methods
1
2
3
4
7
11
6
9
10
12
5
8
1
2
3
4
7
11
6
9
10
12
5
8
4
2
3
1
7
11
6
9
10
12
5
8
Meth.1
Meth.2
Meth.3
• Threshold 1: Meth. 2 = 5 true
signals, meth. 1/3 = 4.
• Threshold 1+2: Meth.
2=Meth.3
• First 5 signals: Meth. 1 ≠ Meth.
2 = 3.
ADR No ADR

22. 2222
Performances of these methods
Operational benefit (screening of large databases)
Anecdotal evidence (in opposition to structured) of signals
discovered thanks to the quantitative methods (recent
examples incl. D:A:D and MI)
Time benefit in some cases (Hochberg & EV study)
NND ~ 7/15 (depending whether the study is retrospective or
prospective)
Idea: Quant. Methods + DMEs/TMEs

23. 2323
New approaches to signal detection
Deviation of Obs. vs Expect. distr. from a fitted distribution
(Jim)
Modelling of the hazard function of the time to onset (DSRU /
François) hazard # mechanism
Use of longitudinal databases (record linkage and electronic
health records – OMOP / Noren / Callreus) ~ incidence rate
ratio
• Same patients different time windows (A. Bate)
• Hospital records of different patients (T. Callreus)

24. MODELLING OF TIME TO
ONSET
24

25. 25
Hazard fcts of parametric survival dist.
Kalbfleisch and Prentice. The statistical analysis of failure time
data. Second ed. Wiley and sons.

26. Reported hazard of occurrence: a phenomenon
involving several mechanisms
26
P(occur.)*P(diag./occur.)*P(rep./diag.)(1)
P = prob. failure conditional on survival until
time t.
Lim f(x)*g(x) = Lim f(x)*Lim g(x)
Then when we take Lim t -> 0 (1) becomes.
h(occur.)*h(diag./occur.)*h(rep./diag.)
PD
Toxicology profile
Efficacy / duration tt
Monitoring and
“RM” activities
Awareness
Awareness
Reporting mechanisms

27. 27Presentation title (to edit, click View > Header and Footer)
Liver injuries reported with bosentan (KM)

28. 28
Liver injuries reported with bosentan
(hazard functions)

29. Bosentan – liver injuries
29
Logical course of events some occurrences need
careful interpretation (blood bilirubin inc. and
[hyper]bilirubinemia)
Pattern AST/ALT unusual for liver injuries (but
not for mitochondrial injuries from hepatocytes)
but consistent with clinical safety data
Residual and constant risk of liver failure
Consistent with the putative mechanism of
toxicity (dose-dpt)
Consistent with the safety profile of bosentan
(lack of independence)
Influence of the risk minimisation activities

30. Influence of reporting mech.
30

31. FUNDAMENTAL CONCEPTS
OF SDR/SIGNAL
31

32. The fundamental difference between a SDR
and a signal +++
32
•PRR is a measure of disproportionality of reporting in a specific
database (observed vs expected value computed on the whole
database)
•The disproportionality analysis is not an inferential exercise (i.e.
the method is not aimed at drawing conclusions about a parent
population on the basis of evidence obtained from a random
sample from this population).
•These “REPORTED statistical associations” detected by
the quantitative methods do not imply any kind of causal
relationship between the administration of the drug and
the occurrence of the adverse event.

33. Different concepts / different definitions
33
SDR (signal of disproportionate reporting): refer to drug-
event pairs highlighted by DMAs. (see EMEA guideline)
NOTE: The term SIGNAL in SDR will not be retained by the
CIOMS VIII.
Signal: A signal is information on an adverse event that is
new or incompletely documented that may have causal
relationship to treatment and is recognized as being
worthy of further explorations (see CIOMS VIII). The SDRs
must be systematically medically confirmed.
(Identified) Risk: An untoward occurrence for which there
is adequate evidence of an association with the medicinal
product of interest (see Guideline on risk management
systems for medicinal products for human use
EMEA/CHMP/96268/2005).

34. 34Presentation title (to edit, click View > Header and Footer)
DMA
Database (drug-events
pairs)
SDRs
SIGNALS
SIGNALS
(other data sources)
Medical judgement
RISKS
Further evaluation / characterisation
Regulatory
action
NO

35. 35
Process flow included in the
EMEA guideline on the use of
statistical methods implemented
in the EV data analysis system
(EMEA/106464/06) July 2008.

36. SIGNAL DETECTION
PROCESS
36

37. 37
Signal
detection: a
complex
multifactorial
process

38. 38
1. Data collection
Data capture
(management)
Data transmission

39. Data capture and data management (1)
39
Fundamental but not in the scope of CIOMS VIII
IT infrastructure and software
The volume of information hence the data
management activities (data coding, entry,
recoding, data quality) is extremely resource
demanding.
Data management will have a critical influence
on the signal detection activities incl.
• Medicinal product information: creation and maintenance
of dictionaries, lack of international standard, absence of
INN or standards in some instances e.g. vaccines
• Medical terminology: criteria for the use of terms,
conversion of legacy data encoded with a different
terminology, …
• Data quality: FUp, duplicates

40. 40
2. Methods, signal
detection & data
analysis

41. 41

42. 42

43. 43
3. Signal
management
(prioritisation,
evaluation, decision
& communication)

44. 44
4. Link with risk
management

45. Signal management
45
• Similarly the CIOMS has identified a signal
management step which includes:
• Triage
• Prioritisation and impact analysis
• Evaluation
• Decision
• Communication (broad sense)
• Follow-up
• Link with risk management

46. FUNDAMENTAL QUESTION OF IMPACT
ANALYSIS
NO VALIDATED METHOD. Assess the Public Health impact of the
signal:
Usually:
-Seriousness
-Frequency of occurrence (absence of evidence is NOT evidence
of absence)
-Particular population at risk
-“worst case scenario” (what would happen if … ?)
-Preventability, reversibility, etc …
46

47. PITFALLS:
PRIORITISATION /
IMPACT ANALYSIS
47

48. 48
Signal prioritisation and serious medical
events: reported rate of fatality as a
prioritisation variable
About the EV-EWG IME list and lists of IMEs in general (e.g.
CIOMS V)
Useful but purpose not always clear (early signal detection?
Focus the detection? Signal prioritisation?)
Based on expert’s judgment
Has not been formally “validated” / tested (no standards)
Probably situation dependant

49. 49
Concept of seriousness # linked to the
outcome # surrogate for grading the severity
of the reactions hence prioritisation
Grading in seriousness: death >> disability (permanent) >>>
life-threatening >>> disability (temp.) >> prolongation hosp.
Variable linked to fatal outcome = reported rate of fatality
For each drug-event pair = No of reported fatal cases / total
number of reported cases
Computed for the intensively monitored products
Reaction 1 Reaction 2 Reaction 3 Outcome (incl. fatal)
Surrogate to predict the outcome

50. 50
Hazardous identification of serious events a
priori
Some examples of reactions not usually considered to be
serious per se which can be linked to most dramatic outcomes
(e.g. dramatic increases of liver aminotransferases e.g. >100
ULN leading to liver failure, liver transplant and death)
Exhaustion/
tiredness
Jaundice
incr. aminotransferases 500ULN
hyperbilirubinemia
Liver transplant Death
Prioritise these events on the associated reported outcome (here death)

51. 51
Reported rate of fatality
Some reactions may be consistently linked to a high reported
mortality rate
Some reactions are serious but do not lead to a fatal outcome
Some reactions are situation dependent (the reported rate of
fatality may be highly variable)
For each of the MedDRA PT involved in a DEC in EudraVigilance,
the following variables were computed across all the products
involved in the reported combinations:
• Mean, min., max., range: max. – min., SD

52. 52
How does it relate to IME status?
Reported rate of fatality for
IMEs > non-IMEs
Number of events for which the
reported rate is high which are
non-IMEs
Very high number of IMEs for
which the reported rate of
fatality is zero.
IMEs useful for prioritisation?
The figure displays the boxplot of the average reported rate of fatality for non-IMEs (left)
compared to IMEs (right) (red and blue line = mean rate for non-IMEs (red) vs IMEs (blue))

53. 53

54. 54

55. Liver injuries
Clear relation between Reported rate # seriousness of injury
and the severity of the outcome
Highest mean rate around 30% (1/3 fatal reports) with a max.
at 75% (3/4)
Some inconsistencies (bilirubin disorders: hyperbilirubinaemia
18.7%, blood bilirubin increased 16.2%, blood bilirubin
unconjugated increased 6.7% and bilirubin conjugated
increased 6.3%)
Unclear or undefined concepts (liver disorders [?]) linked to a
fairly high mean reported rate 18.9%, hepatic function
abnormal 8.5% and liver function test (singular) abnormal
9.1%.

56. 56
Data reduction (PCA)

57. 57
Discussion
Three set of events used for signal detection: mild reported rate
of mortality, moderate and high
Reported rate of fatality can be useful (and should be used) for
signal prioritisation
Needs to be considered with caution (events with rate of zero
include e.g. Torsade de pointes, autism, Breast cancer in situ,
Breast cancer stage I, Dermatitis exfoliative)
Does not replace DMEs
Death is not the only criterion which could be used
EudraVigilance = only serious reactions(!)
Some events are consistently associated either with low rate or
conversely with very high rate

58. PITFALLS: MASKING
EFFECT OF DA
58

59. 59
Masking effect of measures of
disproportionality (here = PRR)
The masking effect has first been described and identified by Gould in
spontaneous reporting system databases (pharmacoepidemiology and
drug safety in 2003 – 8 years ago).
The masking is a statistical artefact by which true signals are hidden by
the presence of information reported with other medicines in the
database. Therefore, the masking involves one given reaction and two
products (the product for which the DA is conducted) and a possible
masking drug.
The masking effect is a potentially important issue for Public Health
which is not perfectly understood or perfectly quantified: some signals
might be missed or identified with delay because of the presence (or a
suspicion on the presence) of masking effect.

60. 60
Masking effect of measures of
disproportionality
In particular, there is no algorithm to identify the potential
masking drugs to remove them from subsequent analyses
aimed at identifying new signals using the statistical methods of
signal detection based on disproportionality analysis.
We have developed an algorithm based on the computation of a
simple The masking ratio has been developed to be intuitive.
The highest masking drugs have the highest masking ratio.
From an underlying mathematical framework, we have
developed a simple expression of the masking ratio (which can
be easily computed on a database incl. No of computations and
IT resources) which allow a fairly rapid identification of the main
culprits.

61. 61

62. 62
Masking effect of measures of
disproportionality
Recent studies have shown effects which were suspected from
the article by Gould, that masking products are usually products
for which the given reaction is known (i.e. listed in the SPC),
therefore likely to have a high PRR (in the database in which
the analysis is conducted) for the adverse drug event / reaction
which is included in the disproportionality analysis.
Unfortunately, the authors could not conclude on any algorithm
considering that this association is not systematically present
(not all products with high PRR will induce a significant masking
even if he masking generally involves products with a high
PRR).

63. 63
Masking effect of measures of
disproportionality (RRR)
Respective proportion of reports in
the database influences the extent
of the masking
The higher the proportion of reports
involving a product for a given
reaction the higher the masking
The lower the proportion involving a
given product over the total number
of reports in the database, the
higher the masking

64. 64

65. 65
Relation between the masking effect and the
PRR (of the masking medicinal product for the
given event)
MR > 1
PRR > 2

66. 66
The highest masking is
induced by products known
to induce the given reaction
(and for which the PRR is
likely to be increased)
Products of the same class
induce the highest masking
for similar reactions
(gambling – ropinirole and
pathological gambling –
cabergoline, Fanconi
syndromes, role of drug-
drug interactions –
rifampicin)

67. CONCLUSIONS
67

68. FUNDAMENTAL ISSUES: take home messages
- Image of horse racing
-Most of the methods rely on disproportionality analysis: strong
underlying assumptions
-SDRs: statistical association : needs to be systematically
medically confirmed
-Process flow: PRIORITISATION & IMPACT ANALYSIS
-PITFALLS: METHOD (e.g. masking), prioritisation (e.g. IMEs)
-Importance of strategy incl. DMEs / TMEs
-PRIOR MEDICAL KNOWLEDGE (Prepared mind)
68

69. 69
Merci
francois.maignen@ema.europa.eu
http://uk.linkedin.com/in/francoismaignen
@EMA_News
@FrancoisMaignen