Presentation at IBC's Biological Assay Development and Validation conference 12 May 2011

1. Developing Bioanalytical Methods
Balancing the Statistical Tightrope

2. “Lee: can I use this number?”
Process Development
GSK, 1997
2

3. “it’s about 40”
“about 40?”
“probably...”
3

4. Enlightenment?

5. 5

6. Blooms Taxonomy
the 4 stages of competence
Incompetent Competent
Conscious
Consciousness
Unconscious
6

7. A Statistical God
Me

8. Using Statistics

9. Why? Six Reasons
1. Potency assays are key in making medicines
2. Bioassays are very variable
3. Statistics will help you understand your data
4. Understanding your data will reveal if control
exists
5. Your level of control allows you to judge RISK
6. Regulators globally require it 9

10. The Regulator & Assay Control
Regulators have been asking for this for years! QbD
1. Pharmaceutical cGMPs for the 21st
Century
2. PAT
3. ICH Q2: Validation of Analytical
Procedures
4. ICH Q8: Pharmaceutical Development
5. ICH Q9: Quality Risk Management
6. ICH Q10: Quality Pharmaceutical
Systems
10

11. Statistics
The complete solution?

12. Or this?
Your assay?
12

13. Or this?
or your assay?
13

14. Statistics - an Amazing Transition
14

15. Bioassays will always be variable
You can improve it
- by understanding it
- Focusing effort in right places
- This brings control
- You can manage expectations
- This is understood by regulators
15

16. Why assay variation matters?
product variation +
A few unsatisfactory
assay variation + batches may even
inaccuracy pass specification
due to a combination
of assay method and
process variability
Many satisfactory OOS batches likely to fail (potentially costing £Ms)
because of combination of assay method & process inaccuracy & variation
16

17. Our Control Strategy
What does the scientist need to achieve?
Define i.e. selectivity, accuracy, precision linearity
Identify & prioritise analytical CNX parameters
Measure
Control Noise eXperimental
parameters parameters parameters
Analyse
Fix & control e.g., MSA, e.g., DoE
Input into
Precision Regression
Method Method
Improve
Ruggedness Robustness
Method Control Strategy & reduce Risk prior to
Control Validation → Routine Use & Continuous Improvement
17

18. Generating Bioassay
Data
18

19. The Rules
1. Speak with your statistician before
generating data
2.See Rule 1
19

20. Lot’s data ≠ Value
20

21. 21

22. Statistics are a tool
22

23. QC Which Tools?
UCL
Stage 4 Technology
QC Tools
CELLULA, Shewhart chart,
YES Transfer
LCL CUSUM
NO
TIME
Stage 1:
Qualification Tool
Stage 3:
Fishbone, Minitab
Validation Tools
Nested, CELLULA
Precision
Stage 2:
Accuracy
Linearity etc.
Development Tools
DX8, JMP, Minitab
Design

24. What’s Appropriate Knowledge?
• Learning takes time
• Will you use it often enough?
• It’s not an academic pursuit
• Activities must add value
 do what’s necessary
24

25. Scope
&
Design

26. Define & Scope
How is the assay performing? Prec/TOL2-sided = 6 x 16.76
100
= 1.01
26

27. Parameters (e.g. 15)
pDNA
NaCl
pH
Tube Length
Time
Seeding Density
Ratio of Transfection
Temperature
Agitation and level
Vector – type, conc
Addition Order

28. Q. How Many parameters?
Q. Which parameters?
Q. What ranges?
A. Existing knowledge
A. Common sense
A. Practical limits

29. Define & Scope
Drill down - map out assay - build understanding & scope
Assay Flow
29

30. Define & Scope
Drill down & map out assay to build understanding & scope
Attention is focused
toward key steps
and the parameters
involved in these
steps
Cause & Effect Diagram (Fishbone) helps think your assay through
Identify & prioritise analytical CNX parameters 30

31. Scope & Screen
Scope ranges with simple experiments
Scoping Experiments
Explore mildest
to most forcing
conditions
31

32. Revealing the Big Hitters
32

33. Temptation

34. Building Understanding
OFAT
Provides estimates
of effects at set
conditions of the NaCl
other factors and
no interaction pDNA
effects. pH
34

35. Building Understanding
Factorial Design 2400 2600
1300
900 1800
Estimates effects at
different conditions to
estimate interactions 350 600
250
300 500
Design of Experiments
DOE
35

36. Optimisation
Optimise the parameters that
survived the initial screening
work towards a
Robust Optimum
36

37. Simulations
The tools allow you to simulate scenarios using the data you’ve built up
Visual simulation of expected performance relative to specification
37

38. Is the Model Correct?
38

39. Validate & Verify
The evaluation of robustness should be considered
during the development phase and should show the
reliability of an analysis with respect to deliberate
variations in method parameters ICH Q2B, 1994
Method stretch…what if?
Ideal Settings
Control Space
Design Space
39

40. Assay Control: control the parameters inside boundaries
40

41. Working within the control
boundaries will keep the
assay under control
Even if you go outside
the control boundaries,
the assay will have
enough flexibility to
deal with it without an
OOS
41

42. Summary - Data Driven Development
Scope Screen Optimize Verify QC/TT
Transfer to QC to
validate on batches
& bring into routine
use
Explore mildest Identify few potential Estimate & utilize
to most forcing key parameters interactions to move Rattle the cage to
conditions Focus on vital few & towards optimum deliver a design
narrow ranges conditions space

43. 43

44. Precision
It may be considered at three levels:
1. Repeatability
2. Intermediate precision
3. Reproducibility
ICH Q2A, 1994

45. Repeatability
1 analyst in 1 laboratory on 1 day injecting 6 times
Summary Statistics
Number of Standard Coefficient Lower 95% CI Upper 95%
Values Mean Deviation of Variation for Mean CI for Mean
t30 PS 6 223.27 6.43 2.88% 216.52 230.02
45

46. Intermediate Precision
• 1 analyst in 1 laboratory on
• 1 day
• injecting 6 samples
• each tested 6 times
As well as sample variation, this study still provides
information on repeatability
46

47. Intermediate Precision
So we compare the mean values for each sample
(over replicate results per sample)
Variance Components
Factor df Variance % Total
Sample 5 27.8535 21%
Repeat 30 102.6361 79%
35 130.4896 100%
Standard
Mean Deviation RSD
47
216.24 11.4232 5.28%

48. and the others…..?
Precision within a laboratory but with
different analysts, on different days, with
different equipment…reflects the real
conditions within one laboratory
ICH Q2A 1995
48

49. Intermediate Precision
Data collect using several analysts using several instruments
over several days:
Y
56000
55500
55000
54500
Peak Area
54000
53500
53000
52500
52000
0 5 10 15 20 25
Sample
49

50. Intermediate Precision
Potentially misleading: large analyst-to-analyst variation
present:
Y
56000
55500
55000
54500
Peak Area
54000
53500
53000
52500
52000
0 5 10 15 20 25
Sample
Analyst 1 Analyst 2 Analyst 3
50

51. Intermediate Precision
better examined looking at multiple
sources of variation within an assay
want to understand
major sources of
variation such as
sample, prep,
analyst etc.
51

52. Intermediate Precision
52

53. Intermediate Precision
Can also perform Unbalanced designs
One operator performs multiple injections on single
preparation;
Two operators perform single injections on multiple
preparations
53

54. Reproducibility
multiple laboratories; typically run as an inter-
laboratory cross-over study, with each participating
lab sending samples to every other lab and
analysing all samples (including own)
…. sent to and analysed by other lab
A B C
Samples from A   
laboratory:
B   
C   
54

55. Reproducibility
Can use analysis of variance (ANOVA) to look for
differences or biases between labs
Alternatively look for “analytical equivalence”

56. Risk Management
The level of effort, formality and documentation..
..should be commensurate with the level of risk
ICH Q9
Evaluation of the risk to quality should be based on
scientific knowledge & ultimately link to the
protection of the patient
Is the bioassay fit for purpose and under control?
56

57. Before & After
How is the assay performing? P/TOL2-sided = 6 x 16.76
100
= 1.01
57

58. Before & After
Better P/TOL2-sided = 6 x 6.99
100
= 0.42
58

59. Risk Management
Method Understanding, Control and Capability (MUCC)
Understand impact of variation
upon risk…
Risk Understanding?
Capable?
Management
Loop
Statistical
Capability
Process Control
& Precision
(SPC) Charts
Control? 59

60. Risk Management
Understanding?
Understanding?
Capable?
P/TOL2-sided = 6 x 16.76
Capable? 100
Control?
= 1.01
Capability
& Precision
60

61. Risk Management
P/TOL2-sided = 6 x 6.99
100
I-MR Chart of t30 PS
Summary Report
= 0.42
Is the process mean stable? I Chart
Evaluate the % of out-of-control points. Investigate out-of-control points.
0% > 5% 225
UCL=220.77
Yes No
210
0.0%
t30 PS
_
X=199.87
195
Comments
180 LCL=178.96
The process mean is stable. No data points are out of control
1 6 11 16 21 26 31 36 41 46
on the I chart. 61
Observation

62. Summary
1.Build a good basic understanding of
stats but don’t need to become guru
2.Involve a statistician, at least at the
beginning
3.Build understanding of your bioassay
(QbD) – it’s a must
4.Get to grips with Bioassay Variability
62

63. “Lee: can I use this number?”
63

64. “Yes – it’s 42 … ”
 0.05 with 95% Confidence
for the statisticians in the audience
64

65. Acknowledgments
Dr. Paul Nelson – Prism TC Ltd
Pictures from “The Cartoon Guide to Statistics”
Larry Gonick & Woollcott Smith
65