This lecture shows the procedures applied when going to validate your laboratory instruments and quantitative test methods also either FDA approved or laboratory developed tests.

1. Validation of Lab Instruments
and Quantitative Test Methods
By: Dr Mostafa Mahmoud MD,
Consultant Microbiologist Labs & Blood Banks Admin, Riyadh,
MOH
Associate Professor of Medical Microbiology & Immunology
Faculty of Medicine – Ain Shams University

2. Method Evaluation
VALIDATION:
- Establishing standards performance through a defined process. Long
studies
-For Non- FDA approved testing.
VERIFICATION:
- Short studies to demonstrate that a test performs in significant
compliance to previously established claims.
- For FDA approved testing.

3. I- Equipment validation:
DEFINTION: Demonstrate that equipment used in validation studies is suitable
for use and is comparable to equipment used for routine analysis. The
equipment must be calibrated, qualified, and maintained properly.
Components of equipment validation:
• Calibration
• Qualifications
Installation Qualification
Operation Qualification
Performance Qualification
• Routine maintenance performed- proper working order

4. Instrument Validation – Experiments
1. Accuracy (correlation or comparison)
2. Precision (Replication)
3. Analytical Sensitivity
4. Analytical Specificity
5. Linearity (to verify reportable range (AMR)

5. When and what to do for instrument validation?
1. New Instrument of a different make or model of current
instrument – Must be validated for all method
performance specifications including: accuracy, precision,
analytic sensitivity, specificity and reportable range.
• 2. Additional Instruments of same make & model as the
current instrument- Each instrument must be validated
separately. If several instruments are validated/verified

6. 2. Additional Instruments of same make & model as the current
instrument- Each instrument must be validated separately. If
several instruments are validated/verified at the same time only
one validation is needed.
Each instrument must be validated for method performance
specifications including: accuracy, precision, reference range and
reportable range (AMR).
a- Accuracy may be verified for the additional instrument by
comparison study with the instrument currently in-use (15-
20 samples).
b- No separate reference range study is needed for 2nd
instrument, assuming comparison study showed absence of
significant bias.

7. 3. Instruments that have been moved from one location to
another in the laboratory - Must be validated for method
performance specifications including: accuracy, precision and
reportable range (AMR).

8. Instrument Validation Summary
• Once the method experiments are complete, summarize the
results in a Method Validation/Verification Summary.
• Clearly state the purpose of the verification, what
platform/method and the number of samples for each
experiment.
• Any discrepant results should be investigated and explained in the
Summary. Test results that show sample problems such as
contamination and degradation should not be used in the
assessment but still listed with an explanation.
• The Summary should also contain a Conclusion stating weather
the instrument study met the acceptance criteria or not and its
suitability for use in the laboratory

9. II- Laboratory Test Validation
CAP Performance Specifications for Analytic (Test) Validation:
A- Analytic accuracy.
B- Precision.
C- Analytic sensitivity (limit of detection, LOD).
D- Analytic specificity (interferences).
E- Reportable range.
F- Reference range

10. When to perform method evaluation studies?
1- Presence of perennial problem by quality control study.
2- When a new different diagnostic kit is to be used.
3- When the used method is changed by adding new
biological material.

11. A- Quantitative Methods validation
I- FDA cleared or approved methods:
Each laboratory that introduces an unmodified, FDA-cleared or
approved test system must demonstrate that it can obtain
performance specifications comparable to those established by
the manufacturer for the following performance characteristics
before reporting patient test results: Accuracy, Precision,
Reportable Range of the test results and verification that the
manufacturer's reference intervals (normal values) are
appropriate for the laboratory's patient population.

12. 1- Analytic Accuracy
‫والحقيقية‬ ‫المقاسة‬ ‫النتيجة‬ ‫بين‬ ‫التوافق‬
• Definition: agreement between test result and “true” result. The difference can
be described as the Systemic error (inaccuracy, or bias) in the method.
• Systemic error = inaccuracy (Constant error – & Proportional error)
Method:
- The comparison method by testing 20 (20-40) samples within the testing range
by the new and reference method.
- The method comparison experiment for accuracy is recommended to be done
over a minimum of 5 days. Continue for another 5 days if discrepancies are
observed.
• Test material sample can include: calibrators/controls, reference material,
proficiency testing material with known values
- If bias found in 20 samples use larger samples (100 samples)

13. • Prepare a comparison plot of all the data to assess the range, outliers, and
linearity.

14. Recovery Experiment:
• In the absence of a reliable comparison method, recovery studies
are used better to use a method near the gold standard one.
• The recovery experiment is performed to estimate proportional
systematic error (difference of error increases as the concentration
of the analyte increases. ).

15. Statistics: Accuracy / Bias (= systematic error):
• Run comparison of methods study (test method vs. reference method,
laboratory’s previous method, or manufacturer’s results, etc.). The line of
best fit (calculated using a statistics program) provides the linear
regression equation Y = a + bX.
• Calculate correlation coefficient “r”.
• If “r” is high (≥ 0.99), use the regression line to find the bias at analyte
concentrations that correspond to critical decision points (ex. glucose: 126
mg/dL).
• If “r” < 0.975, the regression equation will not be reliable; use paired t-
test to determine if a bias is present at the mean of the data.
• “r” should not be used to determine the acceptability of a new method.
“r” measures how well the results from the 2 methods correlate

16. What is Systematic Error?
• Systematic error (change) is a change that is always in one
direction (low or High) and will cause a shift in the mean
value.
• Systematic error or change, is associated with a change
in accuracy.
• Some sources of systematic error are however
unavoidable.

17. Types of Systematic Error
• Constant: an error that is always in the same direction and of the
same magnitude even as the concentration of analytes changes.
• Proportional: an error that is always in one direction and the
magnitude of which is a percentage of the concentration of the
analytes being measured.

19. Causes of Systematic Errors
1. Changes in reagent lot number
2- Changes in the instrument.
2- Changes in calibration

20. 2- Analytic Precision (Reproducibility)
‫ال‬ ‫اختلف‬ ‫لو‬ ‫حتى‬ ‫النتيجة‬ ‫نفس‬ ‫تعطي‬ ‫مختلفة‬ ‫أوقات‬ ‫في‬ ‫اإلعادة‬‫فني‬
• Definition: Precision is the repeatability. The ability of the laboratory
to duplicate results time after time on different days and with
different operators.
• Imprecision = Random Analytical Error (expressed in CV% from the
calculated standard deviation SD and mean).
• Precision is measure of Random Error (fluctuation of measured
values from their mean due to random factors).
• Increase in the random error leads to variation of result above or
below the mean value.

21. Method of measuring Precision:
• patient samples are the first choice followed by control material and
reference solutions.
• Repeat measurements of samples at varying concentrations, within-
run and between run over a period of time should be performed by:-
1- Use a minimum of 2-3 samples near each medical decision levels run
for 3-5 replicates over 5 days will provide sufficient data for within-
run and between-run components to estimate precision.
2- Measure precision by dividing the obtained results over the known
results multiplied by 100.
3- Use spreadsheet to calculate the SD and compare to the
manufacturer claims by using CV%.

22. • The laboratory should verify the manufacturer’s claim for precision
by using the F-test (CLIA), as follows:
• Use F test to see if variance (=SD^2) of test method is statistically
different from old method, or claim of manufacturer
• Example of how to use the F test:
• Obtain the expected SD and number of measurements used in the
replication experiment from the manufacturer’s claims (usually
included in the instrument documentation), e.g., SD =3 mg/dL based
on 31 measurements.
• Obtain the SD and number of measurements from the replication
experiment, e.g., SD = 4 mg/dL based on 21 measurements.
• Calculate the F-value, larger SD squared divided by smaller SD
squared, i.e., (4)2/(3)2 = 16/9 = 1.78.

23. • Look up the critical F-value for 20 degrees of freedom (df=N-1) in
the numerator and 30 df in the denominator in the F-table,
where the value found should be 1.93.
• In this case, the calculated-F is less than the critical-F, which
indicates there is no real difference between the SD observed in
the laboratory and the SD claimed by the manufacturer.
• Conclusion – the manufacturer’s claim is verified when the
calculated F value is less than the critical F value.

24. Random
Error

25. Random
Error

26. Causes of Random error:
1. Inconsistent environmental conditions e.g. temperature humidity
etc,
2. Electrical interferences.
3. Inconsistent handling of materials.
4. Fluctuation of volumes used in the testing.

27. Precision (Replication) Study
• Simple Precision: Repeatability (within run)
• Complex Precision: Reproducibility (between runs, between
days)

29. 3- Reportable Range
• CLIA defines this as the highest and lowest test values that can be
analyzed while maintaining accuracy without dilution or
concentration.
•Method and number of samples:
• To verify reportable range, running 3 points near low end, midpoint,
and high end using calibration/control/reference matrix appropriate
materials.
• The AMR must be reverified at least every 6 months, and following
changes in major system components or lots of analytically critical
reagents
• These samples can be combined with the accuracy/precision
experiments.

30. 4- Reference Range (Normal Values)
‫المجتمع‬ ‫في‬ ‫للتحليل‬ ‫الطبيعي‬ ‫المدى‬
1. No need to define own limits but verify that the adopted one is
appropriate for the served populations.
2. The adopted reference ranges may be from suggested
manufacturer, reference lab, published articles, neighboring lab
or previous limits of own lab.
3. The Reference Range can be verified by testing 20 known normal
samples; if no more than 2 results fall outside the
manufacturer/published range then that reference range can be
considered to be verified.

31. Method if applicable:
• Validation protocol (if needed) include testing 20 samples from
healthy individuals;
- if < 2 outside the proposed limit the values are validated for
reference range.
- If > 2 outside, can repeat with another 20, and accept if ≤ 2 outside
(not worth repeat if > 5 outside proposed limits).
- If still not validated the own reference limit is needed to be done by
examining 120 individuals (200 may be used), criteria for
exclusion are to be followed.

32. II. Non-FDA Cleared
• Each laboratory that modifies an FDA-cleared or approved test
system, or introduces a test system not subject to FDA clearance or
approval (including methods developed in-house and standardized
methods such as text book procedures), or uses a test system in
which performance specifications are not provided by the
manufacturer must, before reporting patient test results, establish
for each test system the performance specifications for the
following performance characteristics, as applicable:
• Accuracy, Precision, Analytical sensitivity, Analytical specificity to
include interfering substances, Reportable range, Reference
intervals, Any other performance characteristics required for test
performance, Determine calibration and control procedures and
document all of the above.

33. 1- Accuracy/ Bias (= systematic error):
• Same as in FDA-cleared tests except: Most sources recommend
comparing at least 40 patient specimens for a Laboratory Developed
Test (LDT).
• The actual number is less important than the quality of the samples.
• The estimate of systematic error is more dependent on wide range of
test results than on a large number of samples.

34. 2- Precision (= random error):
• 2 different control materials that represent low and high medical
decision concentrations. Analyze the low control and high control
at least 20 times each within a run to obtain short term
imprecision.
• Calculate mean, standard deviation and coefficient of variation for
each material.
• Determine if short term imprecision is acceptable before
proceeding to the long term imprecision experiment.

35. • Long-term imprecision experiment:
• Analyze 1 sample of each of the 2 materials on 20 different days to
estimate long-term imprecision.
• Calculate the mean, standard deviation, and coefficient of
variation for each material.
• Determine whether long-term imprecision is acceptable.
• Having different operators perform the precision experiment must
be done for methods that are operator dependent.

36. 3- Reportable Range (analytic measurement range= AMR):
• Same as above in FDA-cleared tests. The AMR must be reverified at
least every 6 months. If the range has not been established, a
linearity experiment will have to be performed.

37. • Linearity Experiment:
• Test a series of known dilutions of a highly elevated specimen or
patient pool.
• The measured or reported test values are compared to the
assigned values or to the dilution values, typically by plotting the
measured values on the y-axis and the assigned or dilution values
on the x-axis.
• (CLSI) recommends a minimum of at least 4, preferably 5 different
concentration levels.

38. 4- Reference Range (Normal Values):
• Same as above in FDA-cleared tests. When there are no well-
established reference intervals are available, additional samples
will be required.
• CLSI recommends to use a minimum of 120 reference individuals
for each group (or subgroup) that needs to be characterized.

39. 5- Analytic Sensitivity (Detection limit)
‫االختبار‬ ‫بواسطة‬ ‫قياسها‬ ‫يمكن‬ ‫قيمة‬ ‫اقل‬ ‫االختبار‬ ‫حساسية‬
• Called lower detection limit. It is the smallest quantity of an analyte
that can be reproducibly distinguished from background levels.
• Positive agreement as compared to reference method.

40. Method of measuring Analytic Sensitivity
By two steps; determination of values obtained by blank values (zero
calibrators) and values obtained by low level positive samples.
Run 20 blanks; if < 3 exceed stated blank value, accept that value
–Run 20 low patient samples “spiked” near the detection limit; if at
least 17 are above the blank value, the detection limit is verified.
The blank and spiked samples are measured 20 times each, the
means and SDs are calculated from the values observed, and the
estimate of detection limit is calculated from.

41. 6- Analytic specificity (Interferences)
‫أخرى‬ ‫عوامل‬ ‫من‬ ‫تداخل‬ ‫بدون‬ ‫المطلوب‬ ‫العنصر‬ ‫قياس‬ ‫فقط‬ ‫االختبار‬ ‫تخصصية‬
Analytical Specificity:– is the ability of a method to detect only the
analyte it is designed to detect.
The interference experiment is performed to estimate the systematic
error caused by other materials that may be present in the specimen
being analyzed.
- It is Negative agreement as compared to reference method.
- No approved protocol.
- Manufacturer claims are used

42. - A pair of test samples are prepared for analysis by the method
under study. The first test sample is prepared by adding a solution of
the suspected interfering material (called "interferer,") to a patient
specimen that contains the sought-for analyte. A second sample
tested after dilution of sample but not the interfering substance .

43. Method Validation Summary
• Once the method experiments are complete, summarize the
results in a Method Validation/Verification Summary.
• Clearly state the purpose of the verification, what
platform/method and the number of samples for each
experiment.
• Any discrepant results should be investigated and explained in the
Summary. Test results that show sample problems such as
contamination and degradation should not be used in the
assessment but still listed with an explanation.
• The Summary should also contain a Conclusion stating weather
the instrument study met the acceptance criteria or not and its
suitability for use in the laboratory

44. Carryover (Needed by CBAHI)
• Contamination of sample to sample in the run.
• Batch samples, can occur due to positive control in ELISA.
• Mainly due to incomplete washing
• New instrument
• Changed Sample Probe: Semi-annually.

45. Validation/Verification Guidelines:
FDA cleared/approved Lab developed tests (LDL)
Accuracy (bias) 20-40 samples across AMR At least 40 samples across
AMR; could be > 100
Precision (random error) 2-3 samples at clinical
decision points run daily for
5 days
Run study for 20 days
Reportable range (AMR) 3 points near low end,
midpoint, and high end
Same
Reference range 20 samples 40-60 samples; 120 or
more ideal.

46. Validation and verification:???
Study required Validation (non-FDA
approved)
Verification (FDA
Approved)
Precision At least 20 days 5 days
Accuracy At least 40 samples (better
100 samples)
20-40 samples
Linearity with AMR
Analytical sensitivity
Analytical specificity
Reference range 120 samples Verify 20 samples

47. Test method validation CBAHI standard
LB.10 The laboratory develops a process for test method validation.
LB.10.1 The laboratory implements policies and procedures on
test method validation including:
LB.10.1.1 Verification of accuracy/precision.
LB.10.1.2 Verification of sensitivity (lower detection limit).
LB.10.1.3 Verification of carryover acceptability.
LB.10.1.4 Verification of the Analytic Measurement (AMR)
LB.10.1.5 Approval of the method for clinical use (CRR??).

48. Test Method Validation
Evidence of compliance to (LB 10) standard:
1- Document Reviews:
- Policies, process and procedures on method validation.
2- Documented Evidences
- Records surveyor-selected method confirms compliance with
policies and procedures.
3- Staff Interview
- Senior personnel are knowledgeable about the concept of
method validation.

49. • LB.13 The laboratory has a system for instruments/methods
correlation.
LB.13.1 When the laboratory uses more than one method and/or
instruments to test for a given analyte, the laboratory develops
and implements policies and procedures on correlation to ensure
the following:
LB.13.1.1 The correlation studies are conducted every six months.
LB.13.1.2 There is clear description of the correlation study.
LB.13.1.3 There are clearly defined acceptance criteria.
LB.13.1.4 There is a process for review and approval of the
correlation results.

50. • Evidence of Compliance (LB.13):
• Document Review
- Policies, process and procedures on methods/instrument
correlation.
• Documented Evidences
- Records surveyor-selected methods/instruments confirms
compliance with policies and procedures.
• Staff Interview
- Senior personnel are knowledgeable about the purpose of
methods/instrument correlation.

51. • LB.14 The laboratory has a system for controlling the quality of test
methods.
LB.14.1 The laboratory implements policies and procedures on quality
control of test methods to satisfy the following:
LB.14.1.1 Assignment of performance and review responsibility (control
specimens are handled and tested in the same manner and by the same
laboratory personnel testing patient samples).
LB.14.1.2 Number and frequency of running controls.
LB.14.1.3 Tolerance limits of controls results.
LB.14.1.4 Corrective action to be taken in the event of unacceptable results.
LB.14.2 The laboratory quality control system conforms to the
manufacturer's instructions.

52. •Evidence of compliance to LB 14
• Document review:
- Policies, process and procedures on controlling the quality of test
methods.
- Policies and procedures on controlling the quality of test methods
confirm to the manufacturer instructions.
• Documented Evidences:
- Records surveyor-selected methods confirms compliance with
policies and procedures.
• Staff Interview:
- Laboratory personnel understanding of the purpose of controlling the
quality of test methods.