Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.

Validation of lab instruments and quantitative test methods

294 visualizaciones

Publicado el

This lecture shows the procedures applied when going to validate your laboratory instruments and quantitative test methods also either FDA approved or laboratory developed tests.

Publicado en: Salud y medicina
  • Sé el primero en comentar

Validation of lab instruments and quantitative test methods

  1. 1. Validation of Lab Instruments and Quantitative Test Methods By: Dr Mostafa Mahmoud MD, Consultant Microbiologist Labs & Blood Banks Admin, Riyadh, MOH Associate Professor of Medical Microbiology & Immunology Faculty of Medicine – Ain Shams University
  2. 2. Method Evaluation VALIDATION: - Establishing standards performance through a defined process. Long studies -For Non- FDA approved testing. VERIFICATION: - Short studies to demonstrate that a test performs in significant compliance to previously established claims. - For FDA approved testing.
  3. 3. I- Equipment validation: DEFINTION: Demonstrate that equipment used in validation studies is suitable for use and is comparable to equipment used for routine analysis. The equipment must be calibrated, qualified, and maintained properly. Components of equipment validation: • Calibration • Qualifications Installation Qualification Operation Qualification Performance Qualification • Routine maintenance performed- proper working order
  4. 4. Instrument Validation – Experiments 1. Accuracy (correlation or comparison) 2. Precision (Replication) 3. Analytical Sensitivity 4. Analytical Specificity 5. Linearity (to verify reportable range (AMR)
  5. 5. When and what to do for instrument validation? 1. New Instrument of a different make or model of current instrument – Must be validated for all method performance specifications including: accuracy, precision, analytic sensitivity, specificity and reportable range. • 2. Additional Instruments of same make & model as the current instrument- Each instrument must be validated separately. If several instruments are validated/verified
  6. 6. 2. Additional Instruments of same make & model as the current instrument- Each instrument must be validated separately. If several instruments are validated/verified at the same time only one validation is needed. Each instrument must be validated for method performance specifications including: accuracy, precision, reference range and reportable range (AMR). a- Accuracy may be verified for the additional instrument by comparison study with the instrument currently in-use (15- 20 samples). b- No separate reference range study is needed for 2nd instrument, assuming comparison study showed absence of significant bias.
  7. 7. 3. Instruments that have been moved from one location to another in the laboratory - Must be validated for method performance specifications including: accuracy, precision and reportable range (AMR).
  8. 8. Instrument Validation Summary • Once the method experiments are complete, summarize the results in a Method Validation/Verification Summary. • Clearly state the purpose of the verification, what platform/method and the number of samples for each experiment. • Any discrepant results should be investigated and explained in the Summary. Test results that show sample problems such as contamination and degradation should not be used in the assessment but still listed with an explanation. • The Summary should also contain a Conclusion stating weather the instrument study met the acceptance criteria or not and its suitability for use in the laboratory
  9. 9. II- Laboratory Test Validation CAP Performance Specifications for Analytic (Test) Validation: A- Analytic accuracy. B- Precision. C- Analytic sensitivity (limit of detection, LOD). D- Analytic specificity (interferences). E- Reportable range. F- Reference range
  10. 10. When to perform method evaluation studies? 1- Presence of perennial problem by quality control study. 2- When a new different diagnostic kit is to be used. 3- When the used method is changed by adding new biological material.
  11. 11. A- Quantitative Methods validation I- FDA cleared or approved methods: Each laboratory that introduces an unmodified, FDA-cleared or approved test system must demonstrate that it can obtain performance specifications comparable to those established by the manufacturer for the following performance characteristics before reporting patient test results: Accuracy, Precision, Reportable Range of the test results and verification that the manufacturer's reference intervals (normal values) are appropriate for the laboratory's patient population.
  12. 12. 1- Analytic Accuracy ‫والحقيقية‬ ‫المقاسة‬ ‫النتيجة‬ ‫بين‬ ‫التوافق‬ • Definition: agreement between test result and “true” result. The difference can be described as the Systemic error (inaccuracy, or bias) in the method. • Systemic error = inaccuracy (Constant error – & Proportional error) Method: - The comparison method by testing 20 (20-40) samples within the testing range by the new and reference method. - The method comparison experiment for accuracy is recommended to be done over a minimum of 5 days. Continue for another 5 days if discrepancies are observed. • Test material sample can include: calibrators/controls, reference material, proficiency testing material with known values - If bias found in 20 samples use larger samples (100 samples)
  13. 13. • Prepare a comparison plot of all the data to assess the range, outliers, and linearity.
  14. 14. Recovery Experiment: • In the absence of a reliable comparison method, recovery studies are used better to use a method near the gold standard one. • The recovery experiment is performed to estimate proportional systematic error (difference of error increases as the concentration of the analyte increases. ).
  15. 15. Statistics: Accuracy / Bias (= systematic error): • Run comparison of methods study (test method vs. reference method, laboratory’s previous method, or manufacturer’s results, etc.). The line of best fit (calculated using a statistics program) provides the linear regression equation Y = a + bX. • Calculate correlation coefficient “r”. • If “r” is high (≥ 0.99), use the regression line to find the bias at analyte concentrations that correspond to critical decision points (ex. glucose: 126 mg/dL). • If “r” < 0.975, the regression equation will not be reliable; use paired t- test to determine if a bias is present at the mean of the data. • “r” should not be used to determine the acceptability of a new method. “r” measures how well the results from the 2 methods correlate
  16. 16. What is Systematic Error? • Systematic error (change) is a change that is always in one direction (low or High) and will cause a shift in the mean value. • Systematic error or change, is associated with a change in accuracy. • Some sources of systematic error are however unavoidable.
  17. 17. Types of Systematic Error • Constant: an error that is always in the same direction and of the same magnitude even as the concentration of analytes changes. • Proportional: an error that is always in one direction and the magnitude of which is a percentage of the concentration of the analytes being measured.
  18. 18. Causes of Systematic Errors 1. Changes in reagent lot number 2- Changes in the instrument. 2- Changes in calibration
  19. 19. 2- Analytic Precision (Reproducibility) ‫ال‬ ‫اختلف‬ ‫لو‬ ‫حتى‬ ‫النتيجة‬ ‫نفس‬ ‫تعطي‬ ‫مختلفة‬ ‫أوقات‬ ‫في‬ ‫اإلعادة‬‫فني‬ • Definition: Precision is the repeatability. The ability of the laboratory to duplicate results time after time on different days and with different operators. • Imprecision = Random Analytical Error (expressed in CV% from the calculated standard deviation SD and mean). • Precision is measure of Random Error (fluctuation of measured values from their mean due to random factors). • Increase in the random error leads to variation of result above or below the mean value.
  20. 20. Method of measuring Precision: • patient samples are the first choice followed by control material and reference solutions. • Repeat measurements of samples at varying concentrations, within- run and between run over a period of time should be performed by:- 1- Use a minimum of 2-3 samples near each medical decision levels run for 3-5 replicates over 5 days will provide sufficient data for within- run and between-run components to estimate precision. 2- Measure precision by dividing the obtained results over the known results multiplied by 100. 3- Use spreadsheet to calculate the SD and compare to the manufacturer claims by using CV%.
  21. 21. • The laboratory should verify the manufacturer’s claim for precision by using the F-test (CLIA), as follows: • Use F test to see if variance (=SD^2) of test method is statistically different from old method, or claim of manufacturer • Example of how to use the F test: • Obtain the expected SD and number of measurements used in the replication experiment from the manufacturer’s claims (usually included in the instrument documentation), e.g., SD =3 mg/dL based on 31 measurements. • Obtain the SD and number of measurements from the replication experiment, e.g., SD = 4 mg/dL based on 21 measurements. • Calculate the F-value, larger SD squared divided by smaller SD squared, i.e., (4)2/(3)2 = 16/9 = 1.78.
  22. 22. • Look up the critical F-value for 20 degrees of freedom (df=N-1) in the numerator and 30 df in the denominator in the F-table, where the value found should be 1.93. • In this case, the calculated-F is less than the critical-F, which indicates there is no real difference between the SD observed in the laboratory and the SD claimed by the manufacturer. • Conclusion – the manufacturer’s claim is verified when the calculated F value is less than the critical F value.
  23. 23. Random Error
  24. 24. Random Error
  25. 25. Causes of Random error: 1. Inconsistent environmental conditions e.g. temperature humidity etc, 2. Electrical interferences. 3. Inconsistent handling of materials. 4. Fluctuation of volumes used in the testing.
  26. 26. Precision (Replication) Study • Simple Precision: Repeatability (within run) • Complex Precision: Reproducibility (between runs, between days)
  27. 27. 3- Reportable Range • CLIA defines this as the highest and lowest test values that can be analyzed while maintaining accuracy without dilution or concentration. •Method and number of samples: • To verify reportable range, running 3 points near low end, midpoint, and high end using calibration/control/reference matrix appropriate materials. • The AMR must be reverified at least every 6 months, and following changes in major system components or lots of analytically critical reagents • These samples can be combined with the accuracy/precision experiments.
  28. 28. 4- Reference Range (Normal Values) ‫المجتمع‬ ‫في‬ ‫للتحليل‬ ‫الطبيعي‬ ‫المدى‬ 1. No need to define own limits but verify that the adopted one is appropriate for the served populations. 2. The adopted reference ranges may be from suggested manufacturer, reference lab, published articles, neighboring lab or previous limits of own lab. 3. The Reference Range can be verified by testing 20 known normal samples; if no more than 2 results fall outside the manufacturer/published range then that reference range can be considered to be verified.
  29. 29. Method if applicable: • Validation protocol (if needed) include testing 20 samples from healthy individuals; - if < 2 outside the proposed limit the values are validated for reference range. - If > 2 outside, can repeat with another 20, and accept if ≤ 2 outside (not worth repeat if > 5 outside proposed limits). - If still not validated the own reference limit is needed to be done by examining 120 individuals (200 may be used), criteria for exclusion are to be followed.
  30. 30. II. Non-FDA Cleared • Each laboratory that modifies an FDA-cleared or approved test system, or introduces a test system not subject to FDA clearance or approval (including methods developed in-house and standardized methods such as text book procedures), or uses a test system in which performance specifications are not provided by the manufacturer must, before reporting patient test results, establish for each test system the performance specifications for the following performance characteristics, as applicable: • Accuracy, Precision, Analytical sensitivity, Analytical specificity to include interfering substances, Reportable range, Reference intervals, Any other performance characteristics required for test performance, Determine calibration and control procedures and document all of the above.
  31. 31. 1- Accuracy/ Bias (= systematic error): • Same as in FDA-cleared tests except: Most sources recommend comparing at least 40 patient specimens for a Laboratory Developed Test (LDT). • The actual number is less important than the quality of the samples. • The estimate of systematic error is more dependent on wide range of test results than on a large number of samples.
  32. 32. 2- Precision (= random error): • 2 different control materials that represent low and high medical decision concentrations. Analyze the low control and high control at least 20 times each within a run to obtain short term imprecision. • Calculate mean, standard deviation and coefficient of variation for each material. • Determine if short term imprecision is acceptable before proceeding to the long term imprecision experiment.
  33. 33. • Long-term imprecision experiment: • Analyze 1 sample of each of the 2 materials on 20 different days to estimate long-term imprecision. • Calculate the mean, standard deviation, and coefficient of variation for each material. • Determine whether long-term imprecision is acceptable. • Having different operators perform the precision experiment must be done for methods that are operator dependent.
  34. 34. 3- Reportable Range (analytic measurement range= AMR): • Same as above in FDA-cleared tests. The AMR must be reverified at least every 6 months. If the range has not been established, a linearity experiment will have to be performed.
  35. 35. • Linearity Experiment: • Test a series of known dilutions of a highly elevated specimen or patient pool. • The measured or reported test values are compared to the assigned values or to the dilution values, typically by plotting the measured values on the y-axis and the assigned or dilution values on the x-axis. • (CLSI) recommends a minimum of at least 4, preferably 5 different concentration levels.
  36. 36. 4- Reference Range (Normal Values): • Same as above in FDA-cleared tests. When there are no well- established reference intervals are available, additional samples will be required. • CLSI recommends to use a minimum of 120 reference individuals for each group (or subgroup) that needs to be characterized.
  37. 37. 5- Analytic Sensitivity (Detection limit) ‫االختبار‬ ‫بواسطة‬ ‫قياسها‬ ‫يمكن‬ ‫قيمة‬ ‫اقل‬ ‫االختبار‬ ‫حساسية‬ • Called lower detection limit. It is the smallest quantity of an analyte that can be reproducibly distinguished from background levels. • Positive agreement as compared to reference method.
  38. 38. Method of measuring Analytic Sensitivity By two steps; determination of values obtained by blank values (zero calibrators) and values obtained by low level positive samples. Run 20 blanks; if < 3 exceed stated blank value, accept that value –Run 20 low patient samples “spiked” near the detection limit; if at least 17 are above the blank value, the detection limit is verified. The blank and spiked samples are measured 20 times each, the means and SDs are calculated from the values observed, and the estimate of detection limit is calculated from.
  39. 39. 6- Analytic specificity (Interferences) ‫أخرى‬ ‫عوامل‬ ‫من‬ ‫تداخل‬ ‫بدون‬ ‫المطلوب‬ ‫العنصر‬ ‫قياس‬ ‫فقط‬ ‫االختبار‬ ‫تخصصية‬ Analytical Specificity:– is the ability of a method to detect only the analyte it is designed to detect. The interference experiment is performed to estimate the systematic error caused by other materials that may be present in the specimen being analyzed. - It is Negative agreement as compared to reference method. - No approved protocol. - Manufacturer claims are used
  40. 40. - A pair of test samples are prepared for analysis by the method under study. The first test sample is prepared by adding a solution of the suspected interfering material (called "interferer,") to a patient specimen that contains the sought-for analyte. A second sample tested after dilution of sample but not the interfering substance .
  41. 41. Method Validation Summary • Once the method experiments are complete, summarize the results in a Method Validation/Verification Summary. • Clearly state the purpose of the verification, what platform/method and the number of samples for each experiment. • Any discrepant results should be investigated and explained in the Summary. Test results that show sample problems such as contamination and degradation should not be used in the assessment but still listed with an explanation. • The Summary should also contain a Conclusion stating weather the instrument study met the acceptance criteria or not and its suitability for use in the laboratory
  42. 42. Carryover (Needed by CBAHI) • Contamination of sample to sample in the run. • Batch samples, can occur due to positive control in ELISA. • Mainly due to incomplete washing • New instrument • Changed Sample Probe: Semi-annually.
  43. 43. Validation/Verification Guidelines: FDA cleared/approved Lab developed tests (LDL) Accuracy (bias) 20-40 samples across AMR At least 40 samples across AMR; could be > 100 Precision (random error) 2-3 samples at clinical decision points run daily for 5 days Run study for 20 days Reportable range (AMR) 3 points near low end, midpoint, and high end Same Reference range 20 samples 40-60 samples; 120 or more ideal.
  44. 44. Validation and verification:??? Study required Validation (non-FDA approved) Verification (FDA Approved) Precision At least 20 days 5 days Accuracy At least 40 samples (better 100 samples) 20-40 samples Linearity with AMR Analytical sensitivity Analytical specificity Reference range 120 samples Verify 20 samples
  45. 45. Test method validation CBAHI standard LB.10 The laboratory develops a process for test method validation. LB.10.1 The laboratory implements policies and procedures on test method validation including: LB.10.1.1 Verification of accuracy/precision. LB.10.1.2 Verification of sensitivity (lower detection limit). LB.10.1.3 Verification of carryover acceptability. LB.10.1.4 Verification of the Analytic Measurement (AMR) LB.10.1.5 Approval of the method for clinical use (CRR??).
  46. 46. Test Method Validation Evidence of compliance to (LB 10) standard: 1- Document Reviews: - Policies, process and procedures on method validation. 2- Documented Evidences - Records surveyor-selected method confirms compliance with policies and procedures. 3- Staff Interview - Senior personnel are knowledgeable about the concept of method validation.
  47. 47. • LB.13 The laboratory has a system for instruments/methods correlation. LB.13.1 When the laboratory uses more than one method and/or instruments to test for a given analyte, the laboratory develops and implements policies and procedures on correlation to ensure the following: LB.13.1.1 The correlation studies are conducted every six months. LB.13.1.2 There is clear description of the correlation study. LB.13.1.3 There are clearly defined acceptance criteria. LB.13.1.4 There is a process for review and approval of the correlation results.
  48. 48. • Evidence of Compliance (LB.13): • Document Review - Policies, process and procedures on methods/instrument correlation. • Documented Evidences - Records surveyor-selected methods/instruments confirms compliance with policies and procedures. • Staff Interview - Senior personnel are knowledgeable about the purpose of methods/instrument correlation.
  49. 49. • LB.14 The laboratory has a system for controlling the quality of test methods. LB.14.1 The laboratory implements policies and procedures on quality control of test methods to satisfy the following: LB.14.1.1 Assignment of performance and review responsibility (control specimens are handled and tested in the same manner and by the same laboratory personnel testing patient samples). LB.14.1.2 Number and frequency of running controls. LB.14.1.3 Tolerance limits of controls results. LB.14.1.4 Corrective action to be taken in the event of unacceptable results. LB.14.2 The laboratory quality control system conforms to the manufacturer's instructions.
  50. 50. •Evidence of compliance to LB 14 • Document review: - Policies, process and procedures on controlling the quality of test methods. - Policies and procedures on controlling the quality of test methods confirm to the manufacturer instructions. • Documented Evidences: - Records surveyor-selected methods confirms compliance with policies and procedures. • Staff Interview: - Laboratory personnel understanding of the purpose of controlling the quality of test methods.

×