Best practices paper on the risks, standards and challenges of Health Risk Management- Testing in the Healthcare domain by Devi.K from Siemens. Paper submitted during QAI's 12th International Software Testing Conference

1. Testing in Healthcare Domain :
Health Risk Management and Challenges
Page 1
12th STC: 2012
Devi.K
Oct 20 2012

2. Agenda
Overview
What is Health Risk Mitigation Testing? –Risks , Standards
Risk Management Process Flow
Page 2
Risk Mitigation Tests - Unique to Healthcare Device
Challenges faced during testing Healthcare Devices
Conclusion

3. Overview
Is Testing of a Healthcare device software any
different?
Yes : Device software follows same principles of development and testing as
any other software device
Page 3
No : Includes special tests (hereon called “Health Risk Mitigation Tests” to
ensure that the software is safe for use on humans for any diagnostic or
therapeutic value

4. What is Health Risk Mitigation Testing?
Tests Performed in order to verify the mitigation of different health and safety
risks with the software
Increased Radiation Exposure & Electrical shock, Injuries,
Extreme temperature contacts,
clinical misdiagnosis,
increased exposure to acoustic power
Governed by different regulatory authorities & standards
Page 4
Governed by different regulatory authorities & standards
Regulatory Authorities: FDA, IEC, AIUM/NEMA
International Safety Standards:
Safety and Reg. Standards: safe for human use(FDA 21 CFR, IEC
62304 etc)
Communication Standards : exchange of images and other Patient info. (
based on standards DICOM, HL7, IHE)
Part of a Risk management process

5. Risk Management Process Flow
Hazard
Identification
Safety
Classification
Risk Estimation
Implementation Verification
Requirements
for Risk
Mitigation
Page 5
Implementation
of Risk
Mitigation
Measures
Testing of
Implemented
Risk Mitigations
Verification
Reports for Risk
Mitigation
Measures

6. Image Quality and Calibration Tests
performed to verify the Image obtained
to mitigate the risk of Clinical Misdiagnosis due to Poor Image Quality &/or
Incorrect Image Calibration
performed using a Phantom or a Simulator
Data Integrity and Connectivity Tests
Risk Mitigation Tests - Unique to Healthcare Device
Page 6
Data Integrity and Connectivity Tests
performed to verify that patient information and data are not lost when data
is saved on the system memory or transferred between different network
devices or to various media devices like USB, External Hard Drives etc.
performed to mitigate each of the following risks.
Undetected Loss of Patient Data due to irrecoverable failure or power
outage or data corruption
Corrupted Patient Data and Reports
Mix Up between data of different Patients
contd…

7. Thermal and Acoustic Power Management Tests
Performed to mitigate each of the following risks:
Injury to Patient due to extreme temperature
Injury to Patient due to Increased acoustic power
Verification of Measurements and Calculations
Performed to mitigate each of the following risks:
Risk Mitigation Tests - Unique to Healthcare Device (contd…)
Page 7
Performed to mitigate each of the following risks:
Incorrect Diagnosis due to Inaccurate Analytical results and
reports
Tests identified based on Equivalence class Partitioning of over 1000
measurements and calculations that required to be verified
Golden Dataset based tests for measurements that have a image dependency

8. Challenges When Testing a HealthCare Device
Common Test methodologies listed below did not provide the desired results; they
had to be modified for testing of medical device software.
Regression testing
Stability testing
Page 8
Exploratory testing

9. Challenges When Testing a HealthCare Device
Regression testing
Problem: Regression tests did not always cover safety critical areas
Solution: HRE-Health Risk Evaluation process of Impact Analysis (based on previous
Experience; clinical scenarios; possible risks and hazard situations)
Advantages: Clinical Interaction scenarios Improve the test coverage
Stability testing
Page 9
Stability testing
Problem: Medical used on patients and high stability
Solution: Established metric MTBF
Advantages: Quantified Improvement in System Stability (MTBF goal = 150 hrs)
Exploratory testing
Problem: defects Missed due to Multiple possible Clinical Workflow /steps
Solution: “Clinical Exploratory Testing; that are ”Typical clinical scenarios based
Advantages: Defects that slipped into Clinical Validation phase reduced by 75%.

10. Testing Healthcare Device software is :
• Similar to any other Software Product under test
• Can be tested by any test engineer with no good
exposure to the
domain
Conclusion
Page 10
domain
• Major aspect that a Healthcare test engineer needs
to always
remember is the humanitarian aspect. That “The
product that
he/she is testing will be used to improve human lives
and it is
safe and effective for intended use.” And, any risk
taken in
testing is a “risk taken with a human life.”

11. THANK YOU
Page 11

12. APPENDIX
Page 12
APPENDIX

13. Risks & Hazards in a Medical Device
Typical Risks & Hazards in a Medical Device :
Increased exposure to Radiation by Patient or Operator
Electrical Shock to Patient or Operator Due to Electrical Hazards
Patient or Operator Injury Due to Mechanical Hazards
Injury to Patient due to extreme temperature
Clinical Misdiagnosis in a patient
Page 13
Clinical Misdiagnosis in a patient
Patient Recall for Re-examination
Injury to Patient due to Increased acoustic power

14. Regulatory Authorities & International standards
Regulatory Authorities :
FDA- stands for Food and Drug Administration
IEC - stands for International Electrotechnical Commission
AIUM/NEMA – AIUM stands for American Institute of Ultrasound in Medicine and
NEMA stands for National Electrical Manufacturers Association- standards for
Acoustic and Thermal Power management in Ultrasound scanner
Page 14
International standards :
Safety and Regulatory Standards- that ensure that the product is safe for use on
humans
Communication Standards - Networking & Communication standards to enable
exchange of medical images and other patient information between different
medical devices & other 3rd party Vendor products.

15. Safety and Regulatory Standards
Some of the International Safety and Regulatory Standards are :
• FDA 21 CFR –Standards for Design validation, Software validation and Risk
analysis and Electronic Records. “FDA 510k” certification implies compliance.
• IEC 62304 -ONLY specific standard for Medical Device Software.
• ISO 13845:2003 –standards for Quality management system specific to
medical device.
• ISO 14971 -standard for “Risk management of medical device”.
Page 15
• ISO 60601- standard for safety and essential performance of medical electrical
equipment and medical electrical systems.
• AIUM/NEMA Standard: by AIUM/NEMA are approved by FDA and IEC for
Thermal and Acoustic power management and the real time display of the
same. (This standard is ONLY applicable to Ultrasound Scanners)
• MDD-provide regulatory requirements to be met to legally place a medical
device on the European market. Devices that comply by MDD are provided
“CE Marking”. CE- stands for "Conformité Européenne" which means
"European Conformity".

16. Communication Standards
Some of the International Standards from this category are:
DICOM – Standard for Digital Imaging and Communications in Medicine-
relates to transfer of patient images.
Health Level Seven (HL7) provides standards for the exchange, integration,
sharing, and retrieval of electronic health information between medical
applications.
Page 16
applications.
IHE-stands for Integrating the Healthcare Enterprise – This is the Integration of
DICOM and HL7 standards to provide a standard for complete Healthcare
workflow solutions.

17. Image Quality and Calibration Tests are :
performed to verify the Image that is generated by an Ultrasound scanner
Need to verify each of the following characteristics:
Image Quality
Correctness in terms of the Image representation
Calibration Accuracy
Risk Mitigation Tests - Image Quality and Calibration
Tests-
Page 17
These tests mitigate each of the following risks:
Clinical Misdiagnosis due to Poor Image Quality
Clinical Misdiagnosis due to Incorrect Image Calibration
performed using a Phantom or a Simulator that mimics the human tissues or
blood flow (figure is of a Human Tissue Mimicking Phantom)

18. Data Integrity Tests :
are performed to mitigate each of the following risks.
Undetected Loss of Patient Data
Loss of data due to irrecoverable failure or power outage.
Loss of data due to Data Corruption (probably due to a Software
Crash)
are performed to verify that patient information and data are not lost when data
Risk Mitigation Tests - Data Integrity Tests
Page 18
are performed to verify that patient information and data are not lost when data
is saved on the system memory.
tested by Simulating a Data Recovery /Crash Recovery mechanism
implementation

19. Connectivity Tests :
are performed to mitigate each of the following risks.
Undetected Loss of Patient Data
Corrupted Patient Data and Reports
Mix Up between data of different Patients
are performed to verify that the patient information and data are not lost when they
are transferred between different network devices or to various media devices like
Risk Mitigation Tests - Connectivity Tests
Page 19
are transferred between different network devices or to various media devices like
USB, External Hard Drives etc.
based on compliance to International Communication standards –DICOM, HL7,
and IHE
are performed using different simulators available that help test the systems to
ensure that they are compliant with the standards
contd…

20. Risk Mitigation Tests - Connectivity Tests (contd.)
A typical Connectivity Workflow Network that needs be simulated and tested in a
lab environment is depicted in the below diagram.
Page 20

21. Thermal and Acoustic Power Management Tests :
performed to ensure Thermal Power (surface temperature of the Transducer)
and acoustic power are within safe limits. These tests mitigate each of the
following risks:
Injury to Patient due to extreme temperature
Injury to Patient due to Increased acoustic power
are System tests that involve the following tests:
Risk Mitigation Tests - Thermal and Acoustic Power
Management Tests
Page 21
are System tests that involve the following tests:
Tests to Thermal Indices (TI) and Mechanical Indices (MI) are always within
the limits as specified by AIUM/NEMA standards.
Tests to ensure the Transducer surface temperature is always within the
limits as specified by AIUM/NEMA standards.
verifies that limits are displayed to the physician as per NEMA standards as
follows :
Tests to ensure that the TI and MI indices and the power transmitted to the
system (called Transmit Power) are displayed to the user under all
conditions.
Tests to ensure that the Transmit Power is system and user-controllable.

22. Verification of Measurements and Calculations::
are performed to ensure that the analysis results and reports provided by the
Ultrasound scanner are accurate. These tests mitigate each of the following risks:
Incorrect Diagnosis due to Inaccurate Analytical results and reports
tests performed based on Equivalence class Partitioning of over 1000
measurements and calculations that required to be verified
Risk Mitigation Tests - Verification of Measurements
and Calculations
Page 22
measurements and calculations that required to be verified
Golden Dataset based Measurement for measurement results that were
generated only for specific anatomical images
mock patient studies are created as Test datasets coined as “Golden
Datasets
contd….

23. The Procedure followed to test based on Golden Datasets is depicted in the
following flow diagram:
Risk Mitigation Tests - Verification of Measurements
and Calculations (contd…)
Page 23

24. Challenges –Regression Testing
Problem: Regression tests identification was challenging since Impact analysis
based on the Code/Design and requirements and based on existing
functionality or based on impact analysis on defects solved was not
effective. It is critical that safety critical areas are covered.
Solution: Impact Analysis and Regression tests based on HRE-Health Risk
Evaluation process that involves:
• Impact Analysis for a feature done based on previous Experience
Page 24
• Impact Analysis for a feature done based on previous Experience
• Impact analysis done based on clinical scenarios that involve the
affected feature
• Impact Analysis considering the possible risks and hazard situations that
may arise in interaction with the impacted feature
• HRE process done with key stakeholders like Clinical Experts, Subject
Matter Experts, Architects and Designers
Advantages:
Clinical Interaction scenarios identified Improve the test coverage

25. Challenges –Stability Testing
Problem: Medical Device will be used on patients for diagnostic data and so it is
extremely important to test and ensure that the device is stable.
Solution: Established metric MTBF as per below steps :
• Created an automated “Clinical Workflow procedure” that consists of typical
Clinical Use cases (called “A Patient Exam”) that mimic everyday workflows in
hospital environments.
• Randomization of each of the Steps that increased possible use cases
Page 25
• Randomization of each of the Steps that increased possible use cases
• MTBF (Mean Time Between Failure) Metric is used to measure the Stability.
MTBF = No. of Test Hours / Total no. of Failures where Failure = incomplete
“exam”
• An MTBF goal set for different stages of development.
• Any failures which brought the MTBF below the goal are resolved on highest
priority during development so as to pull the MTBF back to the Goal.
Advantages: Quantified Improvement in System Stability (goal of 150 hrs of MTBF )
contd…

26. Challenges –Stability Testing(contd…)
An example graph to show how MTBF is closely monitored and tracked on a
daily basis. It is kept under control across all stages of development.
Page 26

27. Challenges –Exploratory Testing
Problem: Multiple Clinical Workflow /steps possible to test a requirement.
Solution:
• Redefined in the Healthcare perspective to be called as “Clinical Exploratory
Testing.”Typical clinical scenarios (called Clinical Workflows) identified and
tested.
• This is planned during all stages of a release.
• Different types of exploratory testing are planned based on the phase of the
Page 27
• Different types of exploratory testing are planned based on the phase of the
project.
Advantages: Understanding the Clinical use case and hence introducing Clinical
Scenario-based exploratory testing, the defects that slipped into Clinical
Validation phase of a release could be reduced by 75%.
contd…..

28. Challenges –Exploratory Testing(contd…)
The types of Exploratory Tests across different release stages are shown in the
below illustration.
Page 28
** Defects are referred to as Charms in the illustration