Regulated software is consequential software, subject to authoritative outside review. Even experienced software testers can have an Alice-In-Wonderland feeling the first time they test a regulated product. Many aspects of software testing are surprisingly exactly the same as testing unregulated software, while others are surprisingly completely different. Why is that? What parts are surprisingly similar or different? Griffin shares experiences of testing FDA regulated systems, and similar software (e.g., financial systems, aircraft controls, insurance, and online gaming) – drawing examples from the participants of the Workshop on Regulated Software Testing (WREST), and himself. We examine the what, why, and how certain software is regulated - and the duties imposed on companies and individuals. We highlight some of the surprising aspects, such as: test design and execution; tools and automation; accuracy and rigor; traceability and authorization; evidence and record keeping; the halting problem; process standardization and predictability; human variability and adaptability; technology choices; morality and ethics; and business pressures. Leave with more insight into and less surprise about the challenges of testing regulated software.

1. Regulated
Software Testing
March 2014 © 2014 Congruent Compliance LLC 1

2. Griffin Jones
Consultant
Agile / Testing / Regulatory
March 2014 © 2014 Congruent Compliance LLC 2
Why Testing Is Not Dead …
… In this Context
• Because there are “Survival Rules”
associated with these systems
• these systems need strong “harsh tests”
(in a Karl Popper way)
• not just simplistic, mindless checking

3. March 2014 © 2014 Congruent Compliance LLC 3
THIS PRESENTATION …
Outline
• What/Why/How Software is Regulated
• 11 Surprising Aspects of Regulated SW
• Same as Unregulated
• Different from Unregulated
• Cognitive Dissonance
• Questions

4. March 2014 © 2014 Congruent Compliance LLC 4
WHAT IS REGULATED
SOFTWARE?
Working Definition
• Software subject to review by an
internal or external regulatory body
WREST (Workshop on Regulated Software Testing)
• Share ideas and provide a forum for
people who are interested in improving
the testing of regulated systems

5. March 2014 © 2014 Congruent Compliance LLC 5
WHY I CARE
AND WHY YOU SHOULD
Authorization
• Criminal, Civil, and Administrative Law
The Regulators are Police
• Inspect, Search, Question, Confiscate,
Fine, Debarment
• Deputies
• Self-Policing

6. March 2014 © 2014 Congruent Compliance LLC 6
WELCOME TO WONDERLAND
“Skill, Pill, and Will”
• Consequences
• Scrutiny of Your Work
• Moral Hazard
• “Just World” Hypothesis
• Good decisions can still
have bad outcomes

7. March 2014 © 2014 Congruent Compliance LLC 7
BUT … SURPRISE!
cts of Software Testing
• Some Aspects are Surprisingly Similar
• Some Aspects are Surprisingly Different
Unregulated versus Regulated,
Aspects of Software Testing

8. March 2014 © 2014 Congruent Compliance LLC 8
11 ASPECTS OF
SOFTWARE TESTING
Surprisingly
Similar
Surprisingly
Different

9. March 2014 © 2014 Congruent Compliance LLC 9
ASPECT 1 A …
Test Design and Execution
• Schools of Testing:
Quality and Standards
• Test Design Patterns
[http://kaner.com/?p=100]
• Checking / Demonstration
of Requirements

10. March 2014 © 2014 Congruent Compliance LLC 10
ASPECT 1 B …
Test Design and Execution
• Failure Mode and Effects
Analysis
• Problem Investigation

11. March 2014 © 2014 Congruent Compliance LLC 11
ASPECT 2 …
Tools and Automation
• Commercial Tools
• Open-Source
• Customized Jigs
• Testability Built into
the Product
• Lack of Tools

12. March 2014 © 2014 Congruent Compliance LLC 12
Accuracy and Rigor
• “If it is not documented, it
didn’t happen”
• Attention to Detail is a Tell
• Seriousness, Under Control,
Honest, Professional
• A Proactive, Self-Healing
Culture
ASPECT 3 …

13. March 2014 © 2014 Congruent Compliance LLC 13
ASPECT 4 …
Traceability and Authorization
• Under Control
• Accountability

14. March 2014 © 2014 Congruent Compliance LLC 14
ASPECT 5 …
Evidence and Record Keeping
• Reasonably recreate
“the project” [Design History File]
• Retained for expected
life of the device (after
last sale), plus two
years

15. March 2014 © 2014 Congruent Compliance LLC 15
ASPECT 6 …
The Halting Problem
• How and on what basis
does someone decide they
have enough information
to stop testing?

16. March 2014 © 2014 Congruent Compliance LLC 16
ASPECT 7 …
Process Standardization
and Predictability
• Process Police
• Measurement Obsession
• Mechanization
• Reductionism versus
Holistic System Thinking

17. March 2014 © 2014 Congruent Compliance LLC 17
ASPECT 8 …
Human Variability and
Adaptability
• Ordinary, flawed people
• Their water also boils at 100 oC
• Dependent on their skill,
experience and judgment

18. March 2014 © 2014 Congruent Compliance LLC 18
ASPECT 9 …
Technology Choices
• Old but well understood technology; or
• Just invented technology

19. March 2014 © 2014 Congruent Compliance LLC 19
ASPECT 10 …
Morality and Ethics
• Harms Innocent and Vulnerable
• Destroys the Business
• Becomes Public
• Not obvious: Online Games
• “Can you handle the Truth?”

20. March 2014 © 2014 Congruent Compliance LLC 20
ASPECT 11
Business Pressures
• Competitive Markets
• Reimbursement Codes
• Regulated Marketing
• Regulatory Uncertainty

21. March 2014 © 2014 Congruent Compliance LLC 21
THE BIG TAKE AWAY …
USE COGNITIVE DISSONANCE
When working on Unregulated SW …
• I constantly ask myself:
“Would we be doing this for regulated
SW?”
• Reconsider the purpose and form of
activities where you answer “NO!”

22. March 2014 © 2014 Congruent Compliance LLC 22
… THE BIG TAKE AWAY
USE COGNITIVE DISSONANCE
When working on Regulated SW …
• I constantly ask myself:
“What basic unregulated industry
practices are we are not doing?”
• Reconsider adopting those practices
• How will you justify not doing them?

23. March 2014 © 2014 Congruent Compliance LLC 23
Regulated Software Testing
• What/Why/How Software is Regulated
• 11 Surprising Aspects of Regulated SW
• Same and Different from Unregulated
• Test Design and Execution
• Tools and Automation
• Accuracy and Rigor
SUMMARY … A

24. March 2014 © 2014 Congruent Compliance LLC 24
Regulated Software Testing
• 11 Surprising Aspects of Regulated SW
• Traceability and Authorization
• Evidence and Record Keeping
• The Halting Problem
• Process Standardization and
Predictability
SUMMARY … B

25. March 2014 © 2014 Congruent Compliance LLC 25
Regulated Software Testing
• 11 Surprising Aspects of Regulated SW
• Human Variability and Adaptability
• Technology Choices
• Morality and Ethics
• Business Pressures
• Cognitive Dissonance
SUMMARY … C

26. March 2014 © 2014 Congruent Compliance LLC 26
QUESTIONS AND STORIES

27. March 2014 © 2014 Congruent Compliance LLC 27
IMAGE CREDITS
http://www.morguefile.com/archive/#/?q=target&sort=pop&photo_lib=morgueFile
http://www.morguefile.com/archive/#/?q=old%20technology&sort=pop&photo_lib=morgueFile
http://www.morguefile.com/archive/#/?q=key&sort=pop&photo_lib=morgueFile
http://www.flickr.com/photos/bexross/2636921208/in/photostream/
http://en.wikipedia.org/wiki/File:HAL9000.svg
http://upload.wikimedia.org/wikipedia/commons/a/af/All_Gizah_Pyramids.jpg
http://upload.wikimedia.org/wikipedia/commons/9/96/Waymarker_at_Southern_Upland_Way.JPG
http://en.wikipedia.org/wiki/File:Painted_blaze.JPG
http://www.morguefile.com/archive/#/?q=rubber%20duck&sort=pop&photo_lib=morgueFile
http://www.flickr.com/photos/minnesotahistoricalsociety/5494632378/sizes/o/in/photostream/
http://www.ebay.com/itm/Star-Trek-Original-Series-Science-Tricorder-Replica-/190807969198?_trksid=p2054897.l4276
http://upload.wikimedia.org/wikipedia/en/7/72/Alicesadventuresinwonderland1898.jpg
http://upload.wikimedia.org/wikipedia/commons/7/78/Paris_2010_-_Le_Penseur.jpg
http://en.wikipedia.org/wiki/File:Keyboard_typing.png
http://upload.wikimedia.org/wikipedia/commons/b/bc/Library_of_Ashurbanipal_The_Flood_Tablet.jpg
http://en.wikipedia.org/wiki/File:Storage_containers_in_Svalbard_Global_Seed_Vault_01.jpg
http://upload.wikimedia.org/wikipedia/commons/1/19/Sevens_scrum.jpg
http://en.wikipedia.org/wiki/File:July_4_crowd_at_Vienna_Metro_station.jpg
http://blogs.msdn.com/b/geektester/archive/2010/12/30/life-saver-or-life-taker-therac-25-impact-of-poor-testing-testing-tragedies-1-
learning-from-past.aspx
http://en.wikipedia.org/wiki/File:Jack-in-the-box.jpg
http://www.wrestworkshop.com/

28. Thank you for attending this session.
Please fill out the evaluation form.
Griffin Jones
Griffin.Jones@CongruentCompliance.com
March 2014 © 2014 Congruent Compliance LLC 28