Strategies for improving the quality of automated tests. Wednesday Last Track Selenium Conference 2011

1. Testing Tests
Ideas for better quality test software

2. Brian Colfer
brian.colfer@gmail.com
@briancolfer

3. Introduction
• Testing software is subject to error
• False negatives are the main problem
• Use Various strategies to help detect when
there is a problem

4. Defects in tests
• Same source as product defects:
• Misunderstood requirements
• Logic errors

5. Two types
• False positives: type I errors we detect
something when it doesn't exist
• False negatives: type II errors we don't detect
something when it does exit

6. Testing the test
framework
• Not controversial
• Normal test tools and techniques
• Frameworks are normal products

7. Track defects in tests
• Have QA framework and tests as a part of
defect tracking system
• Have a release process for software
framework and tests

8. Type I errors
• Not as much of a worry but costs
• Further examination will catch the error
• Costs in delays and analysis effort

9. Type II errors
• Insidious, you don't know that you don't know
• Costs of the defect: normal user defect costs,
repair costs and test repair costs.
• Deterioration of confidence in software

10. Strategies
• Unit tests
• Reviews
• Quality Control metrics

11. Unit tests
• Unit tests validate that test code is correct not
the application under test
• Use TDD or BDD to write tests
• Run unit tests as part of a CI practice for test
development

12. Reviews
• Step through the code looking for defects
• Make it a frequent practice
• Keep the scope digestible

13. Quality control metrics
• Traditional QC doesn’t apply to software
• QC will look for changes in power and
sensitivity of the testing
• Regression tests are supposed to not find
errors
• Track defects found by automated tests
• Establish a baseline rate of defect detection

14. Multi method testing
• Use fault injection to find test errors and
ungraceful application failure
• Exploratory testing
• Don’t make your tests too DRY