In the semiconductor industry, Design For Testability (DFT) is an essential part of the architecture and design of components. Software designers on the other hand do not pay much (if any) attention to the testing needs of their code.
In this talk we review some core DFT principles like Built-In Self Test, Test Point Insertion, Fault Modeling and Fault Simulation and map them to software testing. Examples of using DFT to create testable software will be given. DFT fits in especially well with the increasing use of Test Automation and Agile Methodologies.
We hope this talk will empower test leads and engineers with knowledge they can use to get their developer counterparts to modify the application-under-test to significantly increase automation, enhance test coverage, run tests faster and reduce the costs of testing.
2. Agenda
Background and Motivation
DFT in VLSI and hardware design
Issues in Software Testing
DFT for Software Development
Automation and Testability
Some Examples
Cons, Review
3. The Anti-Pattern
If it works,
its the developer,
if not blame QA!
4. VLSI/PCB Testing Issues
Mass produced
Each piece needs to be validated
Complexity
− Number of subsystems
− Amount of logic
Access to internal logic
Testing costs
Cost of recall
7. DFT Principles
Controllability
Observability
− Test Point Insertion
Built-In Self Test (BIST)
Fault Modelling
Fault Simulation
Test Pattern Generation
8.
9.
10. Software Testing
“Testability is a design issue and needs to be
addressed with the design of the rest of
the system. Projects that defer testing to
the later stages of a project will find their
programmers unwilling to consider
testability changes by the time testers
actually roll onto the project and start
making suggestions. Testability takes
cooperation, appreciation and a team
commitment to reliability.”
- Bret Pettichord
11. Cost Of Bugs
Where Found Requirements Architecture Coding System Test Production
Where Introduced
Requirements 1x 3x 5-10x 10x 10x-100x
Architecture 1x 10x 15x 25-100x
Coding 1x 10x 10-25x
12.
13. Software Testability
Suitability: clarity of specifications QA has
Observability: we can only test what is visible
Simplicity: easier design/UI makes testing easier
Controllability: better the control, better the coverage
and automation
Stability: how often it changes
Performance: how fast it works
Diagnosability: writing effective bug reports!
15. Good Automated Tests are
Repeatable
Easy to Write
Easy to Understand
Fast
Way Easier with a Testable Software
16. When/How to Automate?
Manager & Team are commited
Testers/Dev either experienced or interested
in learning to script
Product release cycles managed well
Functionality / UI changes under control
Early/Incremental approaches work best
Build integration, reporting
Each bug results in a automated test
17. Automation As Coding
Automation scripts and frameworks are CODE!
Use tools and scripting languages
Evaluate tools on real problems
Follow development processes
− Spiral/Agile
Use Source Control same as rest of code
These will have bugs as well!
18. Test “oracle”
Expected Result
− Generated once from previous run or
− Manually specified or
− Legacy system
Time adjustment
Results
− Database
− File
− Inline with code
20. Unit Tests
Each module has independent set
Developer written/maintained
setup, teardown
xUnit, TestNG
Runs on build
Can run sub-set on install
TDD
21. Diagnosability
All environments: Test / Staging / Production
Errors result in visual messages
Errors are raised where they occur
Errors can be localized
Details are sufficient to fix issue
Ability to send diag data to dev
22. Logging Module
Log levels (VERBOSE, INFO, WARNING, ERROR, CRITICAL, TIME)
Define CRITICAL/ERROR levels well and use them!
These should result in urgent notifications
Lower levels in Production
Time, App, Component, ThreadId, Message
8/2/2010 14:22pm: Notifier: Pop3: 8242: Sending welcome email to userid 334
Delimited columns for import
Rotation based on Time, Size
23. Built-in Self Test
Inserting test code/interfaces
Set/Reset to bring state to known value
Reporting to get current state
Assertions about values/state
assert(order.billed==true)
“oracle” based regression
24. Some Examples
Web Application
− Software As A Service
− Browser Only UI
− Ajax / Dynamic HTML
Banking (Client-Server)
− Installed Application
− Desktop Client
− Server API
25. Web Application
HTML Page Title
Id values for important divs/controls
Hidden values (non-textual, graphs, tables)
Measurable Ajax responses
Tools: Selenium, Sahi, Watir, WebTest, curl
Logging incl. browser/ip/session cookie
Ability to simulate time zone, language
27. Desktop
Unique identifiers to GUI controls
Key Replay tools, SendKeys
API test suites bypassing UI
Client logs, ability to email
Ability to run db queries in scripts
28. Desktop - 2
Automation Friendly Third Party Controls
− Ability to select cell
− Copy-enable text fields
− Key shortcuts to Forms UI for control focus,
clicking, navigation
− Access by value (tree/list controls)
OCR for images/text consoles
30. Dates
Never use system date directly
Config override
<?xml version="1.0" encoding="utf-8" ?>
<configuration>
<appSettings>
<add key=“SystemDate" value=“$now">
</appSettings>
</configuration>
Nant: XMLPoke
<xmlpoke file="App.config"
xpath="/configuration/appSettings/add[@key = 'SystemDate']/@value"
value="1-1-2009" />
31. Sending Emails
Email subsystem should be a stub in dev
Bypass SMTP: log to file …
Email should be sent only to a whitelist
Use Gmail Id & Atom API
Use Unique Id/Timestamp to distinguish
33. Mocks - 2
Simulate Errors and Exceptions
Provide more logging
Jmock, Dependency Injection
Con:
− Code level
− Needs to change with API
34. Anti-Testability Viewpoints
Security compromise by testability interfaces,
logging
Remove or lock down in prod
Extra coding time
Much lower testing costs, better quality
Privacy issues in logging
Show only partial data
Performance
Almost never an issue in practice!