Presentation from the EuroSTAR Software Testing Conference in 2013 from Amsterdam, Netherlands

1. Adventures in Test Automation –
Breaking the Boundaries of Regression Testing
John Fodeh, Cognizant Technology Solutions
www.eurostarconferences.com
@esconfs
#esconfs
Insert speaker
picture here, no
more than
150x150 pixels

2. Adventures in Test
Automation
Breaking the Boundaries of
Regression Testing
John Fodeh
Cognizant Technology Solutions
john.fodeh@cognizant.com

3. Outline
• Innovation and testing
• The promise of automation
• Going beyond regression testing
• Automated Test Monkeys
• Model-Based and Behaviour-Driven approaches
• The human factor in automation
| ©2012, Cognizant
3

4. The Promise of Test Automation
• Replacing repetitive and
tedious manual tasks
• Ensuring the consistency and
repeatability of tests
• Performing tests that are
difficult to run manually
• Accelerating test execution
| ©2012, Cognizant
4
* Image source: Rosemont Engineering

5. Traditional Test Automation
| ©2012, Cognizant
5
Typically
Automation of regression tests
Purpose
Testing of a previously tested program following
modification to ensure that defects have not been
introduced or uncovered in unchanged areas of the
software, as a result of the changes made. It is performed
when the software or its environment is changed.
[ISTQB Glossary, v2.1]

6. Characteristics of Traditional Test Automation
• Static
• Simple
• Synchronized
• Vulnerable
| ©2012, Cognizant
6

7. Monkey Testing
| ©2012, Cognizant
7
Monkey Testing refers to the
process of randomly exercising a
software program by means of an
automated test tool

8. Test Monkey
Action List
| ©2012, Cognizant
8
#script file: Test4.asc
CP_KEY FOCUS_UP;
TRACKBALL 0, 64;
KEY L;
KEY 5, LONG;
KEY 8;
CP_KEY FOCUS_UP;
Random
Test Tool
#script file: Test4.asc
KEY L;
KEY 5, LONG;
CP_KEY FOCUS_UP;
#KEY BOX_SIZE_UP
KEY 8;
TRACKBALL 0, 64;
KEY SHIFT, R;
Log file
Application
Under Test
KEY FOCUS_UP;
Utilities

9. Basic Features
A test monkey should:
• Select randomly from input range
• Enter input to AUT through the user interface
• Detect “life signs” of AUT
• Have robust logging facility
• (Re-) start and initialize AUT
• Have limited application knowledge but general
environment awareness
| ©2012, Cognizant
9

10. Using Test Monkeys
• Indicator of application reliability
• Incorporated into daily “smoke” test
• Establish confidence in application
• System test tool
• Evaluation of release candidates
• Previously failed scripts are run again as part of
regression tests
• Focus tests on specific areas of interest
• Combinations of sequential and random scripts
• Random scripts with weight on specific actions
| ©2012, Cognizant
10

11. Metrics – Application Reliability
| ©2012, Cognizant
Stability Trend for 4101
120000
100000
80000
60000
40000
20000
0
Release
Criterion
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
Build no.
Mean number of random operations between
Failures
Random operations Start Criterion Release Stability Criterion
11
Entry
Criterion

12. Added Value (1/2)
• Early testing
• Cost effective
• Negative testing
• Long and complex test runs
• Stability indicator
| ©2012, Cognizant
12

13. Limitations
• Misses obvious defects
• Does not emulate real use situations
• Long tests runs can be difficult to debug
• For a reliable statistical basis, long and multiple test
runs are needed
| ©2012, Cognizant
13
Monkey Testing is a supplement
to your manual and automated
testing - not a replacement!

14. Advanced Test Monkeys
• Wide application knowledge
• Application modeled in state table
• Illegal input to test error handling and recovery
• Emulate real and complex use scenarios
• Effective in finding defects
• Higher development and maintenance cost
| ©2012, Cognizant
14

15. Test Monkey
| ©2012, Cognizant
15
Random
Test Tool
State table
Log file
Application
Under Test
Utilities
Start
State
Action End
State
Script
Start state Action Probability
Input
range End state
Scanning Adjust frequency 45 3 20 Scanning
Adjust brightness 20 1 100 Scanning
Abort 5 Displaying (abort)
Edit preferences 10 Editing preferences
Increment depth 20 Scanning

16. Intelligent Randomness
• Probability Tables
• Focus on selected areas
• Weight on specific use
scenarios
• Continuous expansion and
improvement
• Possibility to exclude
certain parts
| ©2012, Cognizant
16
Enter Text
Select Text
Delete
Copy
Cut
…
Task n
Action
25%
10%
5%
5%
5%
* source: Whitmill, Kelly

17. Model-Based Test Monkeys
MStoadteel table Script
| ©2012, Cognizant
17
Random
Test Tool
Log file
Application
Under Test
Utilities
Start
State
Action End
State

18. Next Step - Behavior-Driven approach
• Based on Behavioral specifications
• User Stories defined using ubiquitous language
• Principles of Test Driven Development
• Collaborative approach for business analysts,
Software development & testing
• Shared tools and process
| ©2012, Cognizant
18

19. People Issues
• Applying test automation involves people
• Your team members are not interchangeable machine
parts
• Take that into account when implementing test
automation
| ©2012, Cognizant
19

20. Summary
• No “one size fits all” solution
• The tools, processes and skills are there
• Maturity is about analyzing the current practice and
finding ways to do evolve
• Are you ready for your innovation journey?
| ©2012, Cognizant
20