Srinivasan Desikan presented on automation beyond testing tools. A survey found that test automation is underutilized, with only 27% of respondents satisfied. Difficulties included lack of skilled people and time/effort required. Automation focused mainly on regression testing easy cases rather than more complex cases. Tools used were often commercial with little open source use. The quality of automated test suites was low with manual retesting often still needed. Automation could be expanded to areas beyond test execution like test management and reporting to further improve testing.

1. STEP AUTO Colloquium on Test 12/10/2007
Automation ,6th Dec '07
Automation beyond testing tools
Srinivasan Desikan
© 2007 Hewlett-Packard Development Company, L.P.
The information contained herein is subject to change without notice
Question
How many programmers does it take to change
a light bulb?
Automation many times gets the wrong set of people for the
job
• Either a test engineer who can’t program
• Or a Programmer who can’t test
• Or a person who is neither programmer nor a tester,
given to automation saying “Automate if you have nothing
else to do”
December 10,
2 2007
Automation beyond Testing Tools ,
Srinivasan Desikan -HP 1

2. STEP AUTO Colloquium on Test 12/10/2007
Automation ,6th Dec '07
Automation Survey – Why you are not automating?
No belief Not aw are
0 6
Inadequate
Features
33 Expensive
61
The job of automation include understanding where tools can help and where
it can’t…
The job of automation starts in creating value on top of commercially
available tools as any tool will not meet 100% of your requirements…
December 10,
3 2007
Automation Survey – What did you automate?
Easy
10
Suitable
39
Regression
28
All
23
• Only Low-hanging fruits are plugged
• 23% got into “trap” as
• There is no end to testing ; so 100% is not possible
• There may not be ROI if all are automated (Waste of effort)
• We have a long way to go to say “we are automating complex test
cases” that gives us great business value
December 10,
4 2007
Automation beyond Testing Tools ,
Srinivasan Desikan -HP 2

3. STEP AUTO Colloquium on Test 12/10/2007
Automation ,6th Dec '07
Automation Survey – What tools you used?
OS Features 35
In house 45
Open source 42
COTS 78
0 20 40 60 80 100
• 35+45+42+78 = 180%; Therefore people on an average use 1.8 tools
which is not a good number to get a good coverage on automation
• There is heavy dependence on COTS and awareness and usage of
Open source/in-house tools to improve
December 10,
5 2007
Automation Survey – Difficulties in automation
More efforts 29
25
More time 17
35
Skilled people 48
45
Usage 17
0 10 20 30 40 50 60
• Automation is kept in the freezer because it requires more efforts and
more complex
• Lack of training available on automation. Lack of usage knowledge
infact is a great difficulty in automation
• Eventhough industry produced the required numbers for automation,
the skills of people has a long way to go (Quantity Vs Quality)
December 10,
6 2007
Automation beyond Testing Tools ,
Srinivasan Desikan -HP 3

4. STEP AUTO Colloquium on Test 12/10/2007
Automation ,6th Dec '07
Automation Survey – Satisfied with automation
No
0
Not yet High
29 27
Just
44
• Only 27% of people satisfied with automation ; More we do
automation more we get satisfied with it
• This percentage drops further below if we exclude the people we
were part of the automation project ; Business value out of
automation as perceived by mgmt and outside testing team is very
low
December 10,
7 2007
Automation Survey – Quality of automation
• Only 2% of people share their automation suite with people outside the
team (they are not easy to use, and have potential quality issues)
• 95% of people say that “They repeat the tests manually when
automated tests bring up defects”; this shows
− Trust level of test suite is very low
− It defeats the purpose of having automated the tests
− More needs to be done to make the quality of test suite better
− Quality requirements for automation should be better than or equal to those
criteria set for products
• Only 1% people share their test suite with their customers
− Automation is not done with the perspective that “customers also do testing
and it should save their time too”
− Not many realize that automation is yet another business opportunity (Many
of the commercial tools in market today were in-house tools before)
December 10,
8 2007
Automation beyond Testing Tools ,
Srinivasan Desikan -HP 4

5. STEP AUTO Colloquium on Test 12/10/2007
Automation ,6th Dec '07
Automation beyond testing tools & Test cases
• Test reports are automatically prepared by automation scripts
• Test defects are automatically filed in defect tracking system
• Test metrics are updated/created by automation scripts
• The test results are updated automatically by automation
• Traceability matrix updated with mapping to requirements and test
cases
• Keep looking at defect status and selecting test cases automatically for
regression testing (Keeping the track of what defects corresponds to
what test cases)
• Calculating code coverage and test coverage
• Suggesting the improvements in test management
− A list of test cases that were never executed for a long time
− A list of test cases that always failed (Then why execute them again and
again)
− A list of test cases that was not executed for long (to execute them atleast
now or delete them from database)
− December 10, test cases that always passed (Do we need to really execute them
A list of
9 again and again)
2007
Survey results – Where test engineers spend
time?
5% 5% 10% Automation
5%
Test execution
15%
10% Test Setup
Reports
Waiting for build
test case design
10% Test plan
5% Review s
35% Others
• Very little investment on automation
• Automation in lab setup will give better ROI
December 10,
10 2007
Automation beyond Testing Tools ,
Srinivasan Desikan -HP 5

6. STEP AUTO Colloquium on Test 12/10/2007
Automation ,6th Dec '07
References
1. Test automation survey results – Jayakumar. K.R, AmitySoft,
Chennai (Necessary approvals obtained)
2. Book: “Software Testing – Principles and Practices”,
Srinivasan Desikan & Ramesh Gopalaswamy, ISBN:
817758295X
Disclaimer: The views expressed in slides and in the presentation are my own and not that
of my employer
December 10,
11 2007
Questions?
December 10,
12 2007
Automation beyond Testing Tools ,
Srinivasan Desikan -HP 6

7. STEP AUTO Colloquium on Test 12/10/2007
Automation ,6th Dec '07
Thank You !
December 10,
13 2007
Automation beyond Testing Tools ,
Srinivasan Desikan -HP 7