For Impetus’ White Papers archive, visit- http://www.impetus.com/whitepaper The paper discusses the important factors that need to be kept in mind while designing a robust and successful test automation strategy.

1. Designing a successful Test
Automation strategy:
Connecting the dots
W H I T E P A P E R
Abstract
This white paper discusses how designing an effective test
automation strategy can help address major testing challenges
faced by organizations.
It questions the illusionary benefits of Test Automation and
examines ways in which they can actually be converted into real
ones.
The white paper also discusses the important factors that need
to be kept in mind while designing a robust and successful test
automation strategy. It additionally provides a brief overview of
an automation platform that can help address the test
automation problems that companies encounter.
Impetus Technologies, Inc.
www.impetus.com

2. Designing a successful Test Automation strategy: Connecting the dots
2
Table of Contents
Introduction...........................................................................................................3
Deriving benefits from Testing Automation ..........................................................3
Challenges associated with Test Automation........................................................4
Designing the right automation strategy...............................................................5
The Impetus solution.............................................................................................6
Summary..............................................................................................................10

3. Designing a successful Test Automation strategy: Connecting the dots
3
Introduction
Software Testing is emerging as a critical component of the product life cycle
today. With the fast pace that software development has entered with the
upcoming methodologies and trends, there are increased expectations from
software testing. Considering the pressure on product companies to deliver
high-quality offerings within shorter-and-shorter timelines, the importance of,
and expectations from the function is only expected to increase in the future.
Recent research also supports this assumption. According to another study, in
the year 2004 the number of automated tests that were conducted globally as
against manual tests were a mere five percent. By 2006, however, this number
jumped to 20 percent and today, would be in the region of 50 percent!
Clearly, companies are keen to derive benefits out of test automation. The last
5-6 years have seen an increase in the availability of commercial testing tools.
Customers are also requesting for Open Source tools, thanks to the dynamic and
changeable economic scenario.
The need of the hour is that software testing becomes more mature, strong and
should fit into quicker delivery cycle.
Deriving benefits from Testing Automation
Due to this increase in the expectation from software testing, Test Automation
is gaining popularity day by day. Test automation, as we know, is a process of
writing a computer program to perform testing that would otherwise be done
manually.
Over the last few years organizations have slowly gravitated towards
automation testing. The reasons for this are many. In the initial days of any
product, testing involves manual execution of test suites, that involves various
test cases in a given test cycle. Over time, as the product matures and adds
more features, the test team also keeps on adding more test cases to validate
the functionality that is getting developed. With this addition of test cases,
testing teams start demanding more hours for testing. Eventually all
stakeholders look towards test automation and acknowledge the benefits that
test automation can bring to improvement the testing process. It is felt that
automation will enable quick testing cycles and will keep a check on the testing
costs.
According to a
study conducted by
the US NIST,
software producers
lose 21.2 billion
dollars annually
because of
inadequate testing.

4. Designing a successful Test Automation strategy: Connecting the dots
4
Some of the other stated benefits of test automation include:
Improved test coverage: As the repetitive tests are scripted and can be run
in an unattended mode, the manual test engineers can dedicatedly focus on
the new functionality and unexplored areas of the product.
Reduced test cycle time: With different combinations of browsers, the
single automation suite can be executed, in parallel, in an unattended
mode. This helps in saving manual testers efforts that needs to be put
separately for each browser.
Repetitive execution: The automation suite can be executed any number of
times after creation.
Reliable tests: Tests that perform precisely the same operations each time
they are run, eliminating human error.
Reusable tests: Tests that can run on different versions of the application.
Cost reduction: Regression/Sanity suite can be executed through
automation and at the same time existing resources can be better utilized.
Typically, the first area in software testing that companies usually consider for
test automation in a project is the functional testing via User Interface. It is
believed that the functional UI test cases help in reducing the manual efforts
that are being put in by the testing team and allow the team to focus on new
and critical areas of the product. It is true that automation is the best solution
in this scenario as it promises to save effort and time.
Challenges associated with Test Automation
Several large and mid-sized organizations continue to struggle with existing test
automation activities. The first and foremost reason for this situation is the
insufficient test coverage that is achieved by the existing automated test suites.
As the complexities of applications keep increasing, the percentage of test
coverage achieved using the functional UI automated test cases is not enough.
Technologies like the Cloud or Big Data are also adding to the existing challenges
that prevent companies from achieving better test coverage through test
automation. Therefore, it is a false interpretation that says test coverage will
improve by automating the functional UI tests only.
In the initial stages of test automation, when a cost-benefit analysis is
undertaken, it is assumed that as the team will be involved in automation, RoI
will increase exponentially for the testing activities. The diminishing impacts of
Some studies
indicate that
globally, close to
50 percent of all
test automation
projects do not give
the desired RoI. It
means that the test
automation
outcomes either
fail to deliver on
financial
expectations or do
not satisfy
stakeholder
expectations.

5. Designing a successful Test Automation strategy: Connecting the dots
5
automation are often not considered while doing such calculations. Typically,
what is seen is that once the scripts stabilize along with the product,
stakeholders even stop feeling the use of executing the automation suite on a
regular basis. In fact, the criteria that were decided earlier for automating the
test cases, themselves come under scrutiny.
It is also seen that the cost of maintaining existing test scripts exceeds the initial
investments. This also poses a bigger challenge in sustaining consistent RoI. As
functional UI automation mainly focuses on the regression test suite, the overall
efficiency of the test teams become somewhat constant as the time progresses.
Looking at the above challenges, it is obvious that automating only the
functional UI tests will not be going to yield maximum benefits from
automation. Organizations need to look beyond functional UI test automation
to maximize test coverage and achieve real advantages from automation
activities. The best way to do this is to identify the gaps in the current testing
process, revisit the process, and work to plug the identified loopholes.
It has been found that the only way to correct all automation testing wrongs is
designing the right Automation Strategy for the product.
Designing the right automation strategy
While there are definite advantages of automation in software testing, it is
necessary to use it intelligently and diligently.
As we saw in the last section, one of the biggest challenges facing Test
Automation is Automation scoping, which plays a key role in deciding the fate of
automation activities. Currently, the vision of automation in many of the cases is
limited to the User Interface only.
Let us take a look at the bigger picture of automation, where other important
aspects such as API/web services, non-functional areas like security or
performance and integration based on product and business requirements are
considered. These not only enable the testing team to bring in the dedicated
test focus on all the major layers of the product, but also enable the
identification of critical bugs early in the test cycle.
Companies need to design test solutions that can help their testing teams to
automate test cases in the identified areas. This solution should also optimize
testing activities and integrate them to achieve Business Process automation. It
should be capable enough to run in extended environments and should provide

6. Designing a successful Test Automation strategy: Connecting the dots
6
integration points with multiple automation tools so that each layer gets tested
thoroughly. Finally, the testing solution should also be able to measure coverage
and provide various metrics that help analyze its benefits.
A successful and effective Test Automation Strategy should also address some
of the major challenges of software test automation including:
High maintenance of test scripts
Huge learning curve required to maintain test scripts
Consistent RoI
There are inherent benefits associated with this Test Automation Strategy.
This strategy widens the overall scope of test automation. It not only helps
one validate the functionality from the User Interface, but also enables
introduction of test automation into areas like API, integration, non-
functional etc that are mostly overlooked during automation, in any given
product.
The strategy enables test automation for each and every layer which also
helps in increasing the overall test effectiveness. Hence it helps in achieving
a higher degree of test coverage and enables testing of the product more
thoroughly.
Another benefit is in the area of RoI. Getting consistent RoI has always been
a challenge for any automation initiative. But a robust test automaton
strategy can change this. Having a right test automation strategy enhances
the benefits of the automation activity by maximizing the test coverage and
optimizing the resources involved in the process of automation.
The Impetus solution
The Impetus R&D team is working on a test automation platform that can help
in implementing such a test strategy and address the major test automation
challenges.
Impetus is using iLeap 2.0, a SMART and hybrid test automation platform, that is
based on proven best practices and approaches that includes keyword driven,
data-driven and functional decomposition.

7. Designing a successful Test Automation strategy: Connecting the dots
7
Figure 1 iLeap 2.0 Workflow
To automate test cases, the following steps need to be performed in iLeap 2.0:
The first step is to identify the automation candidates and mark them in the
test case management tool itself to properly categorize the test cases within
the test suites. This information is utilized by iLeap 2.0 to produce the
automation suite.
The testing team must create and store the object repository in accordance
with the format supported by the test tool.
The automated test scripts are written as per the steps/actions that need to
be performed on the application in tabular format. iLeap 2.0 actually
enables engineers to do this using the user interface exposed by the
existing tool.
The next step is to create Test Data. This can be stored in either Excel or
XML or a database, whichever is found to be best from an application
perspective.
Next is defining the execution strategy. In existing test automation
solutions, the automation team needs to re-define the test suites in the
functional testing tool to group and execute the test cases. This is a huge
task in itself as the test suites needs to be modified every time there are any
changes made by the manual testing team. iLeap 2.0 enables the definition
of test suites based on the settings done in the test case management tool.

8. Designing a successful Test Automation strategy: Connecting the dots
8
Therefore, if the test team is making any changes in the test suite
definitions, the test automation suite will always remain up-to-date without
any manual intervention.
Now comes the process of executing the test scripts in the desired
environment. iLeap 2.0 enables the definition of this environment and other
pre-requisites required to execute the test cases in a separate configuration
file.
Once all the settings are done, the test scripts are executed, and iLeap 2.0
enables updation of results in test case management tool directly.
It is essential to follow such a workflow for effective and maximum testing.
iLeap 2.0 is being designed for the widely used functional test automation tools
like Selenium and QTP and can be extended to any of the other popular tools
available in the industry. It is designed in such a way that the testing team can
take out the maximum benefits of the functional testing tool and lowers the
learning curve towards automation.
Furthermore, iLeap 2.0 supports Open Source Testing tools and frameworks
such as TestNG, w3af, autoit, etc., over and beyond the functional testing tools.
In addition to supporting automation of the functional UI test suite, iLeap 2.0 is
mature enough to enable API, web services, integration, UAT and basic security
testing as needed to perform and achieve a complete product testing.

9. Designing a successful Test Automation strategy: Connecting the dots
9
Figure 2 iLeap 2.0 Architecture
The iLeap 2.0 architecture has several components that also provide significant
benefits:
The Object Repository component stores the object/control’s information
such as properties, locater values, etc.
The Test Data component stores the set of data that is needed to perform
the operations on the application under test for single/multiple iterations.
There is the Test case Sequences component that defines the actual test
steps for the test cases that are planned to be executed on any application
under test in tabular format.
iLeap 2.0 has its own set of robust custom libraries that consist of the
custom functions being developed at Impetus.
There are Configuration and Environment Settings that help in initializing
the parameter values that are necessary for any test run like various paths
of framework, application, database or browsers etc.
iLeap 2.0 has an Execution Strategy component that helps in creating the
automation suites based on testing needs. iLeap 2.0 creates and stores the
automation suites as defined in the TCM tool and executes the desired
suite.

10. Designing a successful Test Automation strategy: Connecting the dots
10
There is the Result Reporting component which ensures that the test
execution results are automatically updated in the TCM tool after each run.
It also maintains an in-memory database for the results so that they can be
exported in the form of XML.
Summary
As this white paper indicates, Test Automation definitely needs to go beyond
the boundaries of the User Interface or functional testing. Testing teams need
to focus on designing effective Test Automation Strategies that can help address
the common testing challenges such as inappropriate test coverage and low RoI.
Also, there are major bottlenecks in testing automation, which can be overcome
through automation platforms such as iLeap 2.0. At the same time, iLeap 2.0
defines the best practices in automation and utilizes the existing features of
functional testing tools to develop a robust automation platform. It is important
for automation teams therefore, to familiarize themselves with the automation
capabilities, in order to derive its benefits.
About Impetus
Impetus is a Software Solutions and Services Company with deep
technical maturity that brings you thought leadership, proactive
innovation, and a track record of success. Our Services and Solutions
portfolio includes Carrier grade large systems, Big Data, Cloud,
Enterprise Mobility, and Test and Performance Engineering.
Website: www.impetus.com | Email: inquiry@impetus.com
© 2013 Impetus Technologies, Inc.
All rights reserved. Product and
company names mentioned herein
may be trademarks of their
respective companies. May 2013