Test Automation: A Roadmap For Sucesss

2008-01-20

Test Automation
A Roadmap For Success

Scott Armstrong

Belfast, February 2010

SQS Group Limited

Agenda

Steps for Success
Automation Tool Selection
Test Selection and Prioritisation
Implementing the Strategy
SQS Case Studies
Questions

© SQS Group Ltd. | Test Automation: A Roadmap For Success | 16th February 2010 2

© SQS Group Limited | Presentation
title | July 2008 | page ‹Nr.› 1

2008-01-20

Steps for Success
Approaching Test Automation
Planning the Framework

© SQS Group Ltd. | Test Automation: A Roadmap For Success | 16th February 2010

Approaching Test Automation
Steps for Success

DESIGN
Build maintainable
tests
Target and prioritise
test areas

PLAN
TOOL
Treat as a
Select with care
Development Project
Pilot tool
Manage Expectations
Manage
Commitment at all
Implementation
levels

AUTOMATION
PROJECT

TEAM TEST MANAGEMENT
Code reuse and Defect Management
collaboration Config Management
Partnership between Information and
test and dev Reporting
Requirements



2008-01-20

Planning the Framework
Steps for Success

What are the Management Information Requirements
Contents
Format
What are the Execution Requirements?
On-demand
Automated Nightly Builds
What technologies are we testing?
Embedded
Operating System
What resources are available?
Hardware
Technical Expertise
Are there cultural or organisational considerations?
Outsourced IT Management – no admin access
Current partnership with tool vendor


Selection Process
Identify Requirements
Buy, Open Source or Build



2008-01-20

Selection Process

Define
Problem
Shortlist
Consider
Define Evaluate
automation
required Decide
as a solution Demo
features
Make Trial
business
case
Constraints


Identify Requirements

Start by evaluating your requirements, not by investigating the
tool market.
Look at the current problems – what has prompted the search for
an automated solution:
Manual testing problems (e.g. too time consuming, error
prone);
No time for regression testing when small changes are made
to the software;
Set-up of test data or test cases is error prone;
Inadequate test documentation;
Don’t know how much of the software has been tested;
Testing is ineffective.
Not all of these problems are best solved by automation.
Rank your problems by importance or current cost to the
organisation.



2008-01-20

Buy, Open Source or Build

• Quickly provides a high level of features
Commercial • Well supported in terms of documentation, help and training
• Often an attractive project to be involved with
Tool • Added cost of license fees

• No license fee cost
Open Source • Limited to support from community forums
• Defects in tool may not be addressed
Tool • Tool may lack a clear roadmap; may not be future-proof

• No license fee cost
Build • Development and maintenance can prove expensive
• Designed to meet project requirements
Bespoke Tool • Lacks full range of features


Test Selection
Test Prioritisation



2008-01-20

Test Selection

Some tests cannot be automated.
Some tests are not worth automating.
May take more time to automate tests than to run them
manually
You may not have to automate everything to get significant
benefits
10% of the tests may be run 90% of the time.
Some types of tests are amenable to automation, some not:
Functional tests are often the most straightforward
Performance tests are notoriously difficult to carry out
manually. These are a good candidate for automation.
Non-functional tests can cover maintainability, portablility,
testability, usability, etc, and can be inappropriate for
automation.


Test Prioritisation

Do not automate too much too soon.
Best practices take time to develop.
Go for ‘quick wins’ – the areas where automation will have the
largest impact most quickly.
Prioritisation factors:
A set of breadth tests (sample each system area);
Tests for the most important functions;
Tests that are easiest to automate;
Tests that will give the quickest payback;
Tests that are run the most often.



2008-01-20

Automation Team Fit
Maintenance Model


Automation Team Fit

Specialist
Group/Tool
Support

Subject
Test Team Matter
Expert
Automation
Team

Development Management
Team Buy-in



2008-01-20

Maintenance Models

Knowledge • Client representative with technical background is
trained during automation project
• Maintenance of current scripts can be supported

Transfer independently for future product releases
• Future new tests can be automated in-house and
added to the regression pack

Managed • Client does not have to be involved in the details of
automation development, execution or maintenance
• The necessary technical skills do not need to exist in-

Service house
• Assured best practice automation and quality


SQS Case Studies
Lloyds TSB Wholesale Banking
Local Utility Company



2008-01-20

Lloyds TSB Wholesale Banking
SQS Case Studies

Manual Testing Challenges Our Engagement
• 600+ Tests and growing • Innovative test automation solution implementation
• 200 man days per release • Highly maintainable and extensible solution
• Concerns over coverage • Successful solution handover to client test team
• Regression pack growth impacted by limited
resources

Lloyds TSB

Customer Success Return on Investment
• Increased test execution efficiency by 85%
“Without “The automated • Reduction in deployed defects >50%
automation, we regression suite • Enabled daily smoke test execution
would not have made my job • Overall ROI of £1.9m over 6 years
gone live” more interesting”


Local Utility Company
SQS Case Studies

Manual Testing Challenges Our Engagement
• 100 test scripts with multiple complex data-driven • Collaborative design of framework specific to client
iterations requirements
• 120 man days regression testing per year • Full knowledge transfer to existing test team for
• Low test coverage – limited to ‘spot checks’ ongoing maintenance and development
• Testing of new functionality impacted by large • Modular framework allows new tests to be built with
regression packs minimal effort

Local Utility

Customer Success Return on Investment
• SQS are now engaged on further automation • Saving of 96 man days of execution per year
projects with the parent company • Data-driven scripts increased test coverage by a
• The existing test team continue to expand and factor of approximately 200
maintain the framework to increase their ROI • Automation solution was ported to a further 2
business units, multiplying the ROI



2008-01-20

SQS Group Limited

The Concourse | Northern Ireland Science Park
Queen‘s Road | Queen‘s Island | Belfast BT3 9DT
Phone: +44 (0) 2890 022 660
E-Mail: scott.armstrong@sqs-uk.com
Internet: www.sqs-uk.com

Thank you for your attention



Test Automation: A Roadmap For Sucesss

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Destacado

Destacado (20)

Similar a Test Automation: A Roadmap For Sucesss

Similar a Test Automation: A Roadmap For Sucesss (20)

Más de David O'Dowd

Más de David O'Dowd (20)

Test Automation: A Roadmap For Sucesss