1. sqs.com
Whitepaper
SQS – the world’s leading specialist in software quality
Why effective Test Automation
drives successful and quality-
driven mobile payments
Answers on how to improve cost effectiveness and reduce time to market
Introduction
The adoption of m-payments is increasing, as is the number
and diversity of mobile devices, and software, upon which
these payment systems operate – for Android alone, over
420 devices and 29 versions of the operating system have
been released since 20071
. For organisations developing
m-payments products and services, testing the many per-
mutations of software and hardware is a logistical and quality
assurance nightmare. While the device and software landscape
increases in complexity, commercial pressure is growing to
reduce time-to-market as retailers, mobile network operators
(MNOs), banks and financial institutions alike chase market
share.
The effective use of test automation is critical for organisations
aspiring to release stable m-payments solutions. For most
organisations, frequent, reliable and cost-effective testing
through test automation is the only effective way to solve the
quality assurance problem for m-payments.
Authors: Faisal Hussain (Senior Consultant)
faisal.hussain@sqs.com
Andrew Brown (Senior Consultant)
andrew.brown@sqs.com
Darryl Kennedy (Principal Consultant)
darryl.kennedy@sqs.com
SQS Group Limited, United Kingdom
Published: March 2014
1) As of 4Q 2013

2. Page 2© SQS Group 2014
Whitepaper | Why effective Test Automation drives successful and quality-driven mobile payments
A recent report on automated testing found that: “The important
conclusion is that there exists a large, untapped pool of economic
benefits that are yet to be claimed through highly automated
testing… Companies will increasingly realise economic benefits
in cost savings, improved business agility, efficiency and quality.
Harvesting economic benefits will continue to drive the
shift toward highly automated testing and away from
manual approaches, as companies continue to push for
higher quality execution and greater business agility at
lower cost.”2
This paper explores the challenges of automating m-payments
system testing, and considers the key elements of an effective
solution that will enable the confident delivery of working
m-payments technologies.
2) Worksoft Market Research Report, 2013 Trends in Automated Testing,
http://www.worksoft.com/files/resources/Worksoft-Research-Report-2013-Trends-in-Automated-Testing.pdf
Mobile Payments
Solution
Demand Complexity
Test Automation

3. Page 3© SQS Group 2014
Whitepaper | Why effective Test Automation drives successful and quality-driven mobile payments
1. Testing more for less, test automation ROI
It has long been recognised in traditional software development
that test automation is a proven technique for running large num-
bers of tests more accurately, reliably and more rapidly than can
be achieved through manual testing. Additionally it enables testing
outside of core hours, as the tests are executed by a computer
rather than by a person. Automation is also important for regression
testing – running large numbers of tests against a code base to
ensure that changes have not introduced new bugs to existing code.
“Software test automation is no longer a luxury with a question-
able return on investment (ROI). Instead, automated functional
testing and automated regression testing have crossed over
to a definite necessity, as mobile apps have grown ever more
important and complex. In short, manual approaches simply
cannot keep up with the growing complexity and demands of
mobile application testing.”3
Figure 1 illustrates the benefits of one of our mobile application
development projects, comparing the savings between manual and
automated test execution. In this case, compared to the cost of
manual testing, project break-even point was at the end of year
one and with savings of 93 % in year two and 300 % in year three.
The impact of test automation is not felt solely in the savings
made on testing resource. Automation allows broader and more
frequent testing of the payments system than manual methods,
leading to more bugs being detected earlier and ultimately, a
better quality product being released to production.
Extensive testing often involves the allocation of subject matter
experts from within the business and by automating tests, the
demands on these experts can be reduced, minimising the impact
on day-to-day business operations.
3) The ROI of automated mobile application testing, Mobile Labs, June 2013,
http://mobilelabsinc.com/the-roi-of-automated-mobile-application-testing/
Figure 1: Mobile test automation ROI
£ 60,000
£ 50,000
£ 40,000
£ 30,000
£ 20,000
£ 10,000
£ 0
Manual Testing Automated Testing
Cumulative cost
1 11 21 31 41 51 61 71
Test cycles executed
SQS test automation for SEPA and new product plat-
forms reduces cost by 80 % for a major international
banking organisation
SQS was asked to conduct automated and functional
testing by a leading multi-national bank. The bank
already had Android and iOS retail banking applications
and was planning to enhance these with SEPA and new
product features.SQS developed an offshore-enabled
test automation infrastructure that provided consistency
of testing across the mobile platforms and operating
systems. The testing automation framework was de-
signed for reuse enabling the bank to apply these test
automation methods to future m-payment projects.
A significant reduction in the testing lifecycle was achieved
through automation and as a result, the project was able to
increase the rate at which new versions of the software
were released. The project’s 2,000 test scripts were
executed in just 30 man-days, reducing the cost by 80 %
compared to the equivalent manual testing.
Live Example

4. Page 4© SQS Group 2014
Whitepaper | Why effective Test Automation drives successful and quality-driven mobile payments
2. M-payments testing challenges
For all its benefits, test automation requires investment and
it is important to understand the business case and ROI, none-
theless, for m-payments, test automation is almost always
necessary because of the:
• Significant number of risks to be mitigated
• Large matrix of diverse test combinations
• Many test data permutations to be exercised
2.1. There are a significant number of
risks to be mitigated in m-payments
M-payments services present a wide variety of risks, which
can only by mitigated by an effective testing solution with the
capacity to achieve high levels of coverage. Trying to mitigate
all of these risks effectively without the use of automation
would be a very complex, time consuming and precarious
endeavour. Risks associated with m-payments include:
• User experience/ confidence
• Security
• Maintainability
• Interoperability
• Performance
2.1.1. User experience / confidence
An m-payments system that consumers do not like, or are
unwilling to adopt, is doomed to failure. The user base for
m-payments systems is enormous, and growing daily, as is
the IT literacy of this user base so any weaknesses in the user
experience will be very apparent and communicated widely.
2.1.2. Security
Security risks are significant due to the nature of the trans-
actions involved and the relative immaturity of the technical
solutions being implemented. The way a payment system
is perceived by its user base is crucial, simply being secure
for m-payments is not enough, the system must be seen to
be secure otherwise any weakness (or perceived weakness)
will lead to concerns about security. Proven weaknesses
in m-payments security for any given provider could have
significant impacts upon the success of its business plan.
For more information about security for m-payments,
see the SQS whitepaper: ‘How will Security Testing help
to reduce risks and build customer confidence in mobile
payments’
2.1.3. Interoperability
M-payments systems need to support a wide range of tech-
nologies, interfaces and contributing systems. They need
to operate seamlessly in a varied and evolving technical
environment. As the Internet of Things grows, the types of
interaction the payment system must support will change
and so too will the need for comprehensive assurance of
m-payments systems. These systems operate in the kind of
varied and diverse environment that has never been seen
before, which presents significant ongoing risks.
2.1.4. Maintainability
As a technology, m-payments is still quite young. New m-payments
systems need to be as maintainable as possible, in order to
remain flexible enough to meet future demand; in development
terms, this means minimising ‘technical debt’. Testing for
maintainability is a significant challenge, and one that is well
served by automation.
2.1.5. Performance
As with user experience there is a growing level of expectation
on the part of the consumer that m-payments systems will
not only work, but that they will work quickly and smoothly
regardless of how heavily loaded the overall system is. Web site
development has shown that any drop in site performance will
see users switch to an alternative very quickly – the same risks
exist for m-payments solutions.

5. Page 5© SQS Group 2014
Whitepaper | Why effective Test Automation drives successful and quality-driven mobile payments
2.2. Large matrix of diverse hardware
and software test combinations
Looking at the market leading iOS and Android compatible
devices, it is apparent that testing device and operating system
combinations alone is becoming a very large and complex task.
Consider the following statistics:
• 29 versions of Android released since 2007
• 420 models of Android mobile phone
• 18 versions of iOS released since 2007
• 6 models of the iPhone have been released since 2007
Disparities between devices, variances in operating systems,
versions and additional manufacturer customisations, together
with differences in internal architecture, screen size and screen
resolution lead to a huge matrix of potential test combinations.
Fully testing 420 device models over 29 Android versions
requires that 12,180 combinations be tested for each test.
Adding the various combinations of iOS devices and operating
system versions introduces a new layer of complexity and the
increasing popularity of Windows Phone adds yet another.
This presents significant challenges to the quality assurance
programme that only automation can resolve.
2.3. Test data permutations
The successful testing of m-payments applications requires
many permutations of the same (or similar) business process
to be executed with different data sets. Test data sets can be
used to check facets of the system such as:
• Acceptable boundary values
• Upper and lower limits
• Customer types
• Localised international payment formats
• Internal, domestic and cross-border payments
Business processes requiring a high volume of different data
sets for testing, are ideal candidates for test automation,
because the technical script maintenance remains low due to
a relatively small test set. In these situations, effective test data
management can yield an extremely high number of test cases.
The business case for the test automation of m-payments
will demonstrate a very high return given the number of data
permutations, which would otherwise need to be executed
across a large set of device and operating system combinations
manually.

6. Page 6© SQS Group 2014
Whitepaper | Why effective Test Automation drives successful and quality-driven mobile payments
3. Test automation solutions for m-payments
Without automated testing it would be nearly impossible to
mitigate the risks inherent in m-payments systems and benefit
realisation would be significantly challenged. So what test
automation techniques and tactics should be considered?
3.1. Test automation strategy and
management
A well thought out test automation strategy is crucial: clear
requirements, rationalisation of devices and limiting the number
of combinations of operating system are imperative for the
intended market. Such prioritisation typically results in a test
strategy that focuses on testing the most recent versions of
important operating systems in combination with popular mobile
hardware. The strategy should consider all risks and identify a
cohesive automated approach to mitigating these risks.
The table below illustrates how many different factors need to
be taken into account during the creation of the mobile test
automation strategy.
The SQS whitepaper ‘Efficient Test Management can
help you to launch mobile payments faster, smarter and
cheaper’ discusses approaches and strategies for test
management in m-payments system development.
3.2. Test environment management
Test environments that enable representative testing of
real-world conditions are essential for all but the simplest
mobile software. However, for Android alone there are more
than 12,000 permutations of device and operating system
versions. The effort required to test these versions is likely to
be uneconomical for most organisations and so, prioritisation
of environments and risk-based testing is required in order to
create a pragmatic testing programme.
• Wide variety of platforms,
devices, operating systems and
versions
• Multiple configuration options on
mobile devices
• Multiple network and access
options (cable, WiFi, UMTS, LTE,
Bluetooth, NFC)
• Use of different programming
languages, debuggers and
emulators
High overall complexity
• Changing requirements for
deployment management
• Increased need for Device
Management processes and
decision-based functionality
• Need to comply with roll-out
plans
Integration into
payment systems
• The many combinations of
operating systems, versions and
updates increases the number of
releases
• Regression testing effort
increases due to the steady
increase offered by mobile
application functionality
Release frequency and
regression testing effort
Table 1: Considerations for developing a mobile test automation strategy

7. Page 7© SQS Group 2014
Whitepaper | Why effective Test Automation drives successful and quality-driven mobile payments
Figure 2: An example of the number of device and operating system releases in just one year (source: perfecto mobile)
Figure 3: Prioritising devices for testing (source: perfecto mobile)
V2.3
Jan July Dec
Android
Product
Version
1 year - 2011
V3.1 V3.2V2.3.3 V2.3.4 V2.3.5 V2.3.7 V2.3.6V3 V4
Market
iOS, MS, BB
Devices
Must
Major
Market

8. Page 8© SQS Group 2014
Whitepaper | Why effective Test Automation drives successful and quality-driven mobile payments
3.3. Behaviour driven development
and test data driven automation
Enhancing automation in mobile device testing through best
practices such as behaviour-driven development (BDD)4
can
lead to further improvements in both quality and efficiency.
BDD helps establish a common understanding about testing
between developers, testers, business analysts and other
business stakeholders. BDD closes the gap between the people
who implement automated testing and the people who best
understand the business requirements. BDD puts control of
test conditions into the hands of business-facing analysts and
subject matter experts (SMEs), reducing the likelihood of
misunderstandings and enabling developers to validate their
code more easily.
Example of shared test data specifications used by both business analysts and test automation specialists.
4) For the purposes of this paper, behaviour-driven development and test-driven development are synonymous.

9. Page 9© SQS Group 2014
Whitepaper | Why effective Test Automation drives successful and quality-driven mobile payments
3.4. Test environment considerations
for automation
In order to conduct automated testing, the test tools employed
must have programmatic access to the device and software to
be tested. The main techniques for establishing tool access to
devices for testing include:
• Jail-breaking - removing the Operating System (OS)
security control by unlocking/rooting/jail-breaking the
mobile device. The benefit of this approach is that complete
control of the mobile device is then possible. However,
the downside is that this approach is viewed as a non-
standard OS, so any testing on a jail broken device may be
invalid, since it is technically on a different OS version than
a majority of end users. This method is also reliant on an
Internet community to provide these unlocking options.
• Emulators – having the mobile device software running in a
simulation of the mobile OS. This removes the requirement
for real devices and the need for remote access. However,
the downside is that it does not fully match an end users
experience.
• Instrumented applications – each of the test automation
tool providers integrates with native applications via
instrumentation, which requires changes to the application
source code. However, for mobile devices, no operating
system can currently be instrumented; so testing is solely on
the application’s functionality and not the behaviour of the
OS when the application is accessed or running. Some types
of instrumentation may also cause a dramatic increase in
execution time.
• Cloud – some vendors are now hosting a suite of mobile
devices that they claim are native (i.e. not unlocked/
rooted/jail-broken) and provide full object recognition and
remote device access. Although this method offers fantastic
coverage for minimal maintenance, subscriptions can be
costly and dedicated access to the latest handsets can
prove expensive. Security of data within the cloud also
needs to be considered and appropriate risk mitigation put
in place.
• Object recognition – many mobile test automation tools
use either native object recognition or image recognition
to identify each of the controls on the screen. The native
approach depends on the application being written to
specifically support object recognition. Image recognition is
typically slower and often multiple scripts are required for
the different operating systems as the look and feel of Apple
and Android is very different. Further work may also be
required to revise the tests when a new operating system is
released. Further, the need to ensure backwards capability
can lead to the number of regression suites rising very
quickly.

10. Page 10© SQS Group 2014
Whitepaper | Why effective Test Automation drives successful and quality-driven mobile payments
Test automation can bring significant cost savings and market
advantage to organisations launching m-payments systems to
their customer base. A recent SQS client in the finance sector
realised the following benefits after introducing test automation
into its mobile development programme:
• Reduced mobile regression testing time – from three
weeks to two days.
• Improved test coverage – over 100 end-to-end test cases
developed across twenty handsets.
• Improved debugging support – defects reported to R&D
included video, screenshot, operating system logs and
device vitals to facilitate understanding of issues across
teams.
• Improved device coverage – ability to support new devices
(over 40) for automated testing.
• Improved quality – automated end-to-end testing of
transactions across mobile and web systems.
• Improved development intelligence via comprehensive
reports – automated mobile test results available in central
location.
As m-payments technologies spread far and wide across
the consumer market the need to adopt automated testing
solutions will only increase. Organisations that do it well and
implement it first will be those who reap the biggest benefits
from it and establish themselves as leaders – automated
testing solutions are key to achieving this.
4. Conclusion
© SQS Software Quality Systems AG, Cologne 2014. All
rights, in particular the rights to distribution, duplication,
translation, reprint and reproduction by photomechanical or
similar means, by photocopy, microfilm or other electronic
processes, as well as the storage in data processing systems,
even in the form of extracts, are reserved to SQS Software
Quality Systems AG.
Irrespective of the care taken in preparing the text, graphics
and programming sequences, no responsibility is taken for
the correctness of the information in this publication.
All liability of the contributors, the editors, the editorial
office or the publisher for any possible inaccuracies and
their consequences is expressly excluded.
The common names, trade names, goods descriptions etc.
mentioned in this publication may be registered brands or
trademarks, even if this is not specifically stated, and as
such may be subject to statutory provisions.
SQS Software Quality Systems AG
Phone: +49 2203 9154-0 | Fax: +49 2203 9154-55
info@sqs.com | www.sqs.com