Today, consumers spend more time on mobile apps than on the web. With this increased demand and paradigm shift toward mobile devices, the role of the software tester is evolving and becoming more complex. Since mobile testing is a relatively new domain, software testers face the challenge of understanding not only what to test but how to test. Clint Sprauve focuses on real world strategies and techniques for mobile app testing including device provisioning, mobile network virtualization, multi-OS platform coverage, and hybrid app testing. Learn how companies across various industries—insurance, finance, and entertainment—are implementing successful mobile testing strategies and techniques to meet this growing challenge. In addition, Clint highlights what is most important when creating a mobile testing strategy for your organization—object recognition options (native, text, and image), mobile app performance, and device security.

1. T10
Concurrent Class
10/3/2013 11:15:00 AM
"Mobile Testing Success: Real
World Strategies and
Techniques"
Presented by:
Clinton Sprauve
HP
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

2. Clint Sprauve
Hewlett-Packard
Clint Sprauve is a senior solutions architect for HP Software. Previously, Clint was the director
of product strategy for the Borland Solutions Division of Micro Focus, where he created
Borland’s mobile strategy for functional test automation. He has more than twenty years of
experience in the software development and quality assurance industry. Clint has served as the
senior product marketing manager for the Silk Testing Solutions at Borland Software and Segue
Software, and served as a senior technical sales engineer for both companies.

3. 9/19/2013
Mobile Testing
Success
Real World Strategies & Techniques
Clint Sprauve
Solutions Architect & Software Innovation
Coach
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
What You’ll Learn Today
Current State of the Mobile Industry
Mobile Opportunities and Challenges
Mobile Performance
Mobile Success – Case Studies
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
1

4. 9/19/2013
Android – Device Fragmentation
According to Google’s own numbers, only 37.9
percent of Android devices are running Jelly Bean,
the latest version of Android. That number is only
slightly higher than that of Android 2.3
Gingerbread, which still commands a 34.1 percent
share in spite of being released in 2010.
source: http://venturebeat.com/2013/07/30/android-fragmention-2013/
3
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Android – Device Fragmentation
“The availability of cheap Android phones (rarely
running the most recent version) means that
[Android phones] have a much greater global
reach than iOS, so app developers have a wider
audience to build for…”
source: http://venturebeat.com/2013/07/30/android-fragmention-2013/
4
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
2

5. 9/19/2013
Why Fragment the OS?
• At it’s core, it’s about choice
• Smartphone manufacturers can
sell a variety of phones
• Different configurations
• Different prices
• Different consumer market
segments
source: http://venturebeat.com/2013/07/30/android-fragmention-2013/
5
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Android vs. iOS
6
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
3

6. 9/19/2013
Mobile creates business disruptions
Location-aware, frictionlesspayment taxis & private cars
Accelerated check deposit
processing from image
Check in, change and monitor
flights
7
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Business challenges
The competition is going mobile
Accelerated time to market
Improving customer interaction
Transforming and streamlining processes
New sales channels
8
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
4

7. 9/19/2013
IT challenges
Chaos and change
Mobile drives IT to face a new reality
New/unknown technology
Accelerate delivery
Complexity
Performance
Risk
9
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Mobile success and failure is very visible
Very poor app
ranking in Google
play
Competitors have
4 star apps
Where is your app?
April 2012
10
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
5

8. 9/19/2013
Could better mobile testing have prevented this?
A very public launch
With very public
quality issues led to …
public apologies and
executive turn over
11
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Why is mobile testing so difficult?
1. User expectations are exceedingly high… it must work right the first time
-
Incredibly competitive market
2. Real devices are critical
-
Test results can be dependent on device, OS, and network provider
3. Speed and velocity is essential
-
Pace of change in mobile platforms and devices is incredibly fast
Mobile development projects must be agile in fast and frequent sprints
4. Mobile is not an isolated effort.
-
12
Must be integrated in normal deliver processes
Full lifecycle approach leads to quality outcomes
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
6

9. 9/19/2013
Key considerations for mobile testing success
Five dimensions of mobile testing
Reality
Real
smartphones
Automation
Device
agnostic
Real Tablets
Real mobile
service
13
Flexible and
robust
Test device
management
Cloud based
service
Shared,
remote
access
Secure,
centralized
ALM
integration
Enterprise
visibility
End to end
lifecycle
Development
Functional
Short
learning
curve
Performance
Monitoring
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Is this your mobile test lab?
Real devices, but how will it scale?
•
•
•
•
•
•
•
14
Automation
Security
Location
Change mgt
Governance
Risk exposure
Cost
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
7

10. 9/19/2013
Mobile Performance
Why mobile performance has become a competitive differentiator
Mobile applications are everywhere
Impacting:
• Front and back-end economics
• Loss of revenue
• Loss of customer
• Available anytime, anywhere
• Significant business opportunity
• Additional sales channel
• Analyst influence
15
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
What happens when customers “struggle” on a
mobile device? (TeaLeaf Industries, 2011)
You
43% Abandon the
transaction, try later
on computer
16% Become more
likely to prefer a
competitor
14% Log a
complaint with
customer service
12% Go directly to a
competitor
Your
Competitor
16
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
8

11. 9/19/2013
Network Virtualization – Reduce Mobile Risk
By not considering actual network conditions, it can be a trap!
Without real
network conditions
Real network conditions
Capacity > 500+ Users
VS
Capacity < 200 Users
Performance test results in a false sense of security.
With real network conditions; Test results accurate and reliable.
17
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
ROI: Value of Network Virtualization
HP/Shunra customer survey:
•Network Virtualization eliminated >15 performance related production incidents per year
• ROI < 90 Days
# of Performance Related
Production Incidents Per Month
Relative Costs to Fix a Problem (very expensive to fix in production)
6
5
5
4
3.7
3
2
1
0
(Steve McConnell’s “Code Complete”, pp. 960 - Microsoft Press)
Results Before vs. After Network Virtualization
*Source: 2011 Shunra for HP Customer Survey
18
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
9

12. 9/19/2013
Functional Testing
Object Recognition Types
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Object Recognition Types
Native Recognition - recognize objects by their native id and work with the actual
components within the application (i.e., text box, text field, button)
Image Recognition - captures what is seen on the screen of the device. It captures the
pictures and images and uses complex algorithms to recognize objects while
executing the test
Optical Character Recognition (OCR) - allows the tester to extract the text from
screen (also known as “screen scrapping”)
20
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
10

13. 9/19/2013
Native Object Recognition
Advantages
Flexible and
Robust
Easier to
maintain
21
Adjust to UI
Changes
quicker
Faster test
execution
Disadvantages
Difficult with
customer
objects
Hard to keep
up with OS
Updates
More coding
knowledge
required
Multi-OS Updates
hinder
automation
activities
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Image Recognition
Advantages
OS
Independent
Easier to build
22
Portable
Works with
what you see
on the screen
Disadvantages
Slow
Execution
Difficult to
maintain
Increased
test
maintenance
Reuse is
extremely
difficult
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
11

14. 9/19/2013
Optical Character Recognition (OCR)
Advantages
OS
Independent
Portable
Easier to build
Works with
what you see
on the screen
Disadvantages
Slow
Execution
Difficult to
maintain
Increased
test
maintenance
Reuse is
extremely
difficult
Not always
dealing with
text
23
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Case Study
Large US Airline
Challenges: Struggled with automation of mobile testing, leveraging existing QA
processes, tools & knowhow
Engagement
• Started with a private cloud & function test automation
• Added complementary public cloud - device coverage.
• Performance plans
Currently: Extremely fast ramp up, Running regression, functional, automation,
offshore testing team.
24
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
12

15. 9/19/2013
Case Study
Large US Financial Services Firm
Challenges: Handset procurement and selection, regression testing around new
versions and new devices, utilizing offshore resources for domestic US networks.
Engagement
• Started with private cloud and functional test based automation
• Added complementary public cloud - device coverage.
• Extended and doubled the capacity recently
• Performance testing plans
Currently: Running regression, functional automation, hundreds of scripts, constant
execution, dedicated team.
25
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Summary
• Take time to understand your company’s
mobile strategy
• Review your team’s skill set
• Performance is as important as design
• Seek help from mobile vendors
• Mobile Testing is complex, but you can be
successful
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
13

16. 9/19/2013
Additional Resources
Resource
URL
Mobile Testing @ HP .com
www.hp.com/go/mobiletesting
Blog and Community
www.hp.com/go/thefutureoftesting
HP Mobile
Testing
27
HP Unified
Functional Testing
HP Performance
Center
HP LoadRunner
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Thank you
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
14