In today's dynamic mobile marketplace—where new handsets and mobile operating systems are released every day—your ability to deal with these changes which impact your mobile product is vital. The mobile application lifecycle today must be short; must be of great quality; cover a myriad of handsets with different sizes, layouts, and enhanced capabilities; and, of course, cover as many operating systems as possible. This lifecycle requires a new methodology and approach. Eran Kinsbruner describes the mobile project challenges and provides real life examples of ways to overcome them. Take back the main mobile market trends and forecasts together with the key automation tools available for your use today. Learn the differences between the various mobile cloud and automation tools to help you select the right tool for your project. See how you can ramp up a successful mobile project, avoid the common pitfalls, and shorten the time to market—all while delivering a top-notch quality product.

1. T10
Mobile Testing
5/2/2013 11:15:00 AM
Mobile Testing Methodologies: Trends,
Successes, and Pitfalls
Presented by:
Eran Kinsbruner
Perfecto Mobile
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

2. Eran Kinsbruner
Eran Kinsbruner is director of product marketing at PerfectoMobile, one of the leading mobile cloud and
automation companies. Formerly CTO for mobile testing and Texas Instruments project manager at
Matrix, Eran has been in testing since 1999 with experience that includes managing teams at Qulicke &
Soffa, Sun Microsystems, General Electric, and NeuStar. The co-inventor of a test exclusion automated
mechanism for mobile J2ME testing at Sun Microsystems, Eran has experience in the mobile testing
world. You can find Eran on Facebook, Twitter @ek121268, LinkedIn, and his professional mobile testing
blog at ek121268.wordpress.com.

3. 4/16/2013
Mobile Testing Methodologies Trends,
Successes, and Pitfalls
StarEast - 2013
Enabling Enterprise mobility
Extending Software Quality to Mobile
Perfectomobile
Mobile Testing Center of Excellence Group
Agenda
About me and my company
Mobile Market Trends
Mobile Quality Trends
Mobile Quality Challenges
Mobile Risk Management
Mobile Testing Best Practices
Perfecto Mobile
1

4. 4/16/2013
∙
∙
∙
∙
∙
Founded 2006
80+ employees
Office in Boston, London, Israel
Over 600 customers
Fortune 500 enterprises,
Telcos, SMBs
∙ SaaS Platform + Licenses
∙ Recurring revenue model
Mobile Application Quality
across the mobile application lifecycle
Monitoring
Testing
Automation
ALM
Integration
Offshoring
Continuous
Integration
Public Cloud
Private Cloud
Mobile Market Trends
Perfecto Mobile
2

5. 4/16/2013
Global Mobile Market Stats
Source: nielsen, 2012
Mobile OS market expected
to grow (even more)
FireFox OS
PEAK
KEON
Tizen
Windows
Phone
Samsung/Intel
New
Mobile
OS’s
RIM’s
BlackBerry OS
10
Ubuntu
Z10 & Q10
SailFish
By Jolla
Perfecto Mobile
3

6. 4/16/2013
Cost Of Mobile Quality Rises
Mobile accounts for 11% of U.S e-Commerce
Existing Mobile OS’s – End of 2012
Source:IDC
Samsung accounted for 63 Million out of the 159 Million in Q42012
Perfecto Mobile
4

7. 4/16/2013
Worldwide smartphone vendors Market share
Source: IDC Worldwide Mobile Phone Tracker, January 24, 2013
Market insights – Forecasts and trends
Mobile based shopping expected to grow
by 24% in 2013 (U.S)
Perfecto Mobile
5

8. 4/16/2013
Market trends - 2013
Mobile OS battle continues
•New open source competitors trying to influence the eco system
•Windows Phone and BlackBerry OS 10 – critical year for their positioning in the market
Mobile apps become more complex – impact testing and automation
HTML5 and Mobile web expected to grow significantly
EMM establishment – Processes, Apps, Tools, Roles (CMOO)
Mini-tablets and “Phablets” will successfully blur the lines between device types
Mobile Quality Trends
Perfecto Mobile
6

9. 4/16/2013
World Quality report 2012-2013
2012Mobile Testing is Behind the Curve
Source: CapGemini
Mobile Environment
Fragmented
Dynamic
Complex
Business critical
Source: Google, 2012
Perfecto Mobile
7

10. 4/16/2013
Countless screen resolutions
Source: opensignalmaps, 2012 (Android only)
Consumer Reaction – Less Tolerant than In Desktop
Perfecto Mobile
8

11. 4/16/2013
HTML5 Trend Involves Risks
To help protect y our priv acy , PowerPoint prev ented this external picture from being automatically downloaded. To download and display this picture, click Options in the Message Bar, and then click Enable external content.
Mobile Quality Challenges
Perfecto Mobile
9

12. 4/16/2013
Mobile Introduces various challenges
Performance
Automation
AGILE ALM
Functional
Native
Hybrid
Web
Application Types
On various browsers
Sample Screen Sizes
(Only Smartphone)
QVGA: quarter
VGA (240×320
pixels)
HVGA: half VGA
(320×480 pixels)
WVGA: wide VGA
(480×800 pixels)
FWVGA: full wide
VGA (480×854
pixels)
nHD: one-ninth
high definition
(360×640 pixels)
qHD: one-quarter
high definition
(540×960 pixels)
Various Device/Tablets
OEM’s
Various OS
Types and Versions
2.3.3, 4.0.3 4.3.3, 5.1
4.1.2, 4.2.2 6.1, 6.1.1
Mobile Security Threats
Source: Techcrunch
Perfecto Mobile
10

13. 4/16/2013
Mobile OS Regression bugs – What does it have to do
with my Mobile Testing?
Per Device version
Pre-release testing (iOS6.1.1 Beta)
Apple Bug Reporter
Source: LINK
Android defect database
Mobile Web Objects – Automation Challenges
Perfecto Mobile
11

14. 4/16/2013
Apple iOS – UI Guidelines
Source:
Source:
http://developer.apple.com/library/ios/#
http://developer.apple.com/library/ios/#documentation/UserExperience/Conceptual/Mobil
eHIG/Introduction/Introduction.html
Android OS – UI Guidelines
Source:
http://developer.android.com/design/index.html
Perfecto Mobile
12

15. 4/16/2013
Risks Management
Mobile Quality Risk Analysis
Cost of quality in mobile
grows and impacted by
• Availability
• TTM
• Functionality and UX
• New platform support
(Mobile OS, Mobile devices)
• Previous app version
compatibility
Perfecto Mobile
13

16. 4/16/2013
Mobile end-user Less Tolerant
end-
Listen to the
market
Obviously someone forgot
to do their regression
testing on a Galaxy S3
before releasing this
version to market…
Perfecto Mobile
14

17. 4/16/2013
Device governance
A clear process as part of the “mobile project” lifecycle
Define platforms and categories
Define coverage
Acceptable not to test - You can’t test everything
Define change frequency
Adapt to the market trends as they happen
Define SLA for changes
Define coverage as part of the ALM phase
Coverage visibility reports to project managers and executives
Device Governance is a risk management for mobile projects
Device Governance – Define coverage
You can’t cover everything
But you should define a target, which is the acceptable
risk
iOS is not as simple as you think
At least 3 HW platforms (3GS, 4, 4S, 5)
(iPADs, iPODs)
At least 3 SW platforms
Perfecto Mobile
prev
major
15%
prev
minor
25%
Other
5%
Latest
55%
15

18. 4/16/2013
Enterprises OWNS the risk for Mobile quality
Mobile Malware - Trojans, viruses and rootkits migrating from traditional online banking and
designed specifically for the mobile marketplace. Researchers see an increase in mobile
malware development - in pace with market growth.
ThirdThird-Party Apps - Consumers love their smart phone and tablet applications, but often these
apps come from third parties with questionable security practices. Or worse, the apps are
created by fraudsters and loaded with malware.
Unsecured Wi-Fi - The unsecured wireless network is a toll-free highway for fraudsters to gain
Witollaccess to mobile devices, either to seize control of or gain access to account information.
devices,
information.
User Behavior - Consumers are prone to download third-party apps, use unsecured wireless
thirdapps,
networks, open and click links in SMS text messages and e-mails, and lose their mobile devices.
edevices.
Mobile testing and Hardware (Payment card readers, healthcare units and more)
Uncompromised necessity for testing mobile apps
Should adhere to regulations (GLBA/FFIEC, RDC, FCRA, PCI DSS) around security,
GLBA/FFIEC, RDC, FCRA,
DSS)
authentications, payment and more
Usually support complex business work flows (deposit check via camera)
Multi tier interactions (Databases, trading accounts, other services)
High Availability and Monitoring
Must address mobile end user actions
Using Home and Back button during application run
Network lose during transaction
Roaming during user transaction
Application behavior in various networks
3G, 4G, WiFi
DSS Compliance test case example
Perfecto Mobile
16

19. 4/16/2013
Mobile Is Mandated To Strict Regulations
Source: LINK
Source: Link
Mobile Testing Best Practices
Perfecto Mobile
17

20. 4/16/2013
This is not a mobile test lab!
10 Devices
3 Extra Devices
Many Cables
3 Workstations
One Chair
Building Blocks Of Mobile Testing Strategy
Choose the right tools and platform
Understand your users device and platform choices
Optimize device and platform testing
Automate as much as possible using device agnostic scripting
Use mobile cloud based solutions (Collaborative, Governed, Scalable,
Cost Effective)
Use CD/CI solutions as part of your mobile ALM
Do not re-invent the wheel
reExtend existing ALM tools to Mobile.
Perfecto Mobile
18

21. 4/16/2013
World Quality report 2012-2013
2012-
Source: CapGemini
Building Blocks Of Mobile Testing Strategy
Consider outsource to reduce mobile manual testing costs
Test for performance before releasing to the market
Make sure you do regression on older supported platforms
Do not have your end users do your QA work
Perform pro-active testing on new devices and mobile OS’s
Monitor in production to assure end user experience and
availability
Listen to your end users and act upon their feedback (Markets
have significant impact on your business and branding)
Perfecto Mobile
19

22. 4/16/2013
(True) Mobile
Cloud
Avoid device
management
Real Devices
+ Emulators
Real
smartphones
Manual +
Automation
ALM
Integration
Device
agnostic
Core level
extension
End-toEnd-to-End
Mobile ALM
Development
Functional
Real Tablets
Performance
Secure,
Centralized
Interactive
Emulators
Short learning
curve
Monitoring
Erank@perfectomobile.com
Perfecto Mobile
20

23. 4/16/2013
END
Perfecto Mobile
21