App testing doesn't work if you don't use the mobile devices and platforms your customers are using. We analyzed 360,000 app users and 4,000 device configurations to get the answers you need to test with confidence. In this presentation, you'll learn: » The top mobile device platforms across major geographies » How to decide what mobile devices to develop and test on » How to catch preventable defects that emulators miss

1. Assuring Mobile Test Coverage
Eran Kinsbruner
Gidi Pridor

2. Answering the most common question:
“What should I test on to ensure
efficient market coverage?”

3. v
Mobile Test Coverage Index – Q1 2015
Covers (U.S & EU5): Devices, OS, Screen Sizes, Hardware
Thorough
Market
Analysis
360,000
Real End
User Data
4,000+
Proprietary
Device Data

4. Agenda
Introduction
Principles
The Index
Examples
Best Practices
v
v
v
v
v

5. INTRODUCTION

6. Perfecto mobile
at a glance
The world leader in mobile application continuous
quality helping the world’s biggest brands to
deliver better apps faster
“Perfecto Mobile currently has the
the strongest 3rd party position in
the market” [ Thomas Murphy]

7. Why is it Important?
Keeps crashing my phone. Every time I use
Skype out to call regular phones on my
phone (Nexus 4) crashes and restarts. This
is pretty annoying!
Daniel Waisberg September 27,2013
Broken. No longer works on my
galaxy nexus.
Joshua Smith September 24,2014
Login Issues has been 3 days and
still can’t login to the mobile
app…was able to login to the
site just fine!
Eugene Vygocher September 24,2014

8. Copyright Perfecto Mobile ‘Why Apps Fail’ Report, 2014
Our users ≠ Our Testers…

9. It’s Fundamental all around
For the Business it’s
market coverage
• Great end user experience
• Build requirement for IT
• Optimize user engagement & rating
For the Dev/QA it’s
test coverage
• Optimize process
• Discover bugs early, mitigate risk
• Continuously test on what matters

10. PRINCIPLES

11. What does it take to cover the REAL
end user experience?
v
Device & Platform
Model
OS Version
Screen Size
Form Factors v
Environment
Location
Network
v
Conditions
Signal Strength
Load
Others

12. The
Index
Test Coverage Optimization Process
1. Coverage 2. Lab Sizing
Market Me
My Space
Popular & emerging Analytics
Industry trends & insights
Teams Projects
SDLC
Cycle
Requirements

13. Coverage Target
30% recommended device coverage
driven by market share and device
characteristics
30% and in addition market
leading devices, OS versions
etc.
50% and in addition more legacy &
new device/OS coverage
80%50%30%
10-16
Devices
17-25
Devices
26-32
Devices

14. The Discovery Process
v
Analytics Data Exists?
Yes
- Web Traffic?
- App Stores?
No
v
BU Requirements/ Weights?
APP TYPE
- B2C/B2E/Web/RWD?
- Web, Native, Hybrid?
BYOD vs. Market
Special Prioritizations?
- Demographics
- Platforms
- Geography
- Devices
v
Process & Maturity Related (Lab Size)?
- Agile/Waterfall
- Release cadence
- Automation Coverage
v
Customize Consideration Weights
- Analytics (Me)
- My Vertical
- Market

15. Market – Device/OS Categories
Reference
Devices/OS
Popular
Devices/OS
Legacy
Devices/OS
Emerging
Devices/OS

16. OS Fragmentation
Examples: Samsung S2,
Samsung S3, HTC Incredible
Examples: iPhone 3GS,
iPhone 4, iPhone 4S, iPod’s

17. April OCTOBERJUNE JULY AUGUST SEPTEMBER
Sony Xperia
Z4 Tablet
New iPhones & iPads
Samsung Note 5
Motorola Moto X (New)
iOS 9 Beta
Windows 10
for Phones
DEVICES
PLATFORM
Mobile Market Calendar 2015
Android M ?
MAY
Apple Watch Release
Samsung Galaxy S6
Samsung Galaxy S6 Edge
LG G4
Nokia Lumia 640 XL
ZTE Blade S6 Plus
Huawei P8
Lenovo A6000 Plus
New trending devices,
First Samsung’s to run
Android L OTS
You can already start
Beta testing in June!
Seek for Emulators of
iPhone 6S and more close
to the release date!
Seek for early Beta
through Android Studio
Perfecto Mobile’s Q2
Index Release!!

18. Example: Vertical Analysis
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Finance Retail Insurance Media &
Entertainment
Advertising Airlines
Tablet
Smartphone
Copyright, Perfecto Mobile – MobileCloud Statistics, 2014

19. v
v
My Space
Index
Reports
Perfecto
Cloud Data
App
Store
Target Geo
Demographics Industry Ref.
Company Size

20. U.S Financial Vertical Q1 2015 Snapshot
• 60-75% iOS Devices
• 25-40% Androids Are:
– P1 - Samsung’s
– P2 - Google Nexus
– P3- HTC
– P4- Amazon Kindle
Source: Kantar ComTech
Perfecto Mobile Financial Cloud Data

21. v
Analytics-based
profiling
Product/ marketing
requirements
“ Look inside yourself and you Can see the universe.”
- Brian Herbert

23. THE MOBILE TEST COVERAGE INDEX
MAKE SURE YOUR APP WORKS
ON THE RIGHT DEVICES
Read the Mobile
Test Coverage Index


24. Mobile Test Coverage
Index – Q1 2015
OEM Insights
 Apple rules the U.S market while Samsung the EU5
 More Android OEM’s gain market share
 OnePlus, ZTE, Alcatel, Huawei, Asus
 Windows Phone twice stronger in EU5 than in U.S

25. Q1 2015 – U.S Index
Supports:
• Sufficient Device Mix
• Legacy, Reference, New
• Allows sufficient OS coverage
• Support mix of Screen Sizes
• Support mix of hardware’s

26. Q1 2015 – New & Emerging Preview

27. Summary
Free download The Q1 Mobile Test Coverage Index
Adopt Me, My Space & Market strategy
Note: Mobile Test Coverage is more than Devices & OS’s
Continuously evaluate your mobile quality lab

28. Thank You