"There's an app for everything". App development is moving quickly. Testing needs to keep up: in pursuit of quality. But how to test an App? Not only do we need to comply to customer requirements, we also have to deal with completely new aspects like "store acceptance". Test complexity increases with the growing number of mobile devices that need to be supported. How to cope with that? Apps are of "cloudy" solutions for modern functional problems. This requires dedicated end-to-end testing! Functional and technical, but also non-functional. To enable early technical testing, simulated test environments are made available by the mobile manufacturers, e.g. for portability testing. Before an App goes global, it needs to be accepted by a store, e.g. Apple. Stores apply an extensive, growing list of acceptance criteria, which need to be incorporated in the testing process. Testing has to redefine its position in the App development process.

1. How To Test Mobile Applications
Jeroen Mengerink
Jeroen Mengerink
Jeroen.mengerink@polteq.com
Polteq Test Services B.V., The Netherlands

2. 2
What are Apps?
“App is an abbreviation for application. An app is a piece
of software. It can run on the Internet, on your
computer, or on your phone or other electronic device.”
About.com
“Apps have become a meaningful abbreviation to
something that just works. Apps provide a common and
easily understood idea that has been widely accepted as
a solution – indeed a means to get stuff done quickly
and effectively.”
ipadcto.com

3. 3
Mobile application lifecycle
The mobile App development process:
•First part occurs in all projects
•Second part is store specific
– Incorporates extra requirements

4. 4
The beginning
• All starts with an idea
• Followed by some research
– Does it already exist
– Will people like this
– Which market group(s) to target
– …
• Then a decision to really start creating the App

5. 5
Design
• Screen layouts
– Portrait and landscape
• Screen flows
• Usability guidelines (store requirements)

6. 6
Build
• Functionality
• Specific coding standards
• Performance guidelines (store requirements)
• Mobile devices have:
– Less memory
– Limited processor capabilities
– Smaller storage space

7. 7
Test
• Knowledge needed in different fields
– Software testing
– Telecom
– Mobile OS/platform

8. 8
Test approach
• Reviewing the design
– Customer and store requirements
• Functional testing of the application
– Core functionality as by design
– End-to-end functionality
• User interface
– Colour schemes; layout consistency
• Performance / stress
– Low resources
– Concurrency
• Usability
– Ease of use

9. 9
Test approach
• Compatibility
– Check if native device functionality works
– HW/OS version compatibility
• Certification compliance
– Several certification programs exist
• True Brew Testing
• Java Verified Program
• Symbian Signed Test Criteria
• Submission guideline compliance
– Different stores have different submission guidelines

10. 10
Minimal coverage
• Installation and uninstallation
• Network connectivity
– Following offline profile
– Using WLAN, 2G, 3G,…
– Application performance during network connectivity
problems
• Call / SMS / Alarm handling
• Performance
– Application load time
– During application usage
• Functionality specific test situations

11. 11
Specific test situations
• Incoming calls
• Incoming SMS
• Connection loss
• Orientation changing
• Shaking
• Touch screen
• Notifications
• Advertisements
• In App purchasing
• Energy usage
• Connecting / disconnecting charger
• Screen time out / lock screen

12. 12
Test execution
• Most phone providers provide a simulator
• Simulator used by the providers so:
– Contains real phone functionality
– Hooks for testability
• Also test on the real device!
– Simulator might be outdated
– Simulator might response different from the real thing

13. 13
Customer acceptance
• Communicate changes
– Store guidelines
– Specific device requirements
• Check main flows through the App
– On the actual device(s)
• Initiate end user acceptance

14. 14
App submission
• The binary containing the App
• Name
• Description
• Category
• Subcategory
• Copyright
• Rating
• Keywords
• Screenshots
• Support information
• Large App icon
Usability
guidelines
Docum
entation
guidelines
…

15. 15
App sales and usage
• Last step is marketing
• Get the App sold
• Make sure to keep in touch with the customers
– Reviews of the App
– Patches/fixes

16. 16
Summary
• Awareness of different fields needed:
– Testing; Telecom; Mobile phones
• Test needs to think about more than the customer
– End users
– Store requirements
• Specific device test cases
• Simulators are of great help, but not
the silver bullet

17. 17
Summary
• Test needs to be involved in all stages of the first part
• Test needs to incorporate all extra requirements of the
second part
• Development is way ahead
– Start the pursuit for quality

18. 18
Questions