Mobile applications testing

1. Mobile Apps
Testing
Android
IOS
Blackberry
Windows

2. Introduction
A mobile application is a software
application designed to run on
smartphones, tablet computers and
A mobile application, also known as
other mobile devices. They are
mobile app, is a term used to
available through application
describe software that runs on
distribution platforms, which are
smartphones and mobile phones.
typically operated by the owner of
Mobile applications are designed to
the mobile operating system, such as
educate, entertain, or assist
the Apple App Store, Google
consumers in their daily lives.
Play, Windows Phone Store and
BlackBerry App World. Some apps
are free, while others have a price.

3. Cont …
A mobile application will normally be structured as a
multi-layered application consisting of user
experience, business, and data layers. When
developing a mobile application, you may choose to
develop a thin Web-based client or a rich client. If
you are building a rich client, the business and data
services layers are likely to be located on the device
itself. If you are building a thin client, the business
and data layers will be located on the server..

4. Figure 1 illustrates common rich client mobile application architecture with
components grouped by areas of concern

5. Types of Mobile Application
Web application
• Web apps live on a remote server and are
delivered to users over the Internet through a
browser. Any website component that performs a
function for the user can be considered a Web
app, so even Google's search engine counts. Web
apps can be easier for IT to manage, because they
all run on the same platform.

6. Image: Web Application

7. Native Application
Native apps are applications developed for a
specific platform or device. Because native
apps interact directly with the operating
system, they often perform better than Web
apps. Native apps can also take advantage
of the specific features of an OS or
device, which sets them apart from Web
apps.

8. Image: Native Application

9. Hybrid Application
Most apps are hybrid apps, which combine
native and Web app features. Hybrid apps
work whether or not devices are connected
to the Internet, work with a device's file
system and integrate with Web services. For
example, online banking apps store some
data on devices, but users access them over
the Web.

10. Legacy Application
Legacy apps were built on older technologies but
are still important to business processes. The
challenge for IT comes in keeping that old
technology working while converting it for
new, more efficient platforms.
Fortunately, legacy apps often use languages and
OSes with open or standard programming
interfaces, which can make them easier to
update.

11. Challenges in Mobile Application Testing
Unlike the PC-based environment, the mobile environment
comprises a number of devices with diverse hardware (physical
properties), software configurations (platform, OS etc.) and
communication intricacies. You can’t certify that the
functionality on one device, will work on all other devices with
same properties.
This diversity in mobile computing environments presents
unique challenges in application development, deployment
and quality assurance, thus requiring unique testing strategies
for different platforms.

12. Cont..
Few challenges are:
Operations with memory control of memory intensive leaks
The intensity of power consumption, sensitivity to the charger
The use of disk space, stability in the limited space on the drive, logging, work with memory
cards
Support for various screen resolutions, portrait and landscape modes
Different standards compliance for different platform (Symbian Signed certification, Java
Verified testing, etc.)

13. Various types of testing

14. Functional testing
Functional Testing ensures that the
mobile application is behaving as per
the requirements. Majority of the
testing is driven by the user interface
and call flows. However the
specialized devices with emerging
technologies like smart phones having
good processing power, support
multi-tasking applications with
functional and background
processing. This has multi-fold
increased the functional testing. The
complexity increases when this basic
functional testing is combined with
other types of testing, like port,
laboratory, performance, and memory
leakage testing as mentioned below.
Due to unavailability of actual
connectivity and required
infrastructure, the mock/prototype
and simulated testing further add the
complexity.

15. Port Testing
1. Device Testing:
•The testing is done on different devices
with same platform.
•There are chances that the base
This type of testing is done to test platform/OS version can change. But it is
the same functionality on mostly avoided. However, in case the OS
version gets change for same
different devices with different platform, then major testing is
platforms. It can be classified into recommended in which the complete
2 categories: testing is done again for the variable OS
version.
•Based on the device properties, specific
testing scenarios are identified in which
testing is done. This is termed as minor
testing.

16. Cont…
2. Platform Testing:
• Platform testing is done on
devices of different platform (OS).
For example, functionality testing
on device with J2ME to device
with BREW platforms. Here
because of variable OS
architecture, the same
application is implemented
differently, so complete thorough
testing has to be done.

17. Laboratory Testing
This testing is done specifically at carrier side
where the complete wireless network is
simulated as similar as the real environment.
The actual network problems, which might
arise in the real environment, can be identified
and resolved in this simulated laboratory. This
is very important when application is using
voice and/or data connection to perform some
functions. Although simulators and other test
tools do support simulation of such type of
networks. But essentially they are not reliable
enough to provide the complete assurance
about the real-time network issues. Hence it is
not advisable to launch a mobile application
based on testing conducted only on simulators.

18. Performance Testing
Performance testing means the behavior of the application under certain conditions like low resources
(Memory/Space), multi-user simultaneous access to application servers; time taken in data transfer is high.
The applications performance may be affected from two sides; Server side and Client side.
The major performance issues on the server side are:
Variations in Response times
Streaming resource intensive multimedia packets
Delays/Drop in delivery of packets
Applications crash
In-efficient use of resources

19. Cont…
The major performance
To avoid these issues we
issues on the client side
should do the following
are:
• Use extensible load testing tools
• Applications behave differently on which support custom protocol
various platforms and handsets
• Include performance testing early
• High Memory and CPU in the cycle
consumption
• Using stubs to simulate external
• Mobile application too slow to components
load
• Record work flows using mobile
• Application causing battery drain clients or simulators
out

20. Memory Leakage Testing
When a program does not correctly manage its memory allocations i.e. loses track of a piece of memory that was
allocated and hence will never be released to OS, it is said to leak memory. As result it diminish the performance of an
application or even fails it.
Mobile devices have huge constraint on memory in terms of installation and runtime memory. Although memory on the
devices has grown leaps and bounds but the constraint remains the same. This is due to the fact that along with growth
in device memory, its consumption is also increases with background applications, sync process, multi-threading etc.
With scarcity of memory the device behaves very weird. In some cases, the device runs very slow and in
other, application crashes. This behavior is very random and is very hard to diagnose.
A tester has real hard time to do memory leakage testing. With improvements, the memory instance can be fetched at
any time and with verifying the trend, at least an alarm/warning can be raise to verify the memory usage.
Memory usage mostly increases due to application allocate memory for creating objects , but not releasing the object or
locking the shared memory for specific usage, and hence not releasing it to the pool.
New programming languages although have their own way of memory management like garbage collector in java, but
programmers need to be very careful in dealing with explicit memory management as in C or C++.

21. Interrupt/TAPI Testing
While running a mobile application, there are
interruptions such as incoming calls, text messages, low
memory or low battery notifications. The mobile
application should be able to handle these interruptions
gracefully without causing a disruption.

22. Cont…
Different kinds of interrupts are:
Incoming and Outgoing SMS and
MMS
Incoming and Outgoing calls
Battery Removal
Cable Insertion and Removal
Network outage and recovery
Media Player on/off
Device Power cycle

23. Cont…
Testing the application for interrupt
handling is a very important aspect of
mobile testing as these interrupts are a
very common phenomenon that may occur
anytime during the application run. The
interrupts must be handled properly so that
they do not cause interruption in the
application functionality. The application
must be in a suspended state during the
interruption but must resume continuing its
normal function after the interruption is
over. The hardware interrupts are on the
highest priority and the software interrupts
can be prioritized accordingly

24. Usability testing
Essentially the application usability
plays a crucial role for success in
market. The usability testing is done
to evaluate the application for
achieving its specified goals, efficiency
and acceptance from the users.
Usability testing is usually conducted
by the test users in the test
environment. Users are given the
tasks to accomplish and their
feedback about the problems faced,
ease of use is recorded. This is the
most accurate system to evaluate the
usability of the product as it gives
direct input on how real users use the
system. The major goals of the
usability testing are Performance,
Accuracy, Recall and Emotional
Response. The different methods
used for usability testing are Hallway
testing, Remote usability testing,
Expert Review and Automated Expert
Review

25. Installation Testing
Installation testing is one of the
most important parts of testing
activities. The mobile applications
are either pre-installed on the
device or can be downloaded from
any store. The installation testing is
not necessary if the application is
pre-installed. But in case user is
downloading and installing an
application on the device, this
testing must be done to ensure a
smooth installation without any
difficulties. This testing focuses on
each step of the installation
process that a user has to follow.
Usually the methods followed for
downloading an application are
‘Over the Air’, ‘through cable’, SD
Card, and Bluetooth. This testing
covers installation, upgrade, and
uninstallation of the application.

26. Security Testing
Mobile device security will become more and more important as the user
base grows, so it is essential to test the security of your mobile web
applications, sensitive data storage, and how your application behaves under
various device permission schemes.
A. Most mobile devices assume one user; they do not have multiple accounts.
(Except in the case of multiple email addresses configured per device, which
should be tested especially if your application deals with the device’s Contacts
or Email functions). But in general there is no concept of user
switching, different profiles, or permissions based on user level.
B. It is up to the user whether or not they configure a password/unlock
pattern for their device at all.
C. Outside communication of any sensitive data should be done over SSL/TLS
because it can’t be assumed that the user will configure their Wi-Fi to be
secure.

27. Image:

28. Stress Testing
Mobile device applications have much less overall device memory and power available
so must handle them very efficiently.
Stress testing is a must to find exceptions, hangs, and deadlocks that may go unnoticed
during functional and user interface testing.
1. Load your application with as much data as possible to try to reach its breaking point
2. Perform the same operations over and over again
3. Perform the repeated operations at varying speeds – very quickly or very slowly
4 Leave your application running for a long period of time, both interacting with the
device and just letting it sit idle, or performing some automatic task that takes a long
time.
(E.g. – Slideshow)
5. Randomly send screen taps and keystrokes to your application.
6. Have multiple applications running on your device so you can switch between your
application and other device applications often.

29. Localization
If you’re focusing on multiple
regions/geography, you need to localize
your mobile app. What if one of your users
is based in China and he doesn’t know
anything about English? Having an option
to use application in local language will
help you gain more popularity amongst
local consumers. Hence, it’s better to
translate application user interface and
adapt graphics for a specific culture/locale.
The process can also include translating any
help content associated with the
application. Again, having multilingual app
will add few extra hours in testing your app
but trust me it’s worth! It’s good to have
larger audience than nothing!

30. Google: United Arab Emirates

31. Google: India

32. Certification Testing
In order to get the
compliance certificate
for the application, it
has to be tested against
the various testing
guidelines provided for
various platforms. Each
platform has its own
criteria for certifying the
application. Following
platform specific points
should be taken care
during certification:

33. Cont….
• Need to have valid Certificate for Signing the
application
• Application’s .apk file should be in compliance with
http://www.android.com/us/developer-content-
policy.html
Android: • Proper versioning should be followed in case of
updated build publication
• Application can be published on the Android Market
web site with valid market account credentials
• Android market will scan and test your application and
can report errors if any
• No specific Test suites available as of now

34. Cont…
iPhone:
•Application should have Unique name
•Application should have specific category
•Need URL where you can have your application’s feedback
•Need to follow apple HIG (Human Interface Guidelines) from Apple
(http://developer.apple.com/library/ios/documentation/userexperience/conceptual/mobilehig/MobileHIG.pdf”)
•Strictly main picture of your application should be in 320×480 or 320×460 size
•App store will scan and test your application for various compliance
•Application may get rejected under following cases like
•Indecent Material
•Unexpected delays while running application
•Copying existing functionality

35. Cont…
Symbian
• Need to have .PKG file for building .SIS file
• Application Read Me text file
• Test Results for “Universal Tests” and “Capability” Test criteria
• There are mainly 3 types of Signing: Open Signing, Certificate
signing and express signing.
Windows Mobile
• Should comply the various Marketplace requirements
mentioned under http://msdn.microsoft.com/en-
us/windowsmobile/dd569132

36. Various Platforms
As the desktop computer
runs on different platforms
or operating systems like
Windows, Linux, and Mac
OS; similarly the mobile
device runs on different OS
or platforms. The brief
details of the various
mobile platforms or OS are
as follows:

37. Image:

38. Cont…
iOS: iOS from Apple Inc. is a closed source
Android: Android from Google Inc. is an open
proprietary platform. The applications made on
source Linux derived platform. Till date we have 7
this platform are popular for the rich user
releases of Android, the latest being 2.3. This
interface it provides. iPhone and iPad are the most
platform is growing by leaps and bounds and in a
popular running products using this platform.
year from 2009 to 2010 the market share rose
Third party applications were not supported until
850% from 1.8% to 17.2%.
the 2.0 release in July 2008.

39. Cont…
Blackberry OS: Blackberry OS from RIM (Research in
Motion) is a closed source proprietary platform. This
platform majorly focused on ease in use and was
majorly targeted for business purposes and did not
provide many multimedia supporting features. RIM’s
future strategy focuses more on the newly acquired
QNX where they have already launched a blackberry
playbook with running version of QNX and expect the
first lot of smartphones to be launched in early 2012.

40. Cont…
BREW / BREWMP: BREW is a platform run by Qualcomm which is used by some mobile
manufacturers and service providers. However people using the phone do not come to know
that the platform is BREW as there is no branding involved from BREW side. This platform just
runs in the background with custom “skins” of mobile manufacturers or operators on the top.
Nowadays in many of the phones where BREW was used, it has been replaced by BREWMP.
Bada: Bada is a closed source proprietary platform lately introduced by Samsung electronics to
replace the feature phones to smartphones. The first device to run on bada is “Wave”. With this
device Samsung has also launched an app store containing more than 3000 applications.
Windows7: Windows 7 is again a closed source proprietary platform introduced by Microsoft
with a completely new UI inspired by Microsoft’s “Metro design language”.

41. Carriers
Examples: Good examples of
A mobile carrier is one that
mobile carriers are Verizon
provides connectivity services to
Wireless, AT&T, Vodafone, Airtel ,
mobile phone subscribers.
Aircel and so on.

43. Emulator and Real Device
Mobile emulators are powerful
tools for developing mobile
applications and are widely used
for both manual and automated
mobile application testing. On the
other hand, since mobile
applications are used on real
handsets and not
emulators, testing on real devices
during the QA process is required
to ensure the highest level of
application quality.

44. Mobile Emulators: Pros
Price - Mobile emulators are free
and provided as part of the SDK
with each new OS release.
Simple - Just download the
software, install on your PC and
you're ready to go. Multiple
emulators can be run in a simple
and straightforward manner.
Fast – Since emulators are simple
client software that runs locally on
your PC, they have less latency
than real devices connected to the
local network or in the cloud.

45. Mobile Emulators: Cons
Increased Risk - With emulators, by definition, you are not testing on the same platform
and network used by your users. This means that even if all goes well, you cannot be 100%
sure it will actually work on a real device ("false-positives").
Hardware and Software Differences – Emulators are typically a "plain vanilla" version of
the OS and often do not reflect the specific hardware and software features of each
supported device. In addition, as time passes following an OS version release, the
emulators are not updated to reflect new devices available in the market.
Different network environment - In terms of network configuration, mobile emulators run
on the PC (through any personal firewall), connect to the LAN and access the Internet via
your corporate firewall. Using real handsets, the network is connected to the radio
interface and from there to the Internet. These differences could affect application
behavior.

46. Cont….
No way to test network
interoperability – It is important to
Differences in Computing Resources
test the impact of network-related
– Depending on the processing
events (e.g., incoming call, text
power of the PC running the
message, etc.) and different network
emulator and the type of handset or
technologies
Smartphone being used for
(e.g., HSPDA, WCDMA, UMTS, LTE)
testing, performance on the
on mobile app behavior. Since
emulator may be unrealistically good
emulators are not connected to the
or bad.
mobile network, they do not support
interoperability testing.

47. Real Devices: Pros
Reliability - Testing on real handsets
always gives you accurate results (no
false positives or false negatives).
Interoperability testing - Real device
testing is typically performed in a live
network.
True user experience - Testing on real
devices is the only way to truly
understand the user experience, taking
into account the CPU, memory, screen
size, etc. for a given device.
Performance testing - Easier to expose
performance defects with real
handsets, as well as defects which are
the result of the handset itself or its
environment.

48. Real Devices: Cons
Logistics and costs - Best practices call for testing on up to 30-40 devices in
your market and replacing ~30% of those each quarter to stay up to date.
The efforts involved in procuring and managing these devices are significant.
Cumbersome for development - In the initial development stages, real
handsets are harder to connect to IDE than emulators, which can slow down
the debugging process.
Security issues - If you are using devices connected locally to your
workstation, you will need to make sure USB ports are open. In
addition, mobile devices can easily be stolen, allowing unauthorized access
to your internal network resources.

49. Cont….
Emulator:
http://www.mobilexweb.com/emulators