Build Your Mobile Testing Knowledge

TL
PM Tutorial
4/30/13 1:00PM

Build Your Mobile Testing
Knowledge
Presented by:
Karen Johnson
Software Test Management, Inc.

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

Karen Johnson
Karen N. Johnson is a software test consultant, working on-site and remotely to help her clients build
testing teams and strengthen their testing practice. Karen is considered an expert in software testing and
has been part of the context-driven software testing community for a number of years. She is a frequent
speaker at testing conferences; a contributing author to the book, Beautiful Testing; and co-founder of
WREST, the Workshop on Regulated Software Testing. Karen has published numerous articles and blogs
about her experiences with software testing. Visit Karen’s website at karennjohnson.com.

4/16/2013

Build Your Mobile Knowledge
section 1

Build Your Mobile Testing Knowledge v6

© Karen N. Johnson, 2013

1

Mobile manufacturers
• Manufacturers change
continually.
• Manufacturers vary by
country.
• Understand the market your
app is in vs. the overall
global market.
• Expect to update your
strategy frequently.



2

1

4/16/2013

Device models
• Device models change
frequently as well.
• Devices also vary by
country.
• Device detection: do you
know what your users
have?



3

Operating systems
•
•
•
•

Android from Google, Inc.
BlackBerry OS from RIM
iOS from Apple
Symbian OS from the
Symbian Foundation
• WebOS from HP
• Microsoft Windows Mobile



4

2

4/16/2013

Are you testing with mobile browsers
or testing a mobile app?



5

Mobile apps
iPhone App Store


Android Market


6

3

4/16/2013

Mobile browsers
•
•
•
•
•
•
•
•

Android browser
BlackBerry browser
Safari
Internet Explorer
Firefox
Opera
Dolphin
And more -



7

Network carriers and settings
• Connection is based on
country, carrier, signal and
the device.
• Network carriers can impact
SMS texting and access to
websites.



8

4

4/16/2013

Network Carriers
•
•
•
•
•
•
•
•
•
•
•

China Mobile
Vodafone
AT&T
O2
Orange
Sprint
T-Mobile
Tracfone
U.S. Cellular
Verizon
Virgin Mobile

“ A mobile network operator (MNO),
also known as mobile phone
operator (or simply mobile operator
or mobo ), carrier service provider
(CSP), wireless service provider,
wireless carrier, or cellular company,
is a telephone company that
provides services for mobile phone
subscribers.”
- Source: Wikipedia

Sources: Mobile Forum: http://www.mobileforum.com/
And Wikipedia



9

GSM vs. CDMA
•

•

Global System for Mobile
Communications (GSM, around
80–85 % market) share)
GSM is used everywhere but the
US, India and China

•

•

Code division multiple access
(CDMA) is a channel access
method
CDMA is used in the US, India and
China

Source:
http://en.wikipedia.org/wiki/List_of_mobile_networ
k_operators



10

5

4/16/2013

Connectivity
We are not stationary
when we use mobile.
So why are we testing from
the inside of our quiet
well-connected offices
with fully charged
batteries?



11

Mobile interruptions
•
•
•
•

Forced updates
Incoming calls & texts
Battery issues
Connectivity issues



12

6

4/16/2013

Installation
Application install & updates.



13

UX:
User interface considerations on mobile are radically different than on the desktop.



14

7

4/16/2013

User scenarios
Navigational paths on
mobile are not the same
as on the desktop.



15

OS & firmware updates



16

8

4/16/2013

App Permissions
Your app asks for what?



17

Device settings



18

9

4/16/2013

Functional testing
We still have functional testing.
The UPS example:
– Track packages
– Ship packages
– Locate a store
– Get a quote
– Give feedback



19

Search testing:
classic testing with a new twist: search with barcodes or where device location is
factored into the results.



20

10

4/16/2013

Security: New testing considerations
M-Commerce: the shopping cart with barcodes, location-based (NFC)
coupons, card scanning/swiping and Google wallet.



21

Cookies
•
•
•
•

Remember me?
Forgotten password
Auto-login
Third party login access (i.e.
Use Facebook account to
login)
• Password changed on the
desktop.



22

11

4/16/2013

Location aware
• Testing while moving or
traveling.
• Testing with connectivity
switching, dropped and
intermittent.
• Testing with mobile
interruptions such as
incoming calls.



23

Mobile statistics
• Source: Which source do
you trust?
• Country: Usage statistics
vary greatly by country.
• Analytics: Are you capturing
device usage for your own
application?



24

12

4/16/2013

OS market share
What source do you use to gather information?
• StatCounter
http://gs.statcounter.com/#mobile_os-ww-monthly-201012-201112
• ComScore
http://www.comscore.com/Press_Events/Press_Releases/2011/12/comScore
_Reports_November_2011_U.S._Mobile_Subscriber_Market_Share
• MobiThinking
http://mobithinking.com/mobile-marketing-tools/latest-mobile-stats



25

Device Matrix
source: Mobile Design and Development, by Brian Fling
Class

Markup

CSS

Javascript

Class A
Class B
Class C
Class D
Class F



26

13

4/16/2013

What devices do I need to test?
• Brad Frost - blog
post:http://bradfrostweb.com/blog/mobile/test-on-realmobile-devices-without-breaking-the-bank/
• My comment on his post is that his view is somewhat North
American focused.
• While the iPhone and Android dominate the market, those
two options are not the only players in the market.



27

Device Selection
How will you decide which
devices to test on?


• User statistics
• Country statistics – based
on user population
• Budget
• What the team owns


28

14

4/16/2013

Mobile User Interface, Design &
User Experience
section 2



29

MENUS, FORMS & NAVIGATION



30

15

4/16/2013

Menu Styles
•
•
•
•
•
•
•

Springboard or Grid Menu
List menu
Tab menu
Gallery
Dashboard
Metaphor
Page Carousel

Source: Theresa Neil.
http://www.mobiledesignpatterngallery.com/mobile-patterns.php?colid=65438029-72157627607680275
http://www.slideshare.net/theresaneil/navigation-patterns-for-mobile-apps



31

Menu Styles
Springboard


List


32

16

4/16/2013

Menu Styles
Tab


Gallery


33

Menu Styles
Carousel


Metaphor


34

17

4/16/2013

Navigating
Giving the user a unique way to
move forward & back.



35

Navigating
• Paging
• Forward & backward
• Cookie crumb trail
(websites)



36

18

4/16/2013

Navigating

• Rotate
• Pinch/zoom
• Accelerometer
• Vertical/horizontal
• Paging forward & backward



37

Input Selector/Navigation
•
•
•
•
•
•
•
•
•
•

Rotate
Touchscreen
Click wheel
Rollerball
Pinch & zoom
Typing/keyboard inputs
Accelerometer
Paging backwards/forwards
Cookie crumb trail
Gestures



38

19

4/16/2013

UI controls
• Entry fields
• Lookup buttons
• Radio buttons
• Drop downs



39

Custom Controls
• The slider bar on AA’s
mobile app is an unusual
custom UI control.
• Does your app have any
custom controls?



40

20

4/16/2013

Links
• Do links work?
• Does your site have links to
other sites?
• What about the sitemap
links?



41

Lists
• Alignment
• Consistency
• Disruptions from ads or 3rd
party feeds



42

21

4/16/2013

Color-alternating lists
•

Grouped
rows

•

Zebra
stripping lists

•

Text
Alignment

•

User
Accessibility



43

Colors & Symbols
•
•
•
•

Symbols
Spacing
Rotating
Columns



44

22

4/16/2013

Barcodes and Scanning
• Visibility
• Image tolerance
• Connecting to a
service



45

Currency
User interface considerations
with multiple currencies: is the
correct symbol for the
currency shown?



46

23

4/16/2013

Images
• Where’s the image?
• You might use your desktop
to see if the issue occurs
outside of the mobile
environment.



47

Images
•
•
•
•
•
•
•
•
•

Size of files
Cache
Refresh rate
Full screen
Wi-Fi drops connection
Rotate
Pinch/zoom
Accelerometer
Vertical or horizontal



48

24

4/16/2013

Video & Streaming
• Flash?
• Videos
• Connection speed
• Availability



49


50

Completing Forms
At some point, a user
needs to enter
information. Minimizing
input – such as offering
smart fill fields on location
awareness helps to cut
down on the amount of
information a user has to
enter.


25

4/16/2013

Completing Forms
• Sign-in and search are
two of the most
common forms on
mobile devices.



51

Login



52

26

4/16/2013

Passwords
• Case sensitive
• Changing the case
• Seeing the text entry
• Staying logged in (cookies)



53

SEARCH, SORT, FILTER



54

27

4/16/2013

Search
• Wildcards
• Text entry
• Case sensitivity
• Alpha and numeric entry
• Dropdowns
• Location based searching


55

Search & Filter
• Location-based
filtering
• UX considerations
on filters, applying
& clearing
• UX considerations
– scrolling through
many search
results


56

28

4/16/2013

Search & Sort
When results are pages
long on a mobile device,
offer the user a means to
sort or refine the result
set.



57

Search
• Are there fields that are prefilled based on date, location or
time?
• OpenTable is an example of time and location.
• Are there fields that are not offered on mobile but are offered
on the desktop version?
• Is there a search button? Is there another method to start or
run the search?
• Is there a way to refine the results?



58

29

4/16/2013

LOCATION AWARENESS



59

Find location
Consider appdependencies on phone
settings - like locationbased searching.



60

30

4/16/2013

Search: filtering results



61

M-Commerce: restaurants
•

Find a restaurant (localbased search or
otherwise), book a
reservation, order for
pickup service, and
booking via mobile.

•

From a testing
perspective, what
information does the
app have? Credit cards
and personal
information attached
to an account on a
mobile device open a
new security concern.



62

31

4/16/2013

RESPONSIVE DESIGN



63

What is “responsive?”
Menus or other pages whose
presentation or behavior is
altered on different devices
and screen widths. Flexible
grids, flexible images and the
ability to “ebb and flow” a
design based on screen size.
Ethan Marcotte in 2010 coined
the term in a landmark article
called: A List Apart. See:
http://alistapart.com/article/responsiveweb-design



64

32

4/16/2013

A short lesson on
responsive design
http://blog.creativemagma.com/featur
ed/2209/



65

Media queries
• Definition: “A media query consists of a media type and zero
or more expressions that check for the conditions of particular
media features. Among the media features that can be used
in media queries are ‘width’, ‘height’, and ‘color’. By using
media queries, presentations can be tailored to a specific
range of output devices without changing the content itself. “
- Source: W3C, http://www.w3.org/TR/css3-mediaqueries/#media0



66

33

4/16/2013

PHONE VS. TABLET



67


68

Phone vs. tablet
People often use both at the
same time.
Testing by planning user
scenarios is a good way to
uncover and identify
situations to test.


34

4/16/2013

Phone vs. tablet
•
•
•
•

App versions
Apps adapted for tablets
What about phablets?
Functionality and form factor – does the tablet app work the
same as the mobile phone app?



69

App Version
• A version page is an
opportunity to once
again show the app icon
and/or company logo.
• What about other
maintenance pages?



70

35

4/16/2013

App Version



71

Tablet Version View

• Does your app have a
version number?
• Does your app have
unique tablet or device
settings?



72

36

4/16/2013

Sitemaps
“They visually represent the relationship of
content to other content and provide a map for
how the user will travel through the
informational space.” Brian Fling, author Mobile Design and Development



73

Uniquely mobile

TOUCH, PINCH, ZOOM,
ONE THUMB, GESTURES


74

37

4/16/2013

Touch
Touch Gesture Icons
http://www.mobiletuxedo.com/t
ouch-gesture-icons/
Input Guidelines
http://uxdesign.smashingmagazin
e.com/2011/10/06/not-yourparents-mobile-phone-ux-designguidelines-smartphones/
Touch Gesture Reference Cards
http://www.lukew.com/ff/entry.a
sp?1370



75


76

One thumb design
Fat Thumb: A One-Handed
Alternative To Pinch-ToZoom
http://www.fastcodesign.c
om/1671741/fat-thumb-aone-handed-alternative-topinch-to-zoom#1
Testing One Thumb, One
Eyeball Mobile Use
http://www.lukew.com/ff/
entry.asp?1664


38

4/16/2013

Mobile App & Device Settings
section 3



77

app, device & browser settings
•
•
•
•
•
•
•
•
•
•
•
•
•

Overall app permissions
Specific app permissions
App permission-dependencies
Device settings
Browser settings
Cookies
Wireless & network settings
Data Roaming
SD card settings
Swype
Display settings
Battery Settings/Status
Task Killers



78

39

4/16/2013

Overall App Permissions
Know the app permissions
your mobile app requires.
Consider what the user may or
may not have enabled on the
device for all applications.



79

Specific App permissions
What permissions does
your application require?



80

40

4/16/2013

App permission-dependencies
If your app interfaces with 3rd
party data such as location
finding, what happens when
that access is off, down or
unavailable?



81

Device settings
• Consider the environment of
the device: what has been
enabled or not.
• Consider the ability or
restrictions to downloading
and installing applications.
• Consider upgrade scenarios
and whether permissions have
been changed or reset.
• Install testing has returned.


82

41

4/16/2013

Browser settings



83

Language settings
• Finding what languages are
available on the device
• Choosing a different
language
• Accessing and using a site or
an app that’s multilingual
• Using different keyboards
on devices



84

42

4/16/2013

Cookies
• Does your site use cookies?
• Check the login process
with cookies on/off.



85

Wireless & Network Settings
Have you ever tested your
website when connectivity is
down or disrupted? This
scenario is more likely with
mobile devices.



86

43

4/16/2013

Network Mode
CDMA is used in the US (also in China
and India) but everywhere else uses
GSM or other modes of connection.
It can be challenging to test like a
user when you live in a different
country and some options are not
available on devices where you are
located.



87

Network search
Users can switch from
and to an application or
browser session while
other activities are
taking place (including
incoming phone calls)
does your app handle
that disruption?



88

44

4/16/2013

Connecting to a network
A user could switch to
your app or website when
they’re not connected.



89

Connection errors
If your app uses the SD
card for storage – what if
the card is already full or
another app writes to the
card?



90

45

4/16/2013

Data roaming
Would you enable your
testing team to leave the
office and experience
mobile as they travel
around the office?



91

SD card settings
• Does your app store
data on a card?
• What if the card is
removed?
• What if a different card
is inserted?



92

46

4/16/2013

SIM cards
• A SIM card or Subscriber
Identity Module is a chip.
• SIM holds personal identity
information, cell number,
phone book, text messages
and other data.
• Again, what happens if the
card is full? Changed?
Removed?


93

Swype settings
• Does your app work
with swyping or voice
input?



94

47

4/16/2013

Display settings
Consider whether a user
has made device settings
that can impact how your
application appears.



95

Battery conditions
• Have you tested your
app under low battery
conditions?
• Is your application
responsible for battery
drainage?
-

See Julian Harty’s work
See Android Forum



96

48

4/16/2013

Battery conditions: power save
Is this a variable you need to
consider in testing?



97

Task Killers
•
•
•

What background tasks are
running?
What data is being synced?
Does a Task Killer help or not?

Android
http://www.pcworld.com/article/24
4067/taskkiller_apps_will_they_help
_or_hurt_your_battery_life.html
IOS
http://www.youtube.com/watch?v=
nEJA1yUgR80



98

49

4/16/2013

The Competitive Mobile
Marketplace
section 4



99

What do mobile apps compete on?
According to uTest’s white paper from the product
Applause, there are intuitive metrics such as:
• content
• elegance
• pricing
• usability
• and satisfaction



100

50

4/16/2013

How steep is the competition?
•
•
•
•

Apple Store: 700,00+ apps
Google Play: 700,000+ apps
Blackberry App World: 100,000+ apps
By 2015, the International Data Corporation (IDC) predicts
182.7 billion mobile apps will be downloaded
• In 2010, 10.7 billion apps were downloaded
- source: uTest Applause white paper



101

MARKET RANKING



102

51

4/16/2013

Market Rankings
Applyzer is a product that watches
market rankings. Notice how they
segregate data by country and mobile
app category.



103


104

Market Rankings
Distimo is another market watching
application.


52

4/16/2013

Market Rankings



105


106

Market Rankings
RankedNow is another product
tracking the competitive market.


53

4/16/2013

MOBILE COMPETITORS



107

Competitive Analysis – Vertical Competition
mobile app to app
AccuWeather


The Weather Channel


108

54

4/16/2013

Competitive Analysis – Vertical Competition
tablet app to app



109

Competitive Analysis – Mobile Competition
the app Shazam vs. other ways to spend time

Notice how competitors might not
be offering the same product but
compete equally well for a user’s
time, purchase and use.



110

55

4/16/2013

Mobile competition:
Shazam’s “competitors”

See how this “competitor” isn’t offering
the same product but is vying for the
user’s time.

This competitor is a more typical
“competitor.”


111

Mobile competition:
a sample of differences by store

Competitors vary from store to store.



112

56

4/16/2013

MARKET COMMENTS



113

Charmin – Sit or Squat

This Charmin app was liked until the app was tied to Facebook and requires a FB
Account in order to login or comment. Users have scathing comments about the change.



114

57

4/16/2013

Tide Stain Brain

The Tide app made a change when the company removed user comments on how to
Remove stains and in turn infuriated the user community.



115

Chlorex myStain

The app gains respect and ratings by now being solely focused on pushing their product.



116

58

4/16/2013

Out of Milk

Note the number of users for the free version. And also note the overall ratings by users.



117

Spotify

Note how users comment on updates that cause issues or comment on missing
functionality.


118

59

4/16/2013

Competitive Analysis
Exercise:
1.

2.
3.
4.

5.

Select two mobile applications that
would seem to be competitors such as
AccuWeather and the Weather
Channel.
Review each app for the “intuitive
metrics” listed as factors.
Make notations on each of the factors
for both apps.
For satisfaction, you can use either your
satisfaction with the app, app store
comments or a combination of both.
If you were to apply a scoring to each
app, which app would you choose over
the other app?



119



120

60

Build Your Mobile Testing Knowledge

Recomendados

Recomendados

Más contenido relacionado

Destacado

Destacado (18)

Similar a Build Your Mobile Testing Knowledge

Similar a Build Your Mobile Testing Knowledge (20)

Más de TechWell

Más de TechWell (20)

Último

Último (20)

Build Your Mobile Testing Knowledge