Expert tester and speaker JeanAnn Harrison reveals critical mobile tests that can be used on many mobile devices, even proprietary. Exercises and ideas for testing that you can use and build on. Sponsored by XBOSoft.
2. XBOSoft info
• Founded in 2006
• Dedicated to software quality
• Software QA consulting
• Software testing services
• Offices in San Francisco, Beijing, Oslo and
Amsterdam
4. Housekeeping
• Everyone except the speaker is muted
• Questions via the gotowebinar control on the right side of your
screen
• Questions can be asked throughout the webinar, we’ll try to fit
them in when appropriate
• General Q & A at the end of the webinar
• You will receive info on recording after the webinar
5. Reminders
Make sure your drained device is available, as well as another
device to send email to the drained device and a charger.
When you see the heart “pumping iron” slides JeanAnn will
walk us through a test on your device.
Please type questions in the Chat box.
We are looking for 1-2 volunteers to take the microphone
during their device testing to let everyone know what is
happening.
6. About the Speaker
JeanAnn Harrison regularly speaks at
industry conferences and is published in
software testing magazines.
She has years of experience with mobile
testing and is a well-known figure in the
QA and software testing community.
Besides software testing , JeanAnn really
likes baseball & ice skating.
8. AGENDA
• Introduction
• Note taking & use of slide information
• Quick overview of Mobile/Embedded
Software
• Types of Mobile Apps
• Exploratory Testing Definition
• Defining Skill Set for the Mobile Tester
• Mindmap getting started with Mobile
Testing
• Good Exploratory Testing Skills/Mobile
ET Skills
Jean Ann Harrison Copyright 2013
• Keys to Mobile Exploratory Testing
• Testing Beyond the GUI
• Learning the Architecture
• Organizing Tests/Stories
• Common tools for ET
• Some Test Case Ideas
• Exercises & Test Case Examples
• Summation & Conclusion
Not covered – User Experience, Usability,
Trainability, Web Application & Security Tests
9. MOBILE/EMBEDDED SOFTWARE &
DEVICES
• Mobile phones, tablets
• Proprietary devices: mobile heart monitors, law enforcement
ticket generators, restaurant personnel taking customer orders
on devices, GPS devices, PDAs
• Embedded Software examples: automobile’s computer system,
air traffic control systems, airplane navigation system, drug
infusion pumps in hospital rooms, elevators, cameras, robots
Jean Ann Harrison Copyright 2013
10. TYPES OF MOBILE
APPS
• Native Applications
• Local to device
• Hybrid Applications
• Local to device but
interacts w/internet
• Web Applications
• Not local to device.
All interactions on
internet
Jean Ann Harrison Copyright 2013
11. EXPLORATORY TESTING -
DEFINITION
• Quoting James Bach: “The plainest definition of exploratory
testing is test design and test execution at the same time. This is
the opposite of scripted testing (predefined test procedures,
whether manual or automated). Exploratory tests, unlike
scripted tests, are not defined in advance and carried out
precisely according to plan.”
http://www.satisfice.com/articles/what_is_et.shtml
Jean Ann Harrison Copyright 2013
12. DEFINING SKILL SET FOR
THE MOBILE/EMBEDDED
TESTER
• Some exposure or knowledge about products from the domain in which you
are testing: aerospace, medical, automobile manufacturing, airplanes,
factory systems, robotics, regulated environments, etc.
• Some knowledge of: hard sciences: math, physics, electronics, engineering,
etc for logical thought process.
• Software sciences: psychology, philosophy, sociology, human factors (human
machine interface) for creative & conceptual thought process
• These testers use skills from many knowledge domains, patterns of errors,
and basic testing skills to create tests. They are using their intuition, critical
thinking, and their mental models.
Chapter 1 – Software Test Attacks to Break Mobile & Embedded Devices
Jean Ann Harrison Copyright 2013
14. WHAT YOU NEED TO BE A
GOOD EXPLORATORY
TESTER
• Awareness
• Discipline
• Focus
Jean Ann Harrison Copyright 2013
15. KEYS TO MOBILE
EXPLORATORY TESTING
• Look beyond the GUI
• Timing
• Domain knowledge
• Knowledge of System
(hardware/firmware/software)
• Characteristic of bugs behavior, formulate
patterns
• Use life experiences for inspiration
Jean Ann Harrison Copyright 2013
16. TESTING BEYOND THE GUI =
SYSTEM INTEGRATION TESTING
Jean Ann Harrison Copyright 2013
SIT for Mobile/Embedded contains elements:
Software + Hardware + Software Variants + Hardware
Variants + Timing + Operating System Drivers +
Network Protocols
17. WHAT DO WE TEST?
Jean Ann Harrison Copyright 2013
18. LEARNING THE
ARCHITECTURE
Work closely with
development to learn more
about how the mobile
application under test works.
Jean Ann Harrison Copyright 2013
19. EXERCISE FOR GROUP
• In the question box, enter 1 test idea of what you
want to explore. The point of this exercise is you do
NOT know what the application is, so you can’t tell me
what kind of functional test to perform.
• I’m testing YOU to see what kinds of non-functional
tests you think might be good to explore.
Jean Ann Harrison Copyright 2013
20. COMMON TOOLS FOR ORGANIZATION
• Sticky notes per conditions, per variables, per test case idea
• Keep a log of what you are doing, record conditions
• Utilize the device’s log or find an open source logging tool
• Use tables to record all variables/conditions per test
• Document situation of the test, list out your test conditions
Jean Ann Harrison Copyright 2013
21. ”WE BECOME MORE EXPLORATORY WHEN WE
CAN'T TELL WHAT TESTS SHOULD BE RUN, IN
ADVANCE OF THE TEST CYCLE, OR WHEN WE
HAVEN'T YET HAD THE OPPORTUNITY TO CREATE
THOSE TESTS.” JAMES BACH
• Create your description or theory. “What happens when…?”
• … battery runs low?
• … battery is charging from a dead battery?
• … when the device gets too hot?
• … when the battery is replaced due to defective battery?
• … device memory is full?
• … I receive a notification while using the phone or another application?
• … when there is a time/date change?
• … device searches are too slow?
Jean Ann Harrison Copyright 2013
23. TEST CASE STEPS
1. Pre-select any application to
download onto device
2. Drain your device’s battery charge
completely
3. Notate time
4. Plug in charging device
5. Immediately power up device
6. Record times for first condition
where app is not installed, device
powered up
Jean Ann Harrison Copyright 2013
7. Drain your device’s battery charge
completely
8. Plug in charging device
9. Immediately power up device
10. Notate time
11. Start to download pre-selected app
12. Record times for each battery level
while downloading
24. EXERCISE: EXPLORATION FOR MORE
TESTS
Jean Ann Harrison Copyright 2013
• What other kinds of tests can you apply based on
the condition of starting to charge from a dead
battery?
• Record your tester stories
25. ORGANIZE YOUR
THOUGHTS/TESTS
Jean Ann Harrison Copyright 2013
Battery
level
Time/Temp Application
Installing
Application
Installed/not in
use/started
App in use (note –
various conditions
apply here)
0%-5% Time each level
increment of
charge
Timing of
download?
Would
notifications
factor?
Notifications?
Memory usage
CPU speed
6%-15% Check the temp
at each
increment
Completed
installation?
Note CPU speed,
memory usage
Searches with
application DB, online
DB?
16%-30% Interaction with data
transfer/network
communication
31%-50%
26. CONTROL SYSTEM CONSIDERATIONS
DEVELOPING TESTS:
HARDWARE TO SOFTWARE SIGNAL
INTERFACE
• Know the input and output connections to the software
• Consider differences in variations between devices like calibrations, physical noise,
electron, light, cold, water, dust device, response time, wear & tear on the hardware.
• Identify input & output devices with ranges and resolutions of values (how is the
software installed onto the device?)
• Define the full range of environmental input disturbances (unexpected inputs)
• Define possible environmental output disturbances (unexpected outputs)
• Determine what is or is not possible in the test lab
• Conduct a risk analysis of what is acceptable and what is not
• Understand device limitations: CPU processing capabilities, memory, time.
• The Aerospace Industry use the term, “Test like you fly, fly like you test” to combine all
facets of how a device’s software should be tested.
Originally written by Jon D Hagar Copyright 2013 Jean Ann Harrison Copyright 2013
27. Jean Ann Harrison Copyright 2013
• Download either Twitter or Facebook onto device
• Start either downloaded app
• From another device, send an email to the device’s email account
• Immediately send a tweet or post a status
• Continue to do engage Twitter or Facebook app for at least 1 minute
• Record email notification and time when sent and received.
• What other observations occurred?
EXERCISE: NOTIFICATION TEST
28. EXERCISE: EXPLORING MORE
FOR THE INTERRUPT TEST
• What other kinds of interrupts could be tested?
• What other tests would you consider if something
goes wrong?
Jean Ann Harrison Copyright 2013
29. TEST CASE EXAMPLE: INTERRUPTS ON MOBILE
PHONES
• Go to your App store and choose an application to download
• While the downloading is occurring, call the mobile phone
• Record observations with the download
• You may need to rely on observing a log file while implementing these tests
• If it fails, what kind of error recovery occurs?
Jean Ann Harrison Copyright 2013
30. LET’S TALK DATA AND STORAGE
• Mobile devices contain limited amount of storage space.
• Testing for mobile apps storing data locally on a mobile device means testing the
boundaries of the app’s design. How will developers handle the limitations?
• Work with your developers to understand those boundaries.
• If your app doesn’t have a boundary set, then the app can fill up your device’s
storage capabilities.
• Testing the limits might mean testing how the software polices that boundary by
having a specific limit before automatically downloading only so much information.
• Per usage, email programs only download so much data and most recent data to
the device.
Jean Ann Harrison Copyright 2013
31. TEST CASE EXAMPLE: TESTING
STORAGE LIMITS
• Download several large applications (ex: Angry Birds & Facebook are large apps)
• Go to your Settings and then to “Manage applications”.
• Review the amount of total storage free.
• Keep downloading apps until you reach below 5KB
• Then start an application where you would save data on the device.
• Check your free storage amount.
• Keep saving data until you reach a point where you receive an error message or a
“can’t save” message.
• Check the remaining free storage in the Settings again to see what the threshold is
for free space is maxed out.
• Delete the last data which was just added.
• Send data though the internet, but use the app under test (using the application
where you first started saving data) and not just email.
• What other tests can you do where storage is maxed out?
Jean Ann Harrison Copyright 2013
32. SUMMARIZATION OF TECHNIQUES
• Combining tests can help shorten project schedules
but what also happens is you can witness results.
Automation results do not give you all you need to
witness as it happens.
• Ask yourself “what if I … “ and consider tests not
related to functional behavior or conditions.
• Using tools, especially log files, can reveal memory
usage, getting statistics while conducting tests.
• Documenting includes use of video,
writing/development of tests in story formats, user
action formats
Jean Ann Harrison Copyright 2013
33. CONCLUSION
• Remember: In mobile and embedded just poking at the GUI will leave bugs
“hiding”
• Different tests by a variety of testers (developer and titled testers) is
commonly needed
• Context matters: there is no one set of tests, approaches, testers, or
attacks which will work all the time
• Limited resources on device can make testing challenging and requires an
entire team effort
Jean Ann Harrison Copyright 2013
34. JEAN ANN HARRISON’S
CONTACT
INFORMATION
• EMAIL: jaharrison@projectrealms.com
• Twitter: https://twitter.com/JA_Harrison
Or @JA_Harrison
• Project Realms Inc website:
www.projectrealms.com
• 8/10/2013, I will be facilitating a Weekend
Testing Session on Risk Based Testing for Mobile
Devices… details can be found through my
Twitter updates.
Jean Ann Harrison Copyright 2013
35. REFEREN
CES
• Software Test Attacks to Break Mobile & Embedded Devices by Jon D Hagar to be
published and released fall 2013.
• 2013 STPCon Presentation slides from “Testing Beyond the GUI” by Jon D Hagar &
Jean Ann Harrison
• http://www.ministryoftesting.com/2012/06/getting-started-with-mobile-testing-a-
mindmap/ Getting started with Mobile Testing Mindmap
• http://karennicolejohnson.com/ Karen Johnson’s website
• http://www.satisfice.com/articles/what_is_et.shtml by James Bach, Satisfice, Inc
• http://www.softwaretestpro.com/Item/5567/ A Different Take on Mobile Testing –
Test Beyond the GUI by Jean Ann Harrison
Jean Ann Harrison Copyright 2013
36. IMAGES
REFERENCES
• Slide 3 - Embedded Systems examples
• http://www.jijesoft.com/en/?option=com_content&view=article&id=18&Itemid=45
• http://www.mseedsystems.com/products/view/268/aldelo-for-restaurants-pos-wireless-edition
• http://blog.laptopmag.com/wpress/wp-content/uploads/2013/02/Pandora-Limit-Free-Listening.jpg
• slide 4 - types of Apps
• http://myeventapps.com/whats-the-difference-native-vs-web-apps/2012/794/
• Slide 5 – Exploratory Testing Definition
• http://www.testandtry.com/wp-content/uploads/explore.jpg
• slide 6 - Defining skill set
• http://cdn.cutestpaw.com/wp-content/uploads/2012/11/l-The-Thinker.jpg
Jean Ann Harrison Copyright 2013
37. IMAGES REFERENCES
• Slide 7 - mindmap used with permission from originators Karen Johnson &
James Bach
• www.ministryoftesting.com
• slide 8 - What you need to be a good Exploratory
• Testerhttp://www.wallcoo.net/paint/kagaya_celestial_exploring/Kagaya_art_Ce
lestial_Exploring_INSPIRATION1.html
• Slide 9 - puzzling together pieces
• http://www.aspire2develop.co.uk/images/People%20development%20page%2
0&%20HOME%20page%204%20piece%20jigsaw%20iStock_000002077956Smal
l.jpg
• slide 10 - Testing beyond the GUI
• http://courses.cs.vt.edu/csonline/OS/Lessons/Introduction/onion-skin-
diagram.gif
Jean Ann Harrison Copyright 2013
38. • Slide 12 – Leaning the architecture
• http://2.bp.blogspot.com/-
gO3DIydtaQA/UBoUj4afwsI/AAAAAAAAAvk/cwPAtXkcgJ4/s1600/12152009_muppets7.jpg
• Slides 13, 18, 21 22
• http://www.duvalschools.org/aprtech/student_life/BPA_Team_2/Exercise.html
• Slide 14 – Common organizational tools
• http://www.dislive.com/wp-content/uploads/2013/07/architect-interior-designer-or-
carpenter-workplace-with-desk-design-my-room-interior-design-tool-936x624.jpg
• Slide 15 – James Bach quote of Exploratory Testing
• https://admissionblog.usc.edu/files/2013/05/Question-Mark.jpg
Jean Ann Harrison Copyright 2013
IMAGES REFERENCES
39. • Slide 24 – Let’s Talk Data Storage
• https://encrypted-
tbn0.gstatic.com/images?q=tbn:ANd9GcR5pxcFFYJILFm1pg20ttHxc-
wPTm_05O4Yr7uhEP--PUZhy-bX
• Slide 27 – Conclusion
• http://www.mobileapptesting.com/wp-content/uploads/2011/03/Mobile-
Apps-300x188.jpg
Jean Ann Harrison Copyright 2013
IMAGES REFERENCES
Clareice: Master’s Degree PMP certification from the Project Management Institute and is a Certified Professional Contracts Manager through the National Contract Management Association (NCMA). She has presented at the National Contract Management Association World Congress.Clyneice:She has participated as an examiner for state quality awards for Georgia and Virginia. She is currently an instructor for the International Institute for Software Testing and has presented technical papers at the Software Engineering Institute: SEPG Conference, American Society for Quality: Quality Manager's conference, Quality Assurance Institute International Testing Conference, International Conference on Software Process Improvement and Software Test and Performance Testing Conferences.
We can change the wording but the concept I think is important – chapter 1
Reference “Integration Attack”.Attacking “internals” where bugs may exist but not be seen in the GUI – give examples of system attacks
BASED ON NOTES BELOW: LET’S ASK THE CLASS IF THEY CAN SHARE NON-GUI BUGS THEY HAVE FOUND AND START A DISCUSSION AS A WAY TO SUMMARIZE. IF NONE-What can if “exercise can we do” to engage them??? Some ideas - Share a bug (can you tell us a bug you found and why or better a bug that escaped you and why” - What are the types of testing do you do now exercise? GUI testing, Requirements checks, exploration, attacks, what? - How many test techniques and/or attacks can you name (a white board organized brain storming exercise)