My slides from the workshop on Mobile Testing I gave at the London Tester Gathering Workshops 2014.

1. Mobile Software Testing
Stephen Janaway
© Stephen Janaway 2014

2. Today
• Fun
• Interactive
• Joint learning
• Interrupt when you want.
• Write questions on the question wall.
• Share your experiences.
© Stephen Janaway 2014

3. This Afternoon
• What Is Mobile?
• Testing On Mobile
• Exercise - Using User Agents
• Simulators and Emulators
• Exercise - Using an emulator to test
• Break
• Mobile Specific Use Cases
• Choosing Devices to Test
• Exercise - Mobile Testing Challenge
© Stephen Janaway 2014

4. Please leave your phones
switched ON!!!
© Stephen Janaway 2014

5. Who am
I?Stephen Janaway
• 14 years experience in mobile software
testing.
• Mobile devices
• Mobile networks
• Mobile applications
• Have worked for companies such as
Ericsson, Motorola and Nokia.
• Currently at Net-a-Porter Group.
• www.stephenjanaway.co.uk
• @stephenjanaway
© Stephen Janaway 2014

6. www.testinginthepub.com
@testinginthepub
Search for “Testing In The Pub’” on iTunes
© Stephen Janaway 2014

7. Mobile devices are
© Stephen Janaway 2014

8. Mobile devices are much more
• Alarm Clocks
• Maps
• Link to loved ones
• Hand held shops
• Tickets
• Music Players
• Books
• Games Players
• Cameras
• ……
These are key experiences.
© Stephen Janaway 2014

9. Subscriptions vs. People
© Stephen Janaway 2014

10. Different OS’s
© Stephen Janaway 2014

11. Different Mobile Browsers
© Stephen Janaway 2014

12. Native or Web?
• Native app:
– Specifically designed to run on a device.
– Runs on device OS.
– Testing needs to include download/ upgrade/ uninstall.
– Typically needs to be adapted for different devices.
– May include Web Views (Hybrid apps).
• Mobile website:
– All or some parts of the software are downloaded from the Web each
time it is run.
– Usually accessible from all Web-capable mobile devices.
– HTML5 makes it increasingly difficult to separate web apps from native
apps.
– Testing needs to include different mobile browsers.
© Stephen Janaway 2014

13. What Makes Mobile Testing Different?
• Many platforms.
• Many OS versions in the field.
• Many different hardware configurations.
• Devices with limited memory and processing
capability.
• Limited battery life.
• Network connectivity.
• Application stores.
• Low maturity of automation solutions.
© Stephen Janaway 2014

14. Usability Matters
• Usability is more important in the mobile world:
– Smaller screen.
– More difficult to interact with and typically used one
handed.

15. Fragmentation
© Stephen Janaway 2014

16. Fragmentation
• Lots of different hardware and software
configurations to support.
• Options:
– Don’t use them at all – use simulators or user
agents.
– Buy all or some of them.
– Rent all or some of them.
– Use a cloud based service (Perfecto Mobile,
DeviceAnywhere).
– Crowd source.
© Stephen Janaway 2014

17. Using Simulators
and Browser Emulation
• Simulators:
– Normally found within OS SDKs.
– Good to start testing with.
– Do not recreate exactly how a device functions.
• Browser Emulation:
– User Agent Switching:
– Within a browser (Chrome developer tool is best).
– Changes how the browser renders the page.
© Stephen Janaway 2014

18. Pro’s and Con’s - User Agents
• +
– Easily available
– Cheap or free
• -
– They don’t fully emulate the device (some are better
than others).
– They are not running on low performance hardware.
– Cannot use on real network connections.
– Security is normally permissive.
© Stephen Janaway 2014

19. Exercise - Using User Agents
● Start Chrome and browse to a site (www.net-a-porter.
com for example).
● Goto dev tools (from View/ Developer/ Developer tools).
● Change user agent.
● Investigate throttling - element load order, etc.
● Investigate using touch via keyboard.
● Look at logs.
● Study UX.
● Rotate the display.
● Change user agents - how does the site display on a
different sized screen?
20 mins study and 10 mins de-brief.

20. Exercise - Using User Agents
What was good about
the dev tools?
What could have been
better?

21. Mobile Introduces New Scenarios
• Download and install.
• Un-install.
• Upgrade.
• Rotation.
• Interaction with Sensors.
• Location.
• External media.
© Stephen Janaway 2014

22. Mobile Introduces
a Whole New Language
© Stephen Janaway 2014

23. Pro’s and Con’s - Emulators
• +
– Easily available
– Cheap or free
• -
– They don’t fully emulate the device (some are better
than others).
– They are not running on low performance hardware.
– Cannot use on real network connections.
– Security is normally permissive.
© Stephen Janaway 2014

24. Exercise - Using Emulators
● Copy all files from USB stick.
● Windows:
o Run the -vbox installer -> installs VirtualBox and
Genymotion emulator.
● Mac:
o Install VirtualBox
o Install Genymotion
● When it’s finished, open VirtualBox.
● Import the .ova file (File -> Import Appliance).
● When it’s imported then start Genymotion.
● Select device and press ‘Play’

25. Exercise - Using Emulators
● Open the Applications folder on the USB
drive.
● Install the applications (drag into the
emulator window).
● Explore the apps and emulator options.
Hint - Use the mindmap….
30 mins study and 10 mins de-brief

26. Exercise - Using Emulators
What was good about
the emulator?
What could have been
better?

27. © Stephen Janaway 2014

28. Mobiles Are Complicated
• While your application runs, the phone is:
– Talking to the network
– Working out where you are
– Maybe receiving calls, texts, Facebook and Twitter
updates.
• Performance is important.
• Data usage is important.

29. Don’t Just Test Here…

30. On Here, and At Here…

31. Out Here…

32. In Here…

33. Real Devices
• Keep a device library or rent devices from
one.
• Cannot pick all devices.
• Choose a few at the top and a few at the
bottom.
• Do not forget to find out what is being used
by customers.
• Can be expensive and needs maintenance.
© Stephen Janaway 2014

34. Pro’s and Con’s - Real Devices
• +
– They are the actual device ☺
– Real world hardware and software
– Real world connections
– Enable real world testing
• -
– Expensive
– Rapidly become obsolete
© Stephen Janaway 2014

35. Open Device Labs
• Borrow devices, normally
for free.
• http://opendevicelab.com
• Open Device
Labs = shared community
pools of internet
connected devices for
testing purposes.
© Stephen Janaway 2014

36. My Recommendation
• Buy 2 from the top, 2 from the middle and 2 from
the bottom.
• Understand what your customers use or will use.
• Test first on simulators not only on simulators.
• Use cloud services if needs be, to fill any gaps.
• Consider crowd sourcing.
© Stephen Janaway 2014

37. Exercise - Mobile Challenge
● Use real devices.
● Pick an application and a device.
● You have 15 mins planning, 30 minutes
testing, 15 minutes reporting time.
● Use what you have learnt.
● There are prizes for the best bug or
observation!

38. Raising Defects
• Can you repeat the issue?
• What steps are needed to reproduce the issue?
• Is the issue repeatable on a fresh install of the application?
• What data is available?
• Other important information to include:
– Which device and OS version you were using.
– How the device was connected to a network (including which mobile operator
and/ or Wifi network).
– Where you were.
– What other applications were running.
• Don’t forget to include screen shots.
© Stephen Janaway 2014

39. Screenshots
• iOS -> Hold ‘Home’ and ‘Power’.
• Android:
– ICS and above -> Hold ‘Power’ and ‘Volume
Down’.
– Older -> Buy ‘No Root Screenshot It’
(https://play.google.com/store/apps/details?id=com.edwardkim.android.screenshotitfullnoroot)
• Windows Phone 8 -> Hold ‘Start’ and ‘Power’
• Blackberry -> Capture It OTA app
(http://www.captureitota.com/)
© Stephen Janaway 2014

40. Stephen Janaway
Questions?
www.stephenjanaway.co.uk
@stephenjanaway
www.testinginthepub.com

41. Automation Suggestions
• Determine if you require cross platform
support.
• Avoid tools that use embedded servers or
agents – they alter the production code.
• Look for tools with active online communities.
• Consider the future.
© Stephen Janaway 2014

42. My Picks
• Appium:
– Cross platform.
– Well supported user community.
– About to be more closely integrated in Selenium 3.
• iOS Driver:
– Well supported user community.
– About to be more closely integrated in Selenium 3.
• Calabash:
– Well supported user community.
– Heavily used at the BBC.
© Stephen Janaway 2014

43. Mobile Test Strategy – Top 5 Tips
1. Understand the potential user:
– Which platforms and devices will they use?
– How will they use the app or website?
2. Understand what OS versions are being used and
when new ones are coming.
3. Test using simulators, real devices and users.
4. Understand the cost of automation:
– It is where desktop automation was 6 years ago.
– Best combined with extensive unit and integration testing.
5. Use an exploratory approach to manual testing:
– Mobile device projects are typically short.
© Stephen Janaway 2014