The document discusses different strategies for testing mobile apps, including testing functionality, usability, performance, and more extensively than just functionality. It addresses testing on real devices versus emulators, the need for accessibility testing, and tools for testing areas like contrast, text-to-speech, location services and network bandwidth. The document also discusses visual testing strategies like using AI to detect visual differences and validating user interfaces.
Unveiling Design Patterns: A Visual Guide with UML Diagrams
Stop Testing (Only) The Functionality of Your Mobile Apps!
1. Stop Testing (Only) The
Functionality of Your Mobile Apps!
Anand Bagmar & Justin Ison
@BagmarAnand & @isonic1
2. Why do we need to think differently
about Mobile App Testing?
@BagmarAnand & @isonic1
2
3. Differences in Testing approach for
Web & Native Apps Testing
@BagmarAnand & @isonic1
3
4. ● Interactions
● Device features & capabilities
○ Form factors (sizes, rotations)
○ M-Browsers Vs Native
○ OS variations & compatibility with hardware
○ Hardware
■ Battery
■ CPU / GPU
■ Memory
● Impact due to External factors
○ Network fluctuations & variations when on the move
○ Temperature / Humidity / Moisture
○ Interruptions
● Release Approach
@BagmarAnand & @isonic1
Differences In Web & App Testing Approach
4
5. Differences in Release approach for
Web & Native Apps Testing
@BagmarAnand & @isonic1
5
6. Web Mobile
Rollback Can be achieved Not possible
Upgrade scenarios (OS flavors) Easy to validate Complex
Hot-fix Easy to deploy Can’t be forced on users
External Approvals Not required
Apple / Google App verification
process
Deployment size Does not matter to users App size has impact on users
Compliance Browser compliance
Stock / Custom OS + Hardware
compatibility + Permissions
@BagmarAnand & @isonic1
Differences in Web & App Release approach
6
7. Web Mobile
Rollback Can be achieved Not possible
Upgrade scenarios (OS flavors) Easy to validate Complex
Hot-fix Easy to deploy Can’t be forced on users
External Approvals Not required
Apple / Google App verification
process
Deployment size Does not matter to users App size has impact on users
Compliance Browser compliance
Stock / Custom OS + Hardware
compatibility + Permissions
@BagmarAnand & @isonic1
Differences in Web & App Release approach
7
9. Web Mobile
Rollback Can be achieved Not possible
Upgrade scenarios (OS flavors) Easy to validate Complex
Hot-fix Easy to deploy Can’t be forced on users
External Approvals Not required
Apple / Google App verification
process
Deployment size Does not matter to users App size has impact on users
Compliance Browser compliance
Stock / Custom OS + Hardware
compatibility + Permissions
@BagmarAnand & @isonic1
Differences in Web & App Release approach
9
10. Web Mobile
Rollback Can be achieved Not possible
Upgrade scenarios (OS flavors) Easy to validate Complex
Hot-fix Easy to deploy Can’t be forced on users
External Approvals Not required
Apple / Google App verification
process
Deployment size Does not matter to users App size has impact on users
Compliance Browser compliance
Stock / Custom OS + Hardware
compatibility + Permissions
@BagmarAnand & @isonic1
Differences in Web & App Release approach
10
11. Web Mobile
Rollback Can be achieved Not possible
Upgrade scenarios (OS flavors) Easy to validate Complex
Hot-fix Easy to deploy Can’t be forced on users
External Approvals Not required
Apple / Google App verification
process
Deployment size Does not matter to users App size has impact on users
Compliance Browser compliance
Stock / Custom OS + Hardware
compatibility + Permissions
@BagmarAnand & @isonic1
Differences in Web & App Release approach
11
12. Web Mobile
Rollback Can be achieved Not possible
Upgrade scenarios (OS flavors) Easy to validate Complex
Hot-fix Easy to deploy Can’t be forced on users
External Approvals Not required
Apple / Google App verification
process
Deployment size
Does not (necessarily) matter to
users
App size has impact on users
Compliance Browser compliance
Stock / Custom OS + Hardware
compatibility + Permissions
@BagmarAnand & @isonic1
Differences in Web & App Release approach
12
13. Differences in Web & App Release approach
Web Mobile
Rollback Can be achieved Not possible
Upgrade scenarios (OS flavors) Easy to validate Complex
Hot-fix Easy to deploy Can’t be forced on users
External Approvals Not required
Apple / Google App verification
process
Deployment size
Does not (necessarily) matter to
users
App size has impact on users
Compliance Browser compliance
Stock / Custom OS + Hardware
compatibility + Permissions
@BagmarAnand & @isonic1
13
14. So, how can you add more value to the
quality of your Native Apps?
@BagmarAnand & @isonic1
STRATEGY
14
15. @BagmarAnand & @isonic1
Real Device Vs Emulator-based Testing
https://askanydifference.com/difference-between-emulator-and-simulator/
STRATEGY
15
16. ● When to use Emulators / Simulators?
○ Able to validate all application functionality
○ No performance impact on the application-under-test
● Why use Emulators / Simulators?
○ Reduce cost
○ Scaled as per needs, resulting in faster feedback
@BagmarAnand & @isonic1
When & Why to use Emulators for Testing?
16
17. ● When to use Real Devices for testing?
○ If Emulators / Simulators are used, then “Sanity” / Focussed testing on real devices before
release
○ If Emulators / Simulators cannot validate all application functionality reliably, then invest
in Real-Device testing
○ If Emulators / Simulators cause performance issues or slowness of interactions with the
application-under-test
@BagmarAnand & @isonic1
When to use Real Devices for Testing?
17
18. ● Cases when Emulators / Simulators may not help
○ Streaming content
○ Applications relying on hardware capabilities
○ Applications dependent on customised OS version
@BagmarAnand & @isonic1
Real Device Vs Emulator-based Testing
18
19. ● It’s not enough to develop an app or any digital content for
that matter. You should strive to make your application
accessible to everyone.
● When should you start? The thought of accessibility should
start at the design or discovery phase or any product
and/or features.
● On this topic, we’re going to talk about a few of them and
how you can test your app for accessibility. Some of the
tools available, and even how to automate some of these.
Mobile Accessibility Testing
@BagmarAnand & @isonic1
STRATEGY
19
● According to the World Health Organization, there are
roughly 285 million people worldwide that suffer from low
vision, and/or color blindness.
21. Text To Speech - Voiceover
● Both iOS and Android platforms offer tools to verify voiceover logic.
○ Hearing the text-to-speech on the device/emulator/simulators.
○ Validating the accessibility labels via the layout or various hierarchy viewers apps.
“content-desc” for Android and “Label” for iOS. Both will result in “accessibility-id” in
appium for instance..
@BagmarAnand & @isonic1
21
● Your goal is to ensure people with low vision can utilize your
application, it functions correctly and it makes sense audibly.
24. ● The Android Studio and Xcode frameworks now for the most part handle layout
changes automatically due to different text sizes.
○ However, problems can occur when designers or developers use custom
fonts. They like any humans, make subconscious decisions that may adversely
impact how your app looks, functions and accessible.
Large Text Font & Display Density Size
@BagmarAnand & @isonic1
24
● For those with low vision, increasing the text and even the screen density size is
imperative. Normally these settings are enabled at the devices system level.
○ If these settings are changed from the default your app must be able to
support it.
26. ● Somewhat controversial when it comes to accessibility. But nonetheless, if
there is a population of people needing this, you should support this
feature and make sure it works, and the content of your app is readable.
● A few benefits:
○ Reduces flicker problems
○ Easier to use in poorly lit rooms
○ Less blue light exposure
@BagmarAnand & @isonic1
26
29. ● Nowadays there are many digital platforms such as phones/tablets, smart TV’s,
smartwatches, it’s important we have set standards to ensure our apps are accessible
to users with color blindness.
● And we do! The W3C - Web Content Accessibility Guidelines (WCAG 2.0 / 2.1) are a set
of standards and guidelines for mobile accessibility.
● Along with Levels AA and AAA.
○ Level AA (contrast minimum): Requires a contrast of at least 4.5:1 or 3:1 for large scale text.
○ Level AAA (enhanced): Requires a contrast of at least 7:1 or 4.5:1 for large scale text.
Contrast & Color Blindness
@BagmarAnand & @isonic1
29
● Many people throughout the world suffer from color blindness.
30. ● The Accessibility Scanner app scans your screen and provides suggestions to improve the
accessibility of your app. This app uses the Accessibility Test Framework and provides specific
suggestions after looking at content labels, clickable items, contrast, and more.
● You can automate this with Espresso Accessibility Checking libraries.
○ https://developer.android.com/training/testing/espresso/accessibility-checking
Android - Tools Available to Validate Contrast
@BagmarAnand & @isonic1
30
32. ● Apple provides the Accessibility Inspector testing tool that can be launched from Xcode > Open
Developer Tool > Accessibility Inspector. The Accessibility Inspector presents a utility window
that displays the information properties (and values), action methods, and position in the
accessibility hierarchy.
● Automation options are limited, however.
○ https://mobilea11y.com/guides/xcui/
○ https://github.com/google/GSCXScanner
○ https://www.deque.com/ios-accessibility/
iOS - Tools Available to Validate Contrast
@BagmarAnand & @isonic1
32
36. Accessibility
Now let’s tie everything we just learned
with automation example using Applitools!
Default - Contrast - Large Text - Device Density - Dark Mode
@BagmarAnand & @isonic1
36
39. Let’s look at the results in Applitools!
@BagmarAnand & @isonic1
39
40. ● Consumption Monitoring:
○ CPU
○ Memory
○ Network
○ Battery/Energy
● It's not only important to capture this information, but also storing it
for historical lookup, benchmarking, and detecting trends!
● Android SDK Tools & Xcode provide libraries to test these. We will walk
through the various options available.
Devices Resources & Consumption Testing
@BagmarAnand & @isonic1
STRATEGY
40
● Nowadays, it's not enough to only have tests to ensure quality and
catch regressions. It's imperative to know more about what’s
happening under-the-hood of your application.
47. ● You can also stress test your application with monkey testing. Monitor
the logs for errors and even use the profiling tools we talked about
previously.
● We’ll look at few different frameworks available for iOS and Android.
○ These should be used as an additional tool in your toolbox and not to replace
automation.
Chaos & Monkey Testing
@BagmarAnand & @isonic1
STRATEGY
47
● Testing the unexpected! Regressions tests (automated and manual) are
a must-have but they only take you so far. Using a monkey tester can
discover bugs you never knew existed.
49. ● Perhaps one of the easiest frameworks to
get started with. You can develop your
own Monkeyrunner scripts in Python and
Java.
● Or use the Android Device Bridge APIs and
run it directly from a terminal.
Android - Monkeyrunner
@BagmarAnand & @isonic1
49
52. ● Many applications today use location-based
logic where they geolocate you to display
certain UI views.
● It’s important to validate these cases by
changing the device location.
● In some cases, this is not enough and a VPN
is needed if using network-based location.
Or use additional spoofer / mock location
apps to achieve this.
Location Testing
@BagmarAnand & @isonic1
52
53. ● Many applications today need to handle different networks
and bandwidths. Especially in developing countries where
networks can be spotty.
● Apps need to logically detect when certain network
conditions are met and what view contents need to be
downloaded or otherwise show static views
● Bandwidth can be adjusted for automation as well. Some
links at the end will be provided for further explain how.
Network Bandwidth Testing
@BagmarAnand & @isonic1
53
61. @BagmarAnand & @isonic1
61
Use Visual Assertions Instead of Functional Assertions
● A single assertion for complete
functional coverage
● Bonus: validates the UI
● Does not break when the UI changes
● No coding skills required to maintain
baselines
● Validate UX at scale for all supported
browsers
62. @BagmarAnand & @isonic1
Beta Testing
● Beta Testing
○ Test Release candidate apps with users willing to use and provide feedback for early
releases of Apps
○ Instrumentation / Reports need to be able to separate between Beta releases Vs
actual release versions of the App
STRATEGY
62
63. @BagmarAnand & @isonic1
On-field Testing
● On-field Testing
○ Test Release builds of apps with real users identified in the field
○ Share specific criteria and scenarios for validations
Examples
■ Network and network conditions on which testing is to be done
■ Device criteria on which testing is to be done
■ Test scenarios to be executed with relevant Test Data
○ Leverage companies that offer on-field testing as a service
STRATEGY
63
64. @BagmarAnand & @isonic1
Staged Rollouts
● Staged Rollouts
○ Release apps to a % of users
○ Rollout % can be increased over time based on confidence on quality
○ App can be pulled out if problems are noticed / reported
○ Google PlayStore Staged Rollouts:
https://support.google.com/googleplay/android-developer/answer/6346149?hl=en
○ Apple Phased Releases:
https://help.apple.com/app-store-connect/#/dev3d65fcee1
STRATEGY
64
66. @BagmarAnand & @isonic1
Observability - Logging / Monitoring / Analytics
● Observability - The ability to dig into unknowns on the fly
● To enable Observability, ensure:
○ Capture Crash logs with context
○ Build Instrumentation & Correlation with Context in your app
○ Capture user interactions in form of Business and Technical Analytics Events
○ Enhanced Real-time Monitoring & Alerting for your system components and
infrastructure
○ Build Meaningful, Real-time and Contextual Dashboards
● Test for Observability before a full release
66