The stakes in the mobile app marketplace are very high, with thousands of apps vying for the limited space on users’ mobile devices. Organizations must ensure that their apps work as intended from day one and to do that must implement a successful mobile testing strategy leveraging in-the-wild testing. Matt Johnston describes how to create and implement a tailored in-the-wild testing strategy to boost app success and improve user experience. Matt provides strategies, tips, and real-world examples and advice on topics ranging from fragmentation issues, to the different problems inherent in web and mobile apps, to deciding what devices you must test vs. those you should test. After hearing real-world examples of how testing in the wild affects app quality, leave with an understanding of and actionable information about how to launch apps that perform as intended in the hands of end-users—from day one.

1. W14
Test Techniques
5/7/2014 3:00:00 PM
Testing in the Wild: Practices for
Testing Beyond the Lab
Presented by:
Matt Johnston
Applause
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

2. Matt Johnston
Applause
With more than a decade of experience at companies ranging from early-stage startups to
publicly-traded enterprises, Matt Johnston leads the uTest and Applause organizational strategy
efforts. Since joining uTest as employee #8 in late 2008, Matt has provided a technical
sensibility to uTest’s engagement of testers and engineers. As CMO/CSO, he leads uTest’s
large-scale initiatives, such as partnership formation and helping to define the company’s
expansion strategy. Matt is a frequent speaker at top technology and mobile conferences,
including STARWEST, CTIA, and Mobile App World. A renowned expert on crowdsourcing,
Matt’s work has appeared in the bestseller Crowdsourcing for Dummies.

3. 4/26/2014
1
|
Testing In the Wild: Practices for
Testing Beyond the Lab
Matt Johnston | CMO/CSO, uTest | matt@utest.com | @matjohnston
April, 2014
The Challenge
||
App Quality
Then vs. Now

4. 4/26/2014
2
|
The Way Things Were
You Tell Them What To Think
|
The Way Things Are TODAY
They Tell You What They Think

5. 4/26/2014
3
|
Mobile App Quality Is Judged More
Harshly
• Users less tolerant of
spotty quality
• Social & app stores
give every user a
megaphone
• Switching costs
lower than ever
• Cost of imperfect app
quality has spiked
|
So What Could Possibly Go Wrong?

6. 4/26/2014
4
|
Cars
E-ReadersHousehold Appliances
Fragmentation of users, devices, OSes, browsers, locations
Oh, And It’s Not Just Mobile
Phones & Tablets
Laptops & Desktops
Connected TVs
Gaming Consoles
|
• Inside-out quality perspective is valuable but
incomplete
– Regardless of mobile app type or dev & test methodologies
• Replaced by outside-in quality worldview
– Quality signals come in many forms, and from USERS
- Bug reports
- Test cases executed
- Crashes
- User feedback
- Analytics
– And at many times (continuous)
No Turning Back Now

7. 4/26/2014
5
|
• External Signals Inform Internal Decisions:
– What you think matters less than what users think
– How YOU measure app quality matters less
– What users feel and say matters MORE
– Court of public opinion rules
- App stores
- Social media
What’s That Mean?
The Challenge
|| 10
Ummmm
So What Do We Do?

8. 4/26/2014
6
|
Stuff Gets Built; Then It Breaks
1. Marketing identifies a need
2. Product specifies a solution
3. Engineering builds the product
4. QA tests the application
5. Help desk listens to complaints
6. Sales & execs blame everyone
7. Then we do it all over again and
hope for better results
11
Vicious Cycle
| 12
Lab-Based Testing Innovation
Agile vs.
Waterfall
Manual vs.
Automation Managing vs.
Leading Teams
Exploratory vs.
Test Cases
Testers vs.
Engineers
In-House vs.
Outsourced
Offshore vs.
Nearshore

9. 4/26/2014
7
|
Mob Mentality
BUT crowds often look (and act) like unruly mobs
13
|
From Mob to Community
And mobs don’t work in every category
Example: a skilled service like software testing or
requires an orderly and professional “community”
capable of consistently delivering the desired results
14

10. 4/26/2014
8
|
The challenge is to turn this –
15
|
Into this
16

11. 4/26/2014
9
The Challenge
|| 17
The Crowd:
Testing In-the-Wild
|
Community Management 101
• What Good CM Isn’t
– It’s not customer service
– It’s not marketing or social media
• What Good CM Is
– 1st gen reactive & narrowcast (customer service)
– 2nd gen proactive & broadcast (marketing)
– 3rd gen independent & self-serve (program management)
– 4th gen interdependent & vetting, teaching and policing one
another (community management)
18

12. 4/26/2014
10
|
• CM goals – to build relationships, develop loyalty and nurture
desired behavior
– Get beyond 1:1 discussions
– Get beyond reactive corrections
• CM Challenges – to do the following at scale
– Recruiting Vetting Engaging Incenting Managing
• Tool Kit
– Forums & message boards > email responses
– Help topics & tutorials > repeatedly answering same questions
– Webinars > phone calls
– Profiles
– Matching
– Ratings
– Reputations
– Payments
19
Community Management Tool Kit
|
• Top c-sourcing firms use two forms of compensation
– Monetary
– Reputation
• Performance-based ratings based upon a dozen factors:
– Participation Level:
- Lifetime & recent participation: # active test cycles, # reported issues
– Quality of Participation
- Approval percentage for bugs, test cases and usability surveys
- Accuracy of bug type & severity classifications
- Ability to write test cases & create automated test scripts
20
Rating & Reputation System

13. 4/26/2014
11
The Challenge
||
uTest Case Study:
USA Today
|
• Timeline of USA Today Pioneering
– 1982: 1st nat’l paper to use full color
– 1996: Early adopter of web
– 2008: 1st major news brand with native apps
- iOS, Android, Windows, Kindle, web TVs, gaming consoles
– 2012: Re-designed & re-launched entirely new UI/UX across all
- Print + website + 14 native apps
- First web exceeded print; now apps exceed web
At the Forefront of Innovation

14. 4/26/2014
12
|
• Great for USA Today’s readers
• Great for USA Today’s sales & marketing
• Vast pressure on tech execs to deliver quality across:
– Device makers & models
– OSes & browsers
– Carriers
– Languages (localized content)
– Locations (geo-based intelligence)
Impact on Testing
|
USA Today’s Testing Portfolio
In-House
Manual
Testing
Outsourced
Manual
Testing
In-House
Test
Automation
Outsourced
Test
Automation
In-The-Lab
Testing

15. 4/26/2014
13
The Challenge
||
The Goal:
High Bar For App Quality
|
• Maintain Quality
– Launch nothing less than a 4.5 star app
– Each new version, across 14 native apps
• Increase Test Coverage
– Real-world conditions
– Cost-effective
– Scalable
• Meet Tight Deadlines
– Must have apps ready to win
- For each new platform that gains traction
- For each new device that gains traction
The Goals

16. 4/26/2014
14
The Challenge
||
The Approach:
Achieve In-the-Wild Testing
|
• Traditional Outsourcing
– Lives in a lab
- Further removed from users
- Geo + demo
– Cheaper vs. in-house
– Hidden costs
- Time zones & language
- Scaling concerns and costs
– High noise-to-signal ratio
- No transparency
Two Distinct Choices
• Crowdsourcing & expertsourcing
– Lives beyond the lab, in the wild
- Professional testers + real-world conditions
– Mirrors USA Today’s user base
- Real-world conditions
- Tech + geo + demo
– Blended team
- Recurring testers (incl on-site) + crowd testers
- Dedicated uTest project manager
– Scales to match shifting needs

17. 4/26/2014
15
|
• In-the-wild testing solution for USA Today
– Provide on-demand complement to in-house QA team
– Reduce escape rate by testing beyond the lab
- Across OS, browsers, devices & carriers
- Across locations, languages, industries & hobbies
– Blended test team built into USA Today’s dev & test lifecycle
- Dedicated uTest project manager
– Tester management
– Triaging services
- Team of dedicated testers (remote or on-site) + large team of crowd testers
Their Choice: Partner with uTest
The Challenge
||
Results:
Apps That Win

18. 4/26/2014
16
|
• Reduced escape rate
– ID’d and documented 1,000+ bugs across 6 apps in first 90 days
• Improved usability
– Several iterations of usability testing & analysis
• Crowd details
– Testers from US, Canada, Brazil, UK, France, Germany, China & Japan
– Some projects (eg: brand new iPad app) involved 80+ testers
– Others with tighter scope required smaller teams of 10-12 testers
• Hit the ground running
– Launching Kindle Fire app same day the device launched
– Hand-picked testers pre-purchased various models of device
– Tested their app with all over the world, almost instantly
• Bottom line: better apps
– Enabled USA Today to keep innovating, while maintaining superior quality
– USA Today able to consistently meet 4.5 star app-store rating goal
Results
|
Customer Satisfaction
The Final Word
“uTest gives our organization a much better sense of where we
stand in terms of application stability and performance. For a group
that iterates as often as we do, that type of insight is critical”
-- Tim Carlson,
Director of Mobile Product Development, USA Today

19. 4/26/2014
17
The Challenge
||
In-the-Wild Testing
Specific Takeaways
|
Key Takeaways: In-the-Wild Testing
• Apps universe has forever changed testing
– Exponentially more diverse user environments
– Devices & configurations
– Locations
– User demographics
• In-the-wild testing is a necessity
– Some testing must be done beyond the lab
– In-the-wild where app users live, work & play
• Agility is crucial to app success
– Test team size and tester location enterprise-friendly mobile testing
• Partner selection is vital
– Not all created equal
– Alignment of capabilities, needs and philosophy
– Track record of relevant enterprise success

20. 4/26/2014
18
The Challenge
|| 35
Questions?
Answers
Matt Johnston | CMO @ uTest
mattj@utest.com | @matjohnston