1. 4149 Pennsylvania, Suite 202
Kansas City, MO 64111
Automated User Testing
Presented by
Chrys Sullivan
Founder and CEO, Useagility

2. An educational program sponsored and supported by
User Experience Professionals Association.
Using Automated Testing Tools
to Empower Your User
Research
Join now at www.uxpa.org for member-only pricing
UXPA Seattle Conference 2016
May 31 – June 3
uxpa2016.org

3. Rapidly Test Usability & Measure UX
UserZoom is the All-in-One UX Research Platform for the Data-
Driven Enterprise

4. • Chat box is available if you have any questions
• Look out for a quick poll question
• There will be time for Q&A at the end
• We will be recording the webinar for future viewing
• All attendees will receive a copy of the slides/recording
• Continue the discussion using #UZWebinar
Quick Housekeeping

5. Let’s understand each other’s background, roles and experience levels
Polls

6. About Useagility
6
Twitter: @Useagility
Chrys@Useagility.com
www.Useagility.com

7. Today we’ll learn:
7
How unmoderated testing works
and how you can use it in your organization.
01
Guidelines for selecting the right
testing methodology (moderated vs.
unmoderated) based on your project needs
and goals.
02
Steps for setting up successful unmoderated
studies.
03
Advice for recruitment, screening and use of online
panels.
05
Tips for avoiding the most common pitfalls in
automated UX testing.
04
Using automated testing with agile design and
development sprints to accommodate tight timelines
and satisfy usability needs.
06

8. Evolution of Usability Testing Technology
8
Lab-based testing
- Moderated testing
- In lab setting
Remote, facilitated testing
- Moderated testing
- Human facilitator and
participant interact via web-
conference
Automated, un-facilitated testing
- Unmoderated studies conducted
online via web-based applications
• No human facilitator
• Tasks are pre-set by researcher,
delivered through testing interface

9. Benefits of Unmoderated
9
 Easier recruitment and scheduling
 No worries about no-shows &
replacements
 Automated incentive payments
 Simultaneously test larger number
of participants (Quantitative)
 Less expensive per-study costs than
moderated testing
 Get results more quickly
 Built-in metrics (time on task, heat
maps, rating scale, SUS, etc)

10. Our Favorite Ways to Use Unmoderated User Testing
10
 Help Teams “Fail Fast”
 Testing design wireframes and
prototypes (Axure, Invision)
 Quickly weed out approaches that
don’t work
 Resolve internal disagreement
 Let users decide which design works
best
 Support Expert Review / Heuristics
Evaluations
 Validate suspected usability issues
 Lean UX Research with Agile
 Quickly test prototypes as part of agile
development cycles
 Prioritize backlog
 Measure usability impact for dev prioritization
 A/B testing
 Click-through / conversion testing

11. Limitations & Pitfalls of Unmoderated
11
 User panels and “professional testers”
 Stakeholders can’t observe live during
the study
 Can’t revise tasks once the study has
been launched
 Can’t adjust approach as sessions progress to
optimize depth of learning
 Lacks ability to probe unexpected insights
 No real-time support for users who get
significantly off-track
 IT’S NOT MODERATED

12. Selecting right methodology
12
Quantitative or Qualitative?
Moderated or Unmoderated?
Selecting the right methodology has the greatest impact on quality and
effectiveness of the study.

13. Define Goals
13
Hard to get business stakeholders to
define their goals. Ask them questions:
• What is the primary purpose of the
study?
• What do you need to know coming
out of this study?
• What will you do with the study
findings?

14. Define additional details
14
After goals are defined, understand
other key components:
• What is the scope of study? (Big,
small, how focused)
• What are we testing with?
(Prototype, wireframes, live site)
• Who are the target users?
• What tasks and scenarios will help
achieve study goals?

15. Which methodology is best?
15
 Will you need moderated follow-up and probing questions
to be successful?
 Are you at early concept and UX strategy phase?
 Do you need to go deep on attitudinal findings, mental
model and user preferences?
 Is it the first time you have tested a design? First time
testing with a new user type?
 Will it take longer than 20 – 30 minutes to complete tasks?
If yes = Moderated

16. Which methodology is best?
16
 Is the scope narrowly focused?
 Can testing tasks be accomplished in 30 min or less?
 Is testing of task success and usability a primary objective vs. diving
deep on user attitudes and preferences?
 Is there minimal reliance on attitudinal feedback that requires follow
up probing?
 Is this follow-on testing after moderated?
If Yes = Unmoderated

17. Creating successful test plans for unmoderated, automated usability testing has unique challenges.
Key Steps - Unmoderated Automated Usability Testing
PARTIPANTS TASKS
FOLLOW-UP
QUESTINS
ANALYSIS
Get right participants Design study and tasks
effectively
Use follow-ups
appropriately
Analyze finings
accurately

18. Participants
18
Many ways to recruit for unmoderated:
01
1. Email existing user base
2. Social media
3. Site intercept
4. Panel provide by UX testing vendor
*In all cases, screening for representative sample is
very important*

19. PITFALL #1: Panels
19
PITFALL#1: “Professional Testers” and
“Imposters”
SOLUTION: Over-screen
0101
1. Use 2x the screener questions as normal
2. As first task in study, ask a detailed question to
verify user is who they say
3. Over-recruit so you can throw out participants
you feel may not be valid

20. Writing Tasks
20
Crafting study plan & writing tasks
01
1. Remember, you won’t be there to help or clarify
2. Avoid leading verbiage that may cause bias in task
success
Instead of: View your current grades in Algebra
Use: Imagine you want to know your current grade in
Algebra
3. Ensure prototypes enable users to complete tasks
4. Anticipate how to proceed if they fail a task
5. Determine what metrics or data will be meaningful
given goals and context

21. Use follow-up questions
21
01
1. Use follow-up questions to check for understanding
Example: If the task was to locate their weighted-average
grade in Algebra class, ask what their “weighted-average”
means.
2. Use follow-up questions to probe for deeper insights
on why a user behaved as they did
Example: If the task required multiple steps in a process,
ask “were there parts of that process you were unsure
about or found confusing, tell me more about why you felt
that way.”

22. PITFALL #2 – Tasks skipped, misunderstood, leading
22
PITFALL #2 – skipped, misunderstood and
leading tasks
SOLUTION – Conduct pilot session
0102
1. Conduct one session as pilot
2. Watch results
3. Adjust tasks and flow, add follow up
questions to understand “why” if necessary

23. Analysis
23
Analyzing unmoderated testing
01
1. Watch all sessions in full
• (run at 1.5 speed)
2. Use integrated tools to tag video clips and
make notes

24. Analysis - Getting Lazy!
24
PITFALL #3 – Relying too heavily on metrics
SOLUTION: Use data appropriately and in
context
0103
1. Question self-reported task success rates
2. Balance stated “preference” for design with
analysis of user behavior
3. Give thought to which metrics are applicable and
relevant to your study based on context
• Time-on-task, heat maps, conversion, click-path

25. 25
Wrapping It Up

26. Now You Know
26
What unmoderated testing is and how
to use it
01
Importance of selecting right
methodology: moderated or unmoderated
02
Key steps to setting up and running successful
unmoderated testing
03
Common pitfalls in unmoderated testing and ways
to avoid them
04

27. Thank You!
27
Thank you for joining us today! Useagility contact info:
www.Useagility.com
chrys@useagility.com
Twitter: @Useagility

28. Need more info?
Get answers about UserZoom. We are here to help.
Talk with one of our knowledgeable staff.
Contact us
UserZoom
10 Almaden Boulevard, Suite 250
San Jose, CA 95113
Toll free number: (866) 599 1550
www.linkedin.com/company/userzoom@UserZoomFollow us