Recently, there has been a surge in the number of tools that are available to conduct unmoderated (“automated” or “asynchronous”) remote usability testing. This surge is changing the user experience industry and it forces us, whether we want to or not, to take a closer look at what the benefits and drawbacks are of unmoderated testing and whether or not we should incorporate it into our usability toolbox. In this session we will cover: what you can learn from unmoderated testing, how actionable the data is, how it’s conducted, when it should be conducted, benefits and drawbacks, and an overview of some unmoderated testing tools that are currently available. If there is time, a discussion with attendees about their experience with unmoderated testing will also take place. The content of this session is aimed at anyone who conducts usability testing or is interested in it. The material is good for both the novice and advanced UX practitioner. The content in this session is aimed at being 100% practical and applicable. Attendees should be able to immediately apply what they learn in this session to evaluate whether or not unmoderated usability testing is a good fit for them. This session will also give attendees the information they need to start conducting their own unmoderated tests. The material that will be covered in this session is an extension of an article that was written by the presenter for UXmatters.com: http://www.uxmatters.com/mt/archives/2010/01/unmoderated-remote-usability-testing-good-or-evil.php

1. Unmoderated
Remote Usability Testing:
?
Kyle Soucy
@kylesoucy
IA Summit 2011

2. Good?

3. Good?

4. Evil?

5. Evil?

6. What is Unmoderated
Remote Testing?

9. Unmoderated
Testing Tools
(There are a lot!)

10. We Won’t Talk About...

11. We Will Talk About...

12. We Will Talk About...

13. Has Anyone Here Used
These Tools Before?

14. Has Anyone Here Used
These Tools Before?

15. What You Can Learn
(Depends on the tool and how you use it!)

16. Web Analytics
Image source: http://www.hiero.com/web-analytics.html

17. Task Completion Rates
Image source: Loop11

18. Number of Clicks/
Clickstream
Image source: Userlytics

19. Time on Task/Page
Image source: http://blog.usabilla.com/underdogs-beat-expedia-in-usability-test/

20. Satisfaction Ratings/
Opinion Rankings
It is simple to use:
Strongly Disagree ---1---2---3---4---5---6---7 Strongly Agree
Image source: http://wikidoc.org/index.php/Pain

21. True Intent
Image source: http://ethnio.com/images/live_screener_example_page1.jpg

22. In-context Feedback
Image source: Loop11

23. In-context Feedback
“...I’m only offered a
search by model number.
How ridiculous.”
Image source: Loop11

24. In-context Feedback
Image source: Loop11

25. In-context Feedback
“...I would like to see the
products availability and delivery/
shipping information listed.”
Image source: Loop11

26. In-context Feedback
Image source: Loop11

27. How Actionable
is the Data?
(Depends on the questions you ask!)

28. Conducting
Unmoderated Remote
Usability Tests

29. Process

30. Process
1. Define the study

31. Process
1. Define the study
2. Recruit participants

32. Process
1. Define the study
2. Recruit participants
3. Launch test and send email invites

33. Process
1. Define the study
2. Recruit participants
3. Launch test and send email invites
4. Analyze results

34. Process
3.
Ru
na
1. Define the study Pilo
2. Recruit participants t tes
>
t!
4 / Launch test and send email invites
3.
5 / Analyze results
4.

35. Benefits

36. Test hundreds of people simultaneously while keeping
them in their own natural environment.

37. Test multiple websites simultaneously.

38. Reduced costs... depending on the tool you use.

39. Great way to plant the seed of UCD methodologies
and introduce usability testing into a company.
UX

40. Fewer logistics to manage and no need to worry
about no-shows and getting last-minute replacements.

41. Fast Results

42. Drawbacks

43. Nothing beats watching participants in real time and
being able to ask probing questions about what they
are doing as it’s happening.
Image source: facit digital

44. Some participants may only be interested in earning
the honorarium you’ve provided as an incentive.
$$$

45. You cannot conduct interview-based tasks.

46. What participants report on surveys can be very
different in comparison to what they actually do.
It is simple to use:
Strongly Disagree ---1---2---3---4---5---6---7 Strongly Agree
Image source: http://www.hiero.com/web-analytics.html

47. It’s possible for participants to think they’ve
successfully completed a task when they haven’t.
Image source: http://www.hiero.com/web-analytics.html

48. Does it matter if it took someone longer to complete
a task?
Image source: http://www.hiero.com/web-analytics.html

49. When to Conduct
Automated Testing?

50. Not a replacement for
moderated testing or research.
Best used to compliment your
qualitative research.

51. Unmoderated
Testing Tools
(A closer look...)

52. Unmoderated
Testing Tools
(A closer look...)

53. Automated Testing Tools
with Recruiting Panels

54. The danger...
I wonder how a real
customer would actually use
this website?
Panelist

55. UserTesting.com Video

56. UserTesting.com Video

57. The danger...
BLOCKED
BLOCKED

58. There are no
shortcuts
to conducting
good research.

59. Automated Testing Tools
with Other Recruiting Options

60. Loop11

61. Treejack

63. Chalkmark

64. Chalkmark Results

65. Automated Tools with
Audio and Video

66. Automated Tools with
Webcam Eye Tracking

67. /usableinterface
H andout:
Over view of
Unmo derated
Usa bility
Testing
Tools

68. Resources
• RemoteUsability.com
• Book: Remote Research
by Nate Bolt & Tony Tulathimutte
• Article: Unmoderated, Remote Usability Testing:
Good or Evil? by Kyle Soucy
http://www.uxmatters.com/mt/archives/2010/01/unmoderated-remote-usability-
testing-good-or-evil.php

70. www.usableinterface.com
@kylesoucy