The official slides from our day-long workshop, Escape The Lab

1. Please
stand by
ESCAPE THE LAB
WILL BEGIN AT 9:00
Friday, May 7, 2010

2. ESCAPE THE LAB
LET’S GET STARTED MMKAY?
Friday, May 7, 2010

3. Friday, May 7, 2010

4. I’m Cyd
He’s Nate
Friday, May 7, 2010

5. #ETLAB
Friday, May 7, 2010

6. remote
USABILITY
Friday, May 7, 2010

7. oh
Friday, May 7, 2010

8. and
Friday, May 7, 2010

9. also
Friday, May 7, 2010

10. NO
Friday, May 7, 2010

11. time
Friday, May 7, 2010

12. travel
Friday, May 7, 2010

13. OR
Friday, May 7, 2010

14. lasers
Friday, May 7, 2010

15. HOW TO BE
graceful
When All Your
TECHNOLOGY
BREAKS
Friday, May 7, 2010

16. PLEASE DON’T TRIP
ON THE CABLES
Friday, May 7, 2010

17. Follow the buddy
system
Friday, May 7, 2010

18. How about you all?
Best remote research story
Most important thing to get out of
today?
Biggest fear or skepticism about
remote research?
Friday, May 7, 2010

19. Go TO escapethelab.com NOW
Friday, May 7, 2010

20. Friday, May 7, 2010

21. Remote Research
In person
Remote
Definition:
Research where the
participant is not
sharing physical space
with the researcher.
% of qualitative research
done remotely*
*Source: BP’s best guess
Friday, May 7, 2010

22. What we’ll cover today
Recruiting
Moderated
Unmoderated
Results
Pitfalls
Friday, May 7, 2010

23. We shall begin...
with a little experiment
Friday, May 7, 2010

24. EXERCISE
JUMP IN THE POOL
Table groups
Pick one moderator and one participant
(they need to be in different locations)
10 minutes of moderating
Friday, May 7, 2010

25. MODERATOR Instructions
Your participant has a sheet of paper and a link
Goal: find out what’s hardest about presenting
physical task instructions online
Hint: ask them to give you a play by play of what
they are doing
Follow along with your own paper
NO helping, even if you could.
Friday, May 7, 2010

26. It’s A RACE
First participant to make an origami
giraffe is the winner
Friday, May 7, 2010

27. Friday, May 7, 2010

28. Surprises?
Friday, May 7, 2010

29. When does remote win?
Page
10-15
Friday, May 7, 2010

30. Native Environment
Cheap ethnography, not cheap lab.
Friday, May 7, 2010

31. Real Time
Allows self-motivated tasks.
Friday, May 7, 2010

32. Involved Observers
Easier to collaborate in real time.
Friday, May 7, 2010

33. Geography
Friday, May 7, 2010

34. When does remote fail?
Friday, May 7, 2010

35. Bandwidth
Yes, it’s still an issue.
Friday, May 7, 2010

36. Translation
It’s expensive and difficult to do live.
Friday, May 7, 2010

37. Minors
Consent is a bear.
Especially if they’re under 13.
Friday, May 7, 2010

38. Faces
We’ve all got webcams today.
Most users don’t.
Webcams are invasive.
Friday, May 7, 2010

39. CASE STUDY: Epic FAIL
Friday, May 7, 2010

40. What went wrong
Coins were way more valuable than we
expected
People mis-assessed their English levels
Bandwidth was really a problem
Nobody plays Habbo when their parents
are home to consent
Friday, May 7, 2010

41. making the most
Friday, May 7, 2010

42. TIME-AWARE Tasks
Told to find purple pumps. Has to wear this for sister’s wedding this
weekend.
Doesn’t care.
Whole different interaction.
Friday, May 7, 2010

43. Real Time OBservers
More
on page
109-111
Friday, May 7, 2010

44. Change on the fly
“No one seems engaged in the “Let’s add questions about who they
laptop customizer” get recommendations from”
Friday, May 7, 2010

45. Designing Remote Studies
Friday, May 7, 2010

46. Friday, May 7, 2010

47. Friday, May 7, 2010

48. plan these things
Friday, May 7, 2010

49. plan these things
Method
Recruiting
Logistics
Friday, May 7, 2010

50. MODERATED must-haves
Friday, May 7, 2010

51. MODERATED must-haves
Consent
Friday, May 7, 2010

52. MODERATED must-haves
Consent
Tech set-up
Friday, May 7, 2010

53. MODERATED must-haves
Consent
Tech set-up
The beef
Friday, May 7, 2010

54. MODERATED must-haves
Consent
Tech set-up
The beef
De-installation
Friday, May 7, 2010

55. MODERATED must-haves
Consent
Tech set-up
The beef
De-installation
Thanks & incentive
Friday, May 7, 2010

56. UNMODERATED must-haves
Friday, May 7, 2010

57. UNMODERATED must-haves
Expectation setting
Friday, May 7, 2010

58. UNMODERATED must-haves
Expectation setting
Instructions
Friday, May 7, 2010

59. UNMODERATED must-haves
Expectation setting
Instructions
The beef
Friday, May 7, 2010

60. UNMODERATED must-haves
Expectation setting
Instructions
The beef
Thanks & incentive
Friday, May 7, 2010

61. Recruiting
Friday, May 7, 2010

62. Recruiting
Same: Good “talkers”
Same: Demographic fits
Friday, May 7, 2010

63. Recruiting
Same: Good “talkers”
Same: Demographic fits
Different: Relevant current tasks
Different: Compatible tech
Friday, May 7, 2010

64. Logistics: What & How
Provide test
target
Access
Credentials
Friday, May 7, 2010

65. Logistics: Who & When
How to connect with the user
(we’ll discuss in Tools)
Managing time zones
Session length
Friday, May 7, 2010

66. Paying Incentives
Collect minimal
personal info
Deliver quickly
Appropriate for
location
Friday, May 7, 2010

67. Advanced planning:
hybrid studies
Why use only one method?
Some methods don’t tell you the why
3-5 remote interviews is all you need
Friday, May 7, 2010

68. Portable Research
Friday, May 7, 2010

69. BREAK
Friday, May 7, 2010

70. Recruiting for Remote
Friday, May 7, 2010

71. We all need people
Relevant tasks
Compatible tech
Good talkers
Demographic fits
In the moment
Friday, May 7, 2010

72. Gold Standard Page
52
Live recruiting
Get as close as you can
Friday, May 7, 2010

73. Recruiting Options
Ethnio
Links or house ads to online forms
Recruiting agencies
Customer email lists
Panels
Craigslist/online ads
Friday, May 7, 2010

74. use Ethnio when
Website access
Some budget
Quotas are relatively simple
Friday, May 7, 2010

75. use forms when
You can place a link but not code
Low budget
Quotas are relatively complex
Options:
Google Docs
Wufoo
Friday, May 7, 2010

76. Google docs
Friday, May 7, 2010

77. use Agencies when
No website access
Higher budget
Online recruiting failed
Friday, May 7, 2010

78. Customer lists
Sub-segments
Ownership requirements
No website access
Friday, May 7, 2010

79. PANELS
Friday, May 7, 2010

80. Craigslist & forums
When other methods don’t work
No dough
Takes a while
Embarrassing to admit
Friday, May 7, 2010

81. the Gold Standard again
Live recruiting
Get as close as you can
Friday, May 7, 2010

82. Live Recruiting Math
(for interviews)
Target: 6 recruits/hour
Allow for: 1% response (low average)
You Need: 8-10,000 uniques/day
Friday, May 7, 2010

83. Page
56
Friday, May 7, 2010

84. Screener constraints
Page
60-63
Keep it short
Different from surveys
Who gets paid?
Friday, May 7, 2010

85. Remember the Point
Tasks that align with research goals
Technical compatibility
Control demographics
Friday, May 7, 2010

86. EXERCISE
Friday, May 7, 2010

87. Friday, May 7, 2010

88. Your best questions
Friday, May 7, 2010

89. GO EAT LUNCH
Famished
Friday, May 7, 2010

90. exercise
We recommend:
Split Pea Seduction, 6th & Minna
Custom Burger, 7th & Natoma
Tony Baloney’s, 7th & Howard
Tin Vietnamese, Howard & 6th
Harvest Market, Natoma & 8th
Sightglass Coffee, 7th & Folsom
Friday, May 7, 2010

91. exercise
We recommend:
Split Pea Seduction, 6th & Minna
Custom Burger, 7th & Natoma
Tony Baloney’s, 7th & Howard
Tin Vietnamese, Howard & 6th
Harvest Market, Natoma & 8th
Sightglass Coffee, 7th & Folsom
Friday, May 7, 2010

92. Please
stand by
ESCAPE THE LAB
WILL RESUME AT 1:15
Friday, May 7, 2010

93. welcome back
Friday, May 7, 2010

94. Time
for a
contest!
Tweet why you hate remote
research #ETLAB
Win a copy of Silverback
Friday, May 7, 2010

95. Remote Research
Toolkit
MODERATED
most of
Chapter 8
Friday, May 7, 2010

96. 5 easy pieces
Talk to the user
See what they’re doing
Take notes (optional)
Involve observers (optional)
Record audio & video (optional)
Friday, May 7, 2010

97. Communication
phone Skype
Reliable Cheap or free
Can be expensive Harder for users
for international
Can combine with
Conference calls screensharing to
needed for stress user’s
observers connection
Friday, May 7, 2010

98. Screensharing
Standalone Integrated
Adobe Connect UserVue (RIP)
LiveLook
FuzeMeeting (new)
WebEx*
GoToMeeting
*WebEx also has mobile versions
Friday, May 7, 2010

99. Adobe Connect
Friday, May 7, 2010

100. FuzeMeeting(New)
Friday, May 7, 2010

101. Livelook
Friday, May 7, 2010

102. Webex
Friday, May 7, 2010

103. GoToMeeting
Friday, May 7, 2010

104. Taking Notes
Second monitor if possible
Excel Hack (for time codes)
Friday, May 7, 2010

105. How to rig your notes
What the formulas look like
Auto timestamp!
Leave one column (A) empty for notes.
Set the cells of the next column (B) to record the clock time of each entry in A.
=IF(A2=“”,“”,IF(B2=“”,NOW(),B2))
Set the cells of the next column (C) to record the difference between the first entry and the current
entry
=IF(ISERROR(B2-$B$2),“”,B2-$B$2)
Fill down (Ctrl-D). Expect to use up to 120 rows for a session.
Don’t
Hide column B. You’ll never need to see it
write this
down, it’s on
page 104
Friday, May 7, 2010

106. Observer Involvement
IM
Web Chat Rooms
Integrated Chat
Passing Notes
Friday, May 7, 2010

107. Recording Tools
Hardware Software
Phone Patch Camtasia (Windows)
Cables iShowU HD (Mac)
Friday, May 7, 2010

108. Phone patch set up
Friday, May 7, 2010

109. Friday, May 7, 2010

110. Simple Set-up
Friday, May 7, 2010

111. Intermediate Set-up
Friday, May 7, 2010

112. Advanced Set-up
Friday, May 7, 2010

113. Friday, May 7, 2010

114. exercise
Roll your own
What will be your base set-up?
Write it down
Friday, May 7, 2010

115. Friday, May 7, 2010

116. Your version?
Friday, May 7, 2010

117. Winners!!
Friday, May 7, 2010

118. Moderating for Remote
Friday, May 7, 2010

119. Is it really any
different?
(Short answer: some.)
Page
94-99
Friday, May 7, 2010

120. Question from the BP
inbox yesterday
It would have actually helped to know how many users the authors
recommend for a tar* type of test. In a regular user test , many of us
have seen how findings repeat a er 5 -8 users of each user type are
observed. However, here we don't always know which task a user will be
doing when we interrupt them. So in cases where we really are following
users along as they complete what they came to do, how many useres
should we target ? Given that the authors have been doing this for a
while a recommendation from them on how many users to target and
their notes on the topic would have been really interesting.
*Time-Aware Research
Friday, May 7, 2010

121. Making passionate tasks
work
Friday, May 7, 2010

122. Making passionate tasks
work
Recruit for relevant current tasks
Script for customizable tasks
Friday, May 7, 2010

123. Re-script for remote
Friday, May 7, 2010

124. Re-script for remote
“Ok, I’d like you to start from the homepage,
pretend you are shopping for groceries for
a party, and buy some paper plates.”
Friday, May 7, 2010

125. MODERATED must-haves
Also see
37-43
Friday, May 7, 2010

126. MODERATED must-haves
Consent
Also see
37-43
Friday, May 7, 2010

127. MODERATED must-haves
Consent
Also see
Tech set-up 37-43
Friday, May 7, 2010

128. MODERATED must-haves
Consent
Also see
Tech set-up 37-43
The beef
Friday, May 7, 2010

129. MODERATED must-haves
Consent
Also see
Tech set-up 37-43
The beef
De-installation
Friday, May 7, 2010

130. MODERATED must-haves
Consent
Also see
Tech set-up 37-43
The beef
De-installation
Thanks & incentive
Friday, May 7, 2010

131. Chances are they’ve
never done this before
Friday, May 7, 2010

132. What they worry about
Friday, May 7, 2010

133. Who are you and can I trust you?
Friday, May 7, 2010

134. If you can see my screen,
can you see my face?
Friday, May 7, 2010

135. Will you be able to take
control of my computer?
Friday, May 7, 2010

136. Will you help me get the plug-in
off my computer at the end?
Friday, May 7, 2010

137. Will this work if I have a
firewall/proxy/strict IT department?
Friday, May 7, 2010

138. Is this going to end up on
YouTube?
Friday, May 7, 2010

139. Will I really get the incentive
if I do this?
Friday, May 7, 2010

140. How long will it take?
Friday, May 7, 2010

141. Is it OK if I have to take a quick call
in the middle?
Friday, May 7, 2010

142. Is it OK if my kids/dog/fratmates are
making noise in the background?
Friday, May 7, 2010

143. OXYMORON?
Remote empathy
Friday, May 7, 2010

144. Phone Interviewing
Specifics
Friday, May 7, 2010

145. Things sound leading
Pitfalls
Friday, May 7, 2010

146. Body language is huge
Actual language is important too
Make sure you’re comfortable
Friday, May 7, 2010

147. EXERCISE
Practice screensharing
Moderator, Participant, at least one
observer
Use the script in your packet as a
starting point
Get your participant all set up and
ready to start an interview
Friday, May 7, 2010

148. Friday, May 7, 2010

149. How did it go?
Friday, May 7, 2010

150. Advanced Techniques
High Emotions
Friday, May 7, 2010

151. Advanced Techniques
One to many moderation
Multi-threading
Friday, May 7, 2010

152. Future Frontiers
Friday, May 7, 2010

153. BREAK
Friday, May 7, 2010

154. Remote Research
Toolkit
UNMODERATED
Friday, May 7, 2010

155. 3 parts
Provide test target
Record user behavior
Capture comments (optional)
Friday, May 7, 2010

156. AUTOMATED Tools
Friday, May 7, 2010

157. What do you want to
do?
Get freeform data cheaply
Quantify user behaviors
Conceptual & IA testing
Friday, May 7, 2010

158. This stuff is pretty
cheap
• Usabilla $0-950/yr
• Optimal Workshop $559-1350/yr
• Loop11 $350/study
• OpenHallway $19-199/mo
• UserTesting.com $39/user
• Feedback Army $10/10 users
Friday, May 7, 2010

159. UserTesting.com
Friday, May 7, 2010

160. UserTesting.com
Friday, May 7, 2010

161. UserTesting.com
Friday, May 7, 2010

162. UserTesting.com
Friday, May 7, 2010

163. Using your own recruits
Set up an account
Email support@usertesting.com with:
the URL where your target lives
the scenario (500 char limit)
tasks (2000 char limit)
up to 4 questions to be answered in
writing at the end
Request a UserTesting URL for your
study
Friday, May 7, 2010

164. Loop11
Friday, May 7, 2010

165. Loop11
Friday, May 7, 2010

166. Loop11
Friday, May 7, 2010

167. Loop11
Friday, May 7, 2010

168. Loop11
Friday, May 7, 2010

169. Usabilla
Friday, May 7, 2010

170. Optimal Workshop
Treejack Chalkmark
Friday, May 7, 2010

171. EXERCISE
Choose one of these tools
Sign up for a test account
Create a 15-minute test for any
website you like
Friday, May 7, 2010

172. Friday, May 7, 2010

173. RESULTS
Friday, May 7, 2010

174. Session Videos
Heatmaps
Notes
Completion Rates
Card Sort Groups
Lions, Tigers, Bears...
Friday, May 7, 2010

175. Telling a story
If you have time-coded notes, you can make
highlight videos
If you have user comments, you can do tag
analysis or make word clouds
If you have numbers, you can compare to
benchmarks
If you have card groups you can do cluster
analysis
Friday, May 7, 2010

176. Case Study: Data
Mismatch
Friday, May 7, 2010

177. Qualitative Results
16 remote interviews
Worst findability BP has ever observed
Recommendation to burn it down and start over
Friday, May 7, 2010

178. Quantitative Data
EFFECTIVENESS RATIOS EFFICIENCY
Your task is to
make an
Success users Only
appointment with
a pediatric Time (mm:ss)
arrhythmia Mean time per task 01:36
specialist at
Standard deviation 00:47
Children’s
Hospital. Once you
have found it,
please remember Clicks
or write down the
6
phone number you Mean clicks per task
(exact value: 6)
would call to make
4
an appointment by Standard deviation
(exact value: 4.4)
phone, as you will
be asked for that
information later
on.
Answers distribution
??? ! The phone number search has been...
*[1=Extremely difficult / 7=Extremely easy]
1
2%
(1)
2
2%
(1)
3
10%
(5)
4
18%
(9)
5
18%
(9)
6
24%
(12)
7
26%
(13)
N
50
Mean*
5.2
Friday, May 7, 2010

179. What happened?
Friday, May 7, 2010

180. 32 page report, distilled
Navigation Search ease Hospital
ease of use of use reputation
1 2 3 4 5 6 7
terrible neutral UserZoom’s red excellent
flag minimum
Friday, May 7, 2010

181. What’s different IRL?
Friday, May 7, 2010

182. Recording
1 more button to press
Friday, May 7, 2010

183. Observer input
One more stream on your screen
(Possibly 2 or 3)
Friday, May 7, 2010

184. Notes
Tap, tap, tapping away
Friday, May 7, 2010

185. So, what could go
wrong?
Friday, May 7, 2010

186. 3 points of failure
Technology
yours
participant’s
observer’s
Recruiting
Test Design
Friday, May 7, 2010

187. Control what you can
control
Which is, your set up.
You wouldn’t go backpacking
without checking your gear.
Do it 24 hours in advance. Or else.
Page
240-241
Friday, May 7, 2010

188. Prepare for what
you can’t control
User’s dog trips over power
cord
Internet affected by local
weather
User can’t install
Doesn’t know how to install
Friday, May 7, 2010

189. Mid-Session
Tech
Problems
Friday, May 7, 2010

190. Live Recruiting
Problems
What if they gave a study and nobody
came?
Friday, May 7, 2010

191. Increase exposure
When you’re not getting
enough views
When the wrong type of users
are responding
Friday, May 7, 2010

192. Increase incentives
When it’s cheaper
When you’re almost getting enough
When segments are tough
Friday, May 7, 2010

193. Switch methods
When base traffic level isn’t high enough
Friday, May 7, 2010

194. Script Problems
Cut the BS.
Friday, May 7, 2010

195. The bottom line
If they’re not engaged,
something’s wrong.
Friday, May 7, 2010

196. Burn the script
Find out what matters.
Friday, May 7, 2010

197. QUESTIONS
Friday, May 7, 2010

198. @boltpeters
CYD@BOLTPETERS.COM
NATE@BOLTPETERS.COM
Friday, May 7, 2010

199. kthxbai.
Friday, May 7, 2010