Remote Usability Testing Best Practices and Challenges

Best Practices for Remote Usability
Testing
Comparative Benchmark Results

Jeff Sauro | Measuring Usability LLC
2/28/2012

5 Challenges Facing UX Professionals Today

1. Time
2. Cost
3. Tools & Techniques
4. Representative Users
5. Deliverables
Measuring Usability LLC

Deliverables

Business Requirement Doc Personas Wireframes
Function Requirement Spec Usability Test Plan User Screener
Task Analysis Doc Recruitment Doc Usability Report


Representative


Unmoderated Remote Testing w/ Video: Not a
Panacea but pretty close


Remote Testing Setup using UserZoom & Usertesting

Website is shown in
main window

Users are shown the task and asked to complete it.
Full A/V available online.

Collects Task Time, Completion Rates, Task-Ease and Click Paths

Case Study:


Summary
• 61 Total Remote Unmoderated Users
• 4 Tasks
• Within Subjects
• Randomized Tasks & Websites
• Recruitment Criteria: > 25 and US Based
– 40 % Had Rented a Car from Budget or Enterprise
• Mix of Age, Geography
• 58% Female
• Total Study Length : 20-30 minutes
• Users Recruited and Data collected in 48 Hours

2 Tasks
1. Find the Nearest Location
Find the address of the nearest rental office to the Hilton Hotel located at 921 SW
Sixth Avenue, Portland, Oregon, United States 97204. Write down or copy the
name of the street of the nearest location.

2. Renting a Car
Location: Logan Airport, Boston MA
Rental Period : Fri April, 13th 2012 at 11.00am to Sunday April 15th at 5.00pm.
Class: Intermediate (NO SUV’s)
Extras: GPS Navigation and Car Seat for a 2 Year old

If asked, use the following identification:
Name : John Smith
Email: john112233@mailinator.com
Phone : 303-555-1212
Credit Card: Visa


Task Completion Rates

Error Bars are 90% Confidence Intervals
* p <. 01 ; N = 62

Reasons for Failure by Task


Disaster Rates (Highest Confidence but Failure)


Task-Level Ease : Single Ease Question (SEQ)
Very Easy
Very Hard

* p < . 01 ; N = 62


Task Time: Completed Tasks Only
Seconds

# Users

* p < . 01 Measuring Usability LLC

Single Usability Metric (SUM)

Avg

* p < . 01

Completed Times: Find Nearest Location

Median Time with all Users: 119
Median Time Removed Long Times: 118


Why did the user take 7 minutes?

s.userzoom.com/m/MSBDMjUxUzM_ - Survey taker

Why?


Unmoderated Testing with Video


Verbatims
Totaling
It didn’t give the whole total with the extras-I had to add that myself.
Took the price of the car and manually added on 3 day price of car seat ($10.99*3) and same for GPS but still unsure of total
because unsure if these items are taxed.
I could not get a total price of the rental reservation, and the price I got was not a possible answer on the survey.
It told me that the price did not include the add-ons, and that those *may* be charged at the counter. I don’t know what it means
by may be charged at the counter- does that mean they many not be charged for? I don’t understand why it doesn’t give a
total cost of the rental with the GPS and toddler seat included. I added them in myself with a calculator, but I’m thought it
would just automatically show me what the total will be.
I was unable to find the link to add the GPS. I did read that it is 10.99 per day. I was also not able to find the link to add the car
seat.
you didn’t give an exact total for the booking. you said that the additional charges would be done at the counter. i think you
should have gave me an estimated total so i wouldn’t have had to compute the charges myself.
I just really wasn’t sure of my final pricing or even if I booked it right.
I had to use a calculator to add the GPS and car seat charges to the car rental. I would have preferred if I could have selected the
add-ons, and then it would show the total revised price before I booked the reservation. I was wondering if the add-ons
would also have other charges or taxes. overall easier than the last one.
Hard to know if updated price when adding the GPS and car seat
"The website was confusing to me. I didn’t understand why the rental amount did not display an increase after I added the GPS
and child seat to my order - it seemed like something was working incorrectly. I even accidentally submitted the reservation,
as I was trying to find that total.Basically, the website was not user-friendly. Items were difficult to find, and the interface
was confusing."
It was hard to understand what the total cost would be with the add-ons included.

Finding the Extras
I saw no area to add the extras
I saw no way to add GPS or child seat.
Because I didn’t know at what point it would offer the add-ons, and the total didn’t include the gps and car seat.
It wasn’t clear where to add the extras and once they were added it wasn’t clear what the final total would be. I also thought
payment details needed to be added before an actual booking would take place and they weren’t.
I couldn’t find the address for either task. I couldn’t figure out the options until I noticed them after I chose the car.
.

Reasons for Low Task Ratings on Renting a Car



Total Price Problem

http://accounts.usertesting.com/ViewClip.aspx?file=GSiU1W9elfLZjKxh
C0xflQ%3d%3d


Finding Location & Search Results


Usability Problems: Renting a Car
Extras Total Search Problem
User


User by Problem Matrices
Rent a Car Find a Location
Problem
User

P = .12 P = .15

P = .09 P = .07

Long Tail Problems

http://accounts.usertesting.com/ViewClip.aspx?
file=qLJy0lYuRV%2bZdKcfHYouqQ%3d%3d

Default Location


1st Click Analysis


First Click Analysis

• If a user’s first click is down the wrong-path,
only 46% eventually succeed

• If a user’s first click is down the right-path,
87% eventually succeed

Wolfson, et al (2008) Contextual card sorting (or FirstClick testing): A new methodology for
validating information architectures, UPA Proceedings


Find the Nearest Location: 83% Complete

42% Of Users
84% Success

50% Of Users
80% Success
3% Of Users
100% Success


Find the Nearest Location: 71% Complete

8% Of Users
100% Success

40% Of Users
10% Of Users
71% Success
83% Success

10% Of Users
50% Success

Test Level Metrics


Attitude before and After Usability Test
Very
Favorable
Un-Favorable
Very

Diff %

After users completed the tasks Budget’s Score increased 12% & Enterprises decreased 15%
* P < .05 ;
Error Bars are 90% Confidence Intervals and Paired t-test Used for Statistical Comparison

Net Promoter Score
How likely is it that you’ll recommend this website to a friend?

Enterprise Budget
30th Percentile of Websites 73rd Percentile of Websites


Usability
Credibility
Loyalty
Appearance

www.SUPRQ.com


SUPR-Q and Sub-Scales


Takeaways & Tips
• Pretest Tasks and Study Design
• Within Subjects
• Compared to What?
• Triangulate with Multiple Metrics
• Use Standardized Questions
– SEQ
– SUPR-Q
• Remote Unmoderated Testing w/ Video
– In Parallel
– As a Subset


Find out More

www.LeanUXDenver.com
Sept 19-20th 2012
Hyatt Convention Center
Denver, CO
@LeanUXDenver

MeasuringUsability.com
jeff@measuringusability.com

Twitter @MsrUsability


Remote Usability Testing Best Practices and Challenges

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Destacado

Destacado (20)

Similar a Remote Usability Testing Best Practices and Challenges

Similar a Remote Usability Testing Best Practices and Challenges (20)

Más de UserZoom

Más de UserZoom (20)

Último

Último (20)

Remote Usability Testing Best Practices and Challenges

Notas del editor