1. Webinar:
How to Conduct UX Benchmarking
May 2nd, 2012

2. Intros
Alfonso de la Nuez Kim Oslob
Co-Founder & Co-CEO Sr. Client Services Director
Twitter Hashtag #uzwebinar

3. Overview
 What we mean by UX Benchmarking and why it’s important?
 How to conduct UX Benchmarking studies, using remote
unmoderated usability testing (aka task-based surveys)?
 What are some of the UX metrics you should look for?
 How to measure the quality of a website’s user experience across
various competitors?
 A case study: American Airlines vs Delta Airlines Websites

4. What is Benchmarking?
Definition of Benchmarking:
 The process of comparing the business
processes and performance metrics
including cost, cycle time, productivity,
or quality to another that is widely
considered to be an industry standard
benchmark or best practice.
 It provides a snapshot of the
performance of your business and helps
you understand where you are in
relation to a particular standard.

5. What is UX?
Definition of UX:
 It’s about how a person feels about using a
system. It highlights the experiential,
affective, meaningful and valuable aspects
of HCI and product ownership, but it also
covers a person’s perceptions of the
practical aspects such as utility, ease of
use and efficiency of the system.
 User experience is subjective in nature, as
it is about an individual’s feelings and
thoughts about the system. User
experience is dynamic, because it changes
over time as the circumstances change.

6. Ok… So what is UX Benchmarking?

7. Is UX Benchmarking…
Not really

8. What is UX Benchmarking?
 The process of comparing a website’s (or any
digital product) performance, specifically in
relation to UX metrics, including among
others effectiveness, efficiency, overall
satisfaction and quality perception, to another
that is widely considered to be an industry
standard benchmark or best practice
 UX Benchmarking provides a snapshot of the
performance of your product and helps you
understand where you are in relation to a
particular standard.

9. The question we’d like to address today is:
 How do you conduct UX Benchmarking?

10. In order to Benchmark, first you need to measure

12. Why is UX Benchmarking important?
 Measuring the quality of UX is more important than ever
 Consumers have gained a lot of power in recent years
 More competition, more maturity in the online marketplace
 You can’t really manage what you can’t measure. UX Pros must
deliver more measured value. Mgmt Teams LOVE metrics…
 UX Benchmarking allows you to MEASURE performance in two
ways:
1. Baseline studies: Measure UX over time, compare evolution of your
own product and/or service
2. Against competitors within your industry

13. Common UX research & design challenges
 Difficulties obtaining objective, statistically significant, actionable
data
 Obtaining idea adoption from stakeholders
 Geographic/demographic limitations
 UX research is costly and time-consuming
 UX Benchmarking is therefore quite challenging
This is where large-scale, remote unmoderated
usability testing really helps

14. 3 pillars of UX Benchmarking
1. Plan &
Goals
UX Benchmarking
2. UX research
3. Recruiting
participants
methods &
tools

15. 4 must-dos with UX Benchmarking
1. Plan ahead:
 Define measurable & comparable UX-related KPIs (will show them in the case study ahead)
 Select competitors to benchmark against
 Define target audience
 Identify common site activities/tasks (recommended between-subjects design)
2. Conduct research using task-based testing (see methods and tools later on)
3. Gather both quantitative & qualitative data
 Use the quantitative data to test for statistically significant differences between
means or proportions
 Use qualitative data to support your findings and understand the why
4. Research at least twice a year (or even quarterly) to measure design changes
and trends in expectations

16. Define measurable & comparable UX-related KPIs
 Jeff Sauro,
Measuringusability.com

17. Define measurable & comparable UX-related KPIs
 Jeff Sauro,
Measuringusability.com

18. UX research methods & tools
We recommend the following combination:
• Remote Unmoderated Usability Testing (or task-based survey tools)
• Remote Unmoderated Usability Testing Video Tools

19. What about recruiting participants?
 3 choices to recruit participants:
1. Panel vendor (SSI, Research Now, etc)
2. Private Mailing List
3. Intercept visitors

20. Case study:
Benchmark Study to compare AA.com vs Delta.com

21. Study Background
• Competitive benchmark of American Airlines vs. Delta
Study
Air Lines websites
• Using task-based surveys, or Remote Unmoderated
Usability Testing
• Date: April 24 – April 25, 2012
• 100 participants (50 per site)
Panel
• Must fly for business or leisure more than twice a
year
• Must use websites to make travel arrangements
• Panel source: Survey Sampling International (SSI)
Tasks
• Go through the entire process of booking a flight
• Locate the total price for checking in two bags
www.userzoom.com

22. Task-based Surveys
(aka unmoderated large-scale remote usability testing)
Go through the entire process of booking a flight
Go through the entire process of booking a flight
• Large sample of participants take the study simultaneously
• Natural context
• International/domestic testing
• No moderation needed
• Usability + Behavior + Feedback
www.userzoom.com

23. Study Design
American
Airlines Task 1 Task 2
Screener Pre-task questions
Follow-up q’s
Delta
Air Lines Task 1 Task 2
www.userzoom.com

24. Study Design
Pre-task brand attributes: Which of the following words do
Pre-task you associate with the AIRLINE brand?
Go through the entire process of booking a flight
Tasks Locate the total price for checking in two bags
Self-reported success
How satisfied were you with the process of booking a flight?
Ease of Use
Post-tasks Ease of navigation
Clear where to start
Satisfaction with amount of time to complete task
Overall rating
Post-task brand attributes
Wrap-up What did you like the most about the AIRLINE website
What did you like the least about the AIRLINE website
Likelihood to recommend (Net Promoter Score )
www.userzoom.com

25. Participant Profile
Age range Gender
Under 18
14% 18-20
8% 31% 21-30
45% Male
31-40
55% Female
41-50
27%
20% 51-60
60+
How often do you fly yearly (for business or leisure)?
I don't fly regularly
10+ times
Participants were
5-10 times required to book
their travel online in
2-4 times order to qualify
1 time
0% 20% 40% 60% 80%
www.userzoom.com

26. Brand Perception
%
Pre Post Change Pre Post % Change
Friendly 41% 14% -27% 42% 10% -32%
Customer 27% 27% 0% 26% 40% +14%
Focused
Safe 57% 27% -30% 44% 24% -20%
Good 41% 31% -10% 36% 22% -14%
value
www.userzoom.com

27. Task 1:
Finding a round-trip flight
Your Task: Find a round-trip flight departing May 1st and returning May 6th that you would consider booking on Delta Air
Lines/American Airlines website. You are travelling from San Francisco, CA to New York, NY. After that, please go through
the entire process of booking a flight until you get to the "Purchase" button.
Delta Air Lines American Airlines
Task Success Task Success
Median Success Time Median Success Time
3:43 3:56
On both studies the majority of participants self-reported being able to successfully complete the
booking process. There was not a significant difference in success rate or completion time (90%
Confidence Level).
www.userzoom.com

28. Task 1:
Finding a round-trip flight
Post-Task Questionnaire
Top Box Scores*
How satisfied were you with the process
of booking the flight? 50% 55%
Booking the flight was easy
64% 67%
It was easy to navigate the website
64% 65%
It was clear how to start booking the
flight 62% 82%
I am satisfied with the amount of time it
took to complete the task 64% 80%
Participants that completed the American Airlines task reported being clearer on where to start and
more satisfied with the time it took to complete booking (90% Confidence Level). All percentages
represent top box scores.
*Top box scores show the percentage of participants that agreed strongly with the statements (selected 6-7 on a 1-7 scale)
www.userzoom.com

29. Task 2:
Find the price for checking 2 bags
Your Task: Find the total price for checking in two bags for your flight from San Francisco to New York.
You purchased a standard economy class ticket.
Delta Air Lines American Airlines
Task Success Task Success
Median Success Time Median Success Time
1:48 2:00
On both studies the majority of participants were not able to successfully identify the price of
checking two bags. There was not a significant difference in success rate or completion time (90%
Confidence Level).
www.userzoom.com

30. Task 2:
Heatmaps - First Click
On both Delta and American Airlines participants used similar strategies when searching for baggage
checking fees. The most commonly used strategy was to start by clicking the flight booking form
fields/buttons (28% Delta “To Airport” field and 24% American Airlines “Find Flights” button).
Participants that first located and clicked on a “Baggage” had higher success rates (70% Delta and 60%
American Airlines).
www.userzoom.com

31. Task 2:
Find the price for checking 2 bags
Post-Task Questionnaire
Top Box Scores*
How satisfied were you with the process
of searching for the total price for 30% 20%
checking in two bags?
Finding the total price was easy
32% 24%
It was easy to navigate the website
38% 36%
It was clear how to start searching for
the total price 32% 32%
I am satisfied with the amount of time it
took to complete the task 34% 34%
Participants that completed the Delta task reported being more satisfied with the process and ease of
the task. The differences are not significant (90% Confidence Level). All percentages represent top
box scores.
*Top box scores show the percentage of participants that agreed strongly with the statements (selected 6-7 on a 1-7 scale)
www.userzoom.com

32. Final Questionnaire:
What did you like the most about AA/Delta?
Representative Quotes: Representative Quotes:
“very easy to navigate” “the prices”
“easy to find flights” “easy to navigate”
“The modern look of it”
Several of the Delta participants commented that they liked the appearance of the Delta website.
There were multiple American Airlines participants that responded liking American Airline’s pricing.
www.userzoom.com

33. Final Questionnaire:
What did you like the least about AA/Delta?
Representative Quotes: Representative Quotes:
“the pricing” “finding information regarding
“Not easy to find bag fees” baggage pricing”
“Prices”
“finding things other than straight
flight booking”
For both websites participants commented that the pricing was high and that it was difficult to locate
the bag fees.
www.userzoom.com

34. Final Questionnaire:
Net Promoter Score
NPS = % of Promoters
(9s and 10s) - Detractors
(0 through 6)
0 1 2 3 4 5 6 7 8 9 10 NPS
0% 4% 4% 4% 6% 18% 6% 10% 22% 6% 20% -16%
Delta Air Lines
4% 6% 4% 0% 4% 10% 12% 18% 14% 12% 18% -10%
American Airlines
Detractors Promoters
When asked if they would recommend the Delta/AA website to a family member,
friend, or colleague both sites had negative NPS scores.
www.userzoom.com

35. Some takeaways…
1. UX Benchmarking, while challenging, is not mission impossible
2. Given the right plan, methods and tools, results are really valuable
3. UX Pros must deliver more measured value
4. There are plenty of UX metrics to be measured, including:
 Task success
 Time on task
 NPS
 Reasons for problems, abandonment
 The WHYs
 Etc.

36. Q&A

37. Thanks so much for your time!
Check out our upcoming news and events at www.userzoom.com
Sunnyvale (USA) Barcelona (Spain) Cheshire (UK)
440 N. Wolfe Rd. Sunnyvale, Av. Diagonal 419 3º 2ª 08008 Booths Park. Chelford Road, Knutsford,
CA 94085 Barcelona Phone: +34 93 414 Cheshire WA16 8GS
Phone: +1 (408) 524 7445 7554 Fax: +34 93 209 8380 Phone: + 44 (0)1565 759890
Contact: Alfonso de la Nuez Contact: Xavier Mestres Cell: + 44 (0)7900 472 920
Contact: Arthur Moan