Check out these slides

1. BENCHMARKING MINI-SERIES PART #2
Conducting Quick, Cost-Effective
Benchmarking at Scale
Presented by: Dana Bishop, Sr. Director of UX Research

2. Quick Housekeeping
• Control panel on the side of your screen if you have any
comments during the presentation
• Time at the end for Q&A
• Today’s webinar will be recorded for future viewing
• All attendees will receive a copy of the
slides/recording
• Continue the discussion using #uzwebinar
• Stay tuned until the end for the raffle winner announcement!
Let’s make sure you’re all set up for the webinar!

3. What We’re Covering in Part #2
Planning & conducting UX benchmarking
Collecting and tracking data with scorecards
UserZoom’s qxScore & Scorecard program
10 expert benchmarking tips
Dana Bishop
Sr. Director, UX Research
Dana has 25 years of industry experience,
with a focus on UX Benchmarking and
Syndicated Research.
Email dbishop@userzoom.com

4. Can UX be measured?
● UX Benchmarking is critical because it’s testing your online properties to get
baseline usability metrics for improvements.
● Which are then tested and compared against the baseline over time in order
to understand how your site, app or other digital product is progressing.
● The outcome of benchmark studies also help identify problem areas for
improvement as well as building a vision for future releases.
Yes, absolutely!
MEASURE

5. Why Benchmark?
Quantify How Changes Have
Impacted Product Performance03
Get a holistic view of of the
customer journey02
Understand How to Improve Your
Products01
Engage Stakeholders &
Leadership06
Generate Insights worth
Investigating05
Get Competitive Intelligence04

6. Who should be benchmarking?
• UX Researchers
• Product Owners
• Product Marketing

7. What can you benchmark?
• Desktop site
• Mobile Site
• Mobile App
• Live sites or Prototypes
• Internal Sites or Intranets
Basically anywhere your users and customers are accessing your site or product
online can and should be benchmarked.

8. When should I be Benchmarking?
“We don’t know what our
research roadmap is for
next year”
“Our boss is obsessed with [x]
competitor”
“We want execs to know &
care about what we do”
“We’re about to start
journey mapping / new
product innovation”
TRIGGERS
Benchmarking can provide a useful context
before engaging in discovery. When mapping out research for the
year, benchmarking helps identify
areas of weakness against competitors
across a range of customer journeys.
If they mention some competitors, or the
HiPPO has a company they often bring up
as best practice - a benchmark can answer
questions on the best approach.
Benchmarking can be a useful exercise to
excite and engage stakeholders outside of
UX within an org.

9. ü Fast, convenient, and cost effective
ü Flexibility - professional services research or DIY
ü Study templates
ü Quick results
ü Recruit your target audience
ü Benchmarking all devices / digital properties
ü Get the complete picture with both types of data
ü Ease of including competitors
Why conduct UX benchmarking online?

10. 1. Create a Project Plan - Outline your goals, objectives, and timeline
2. Study Design - participant screener, tasks, follow-up questions, and make sure they tie back
to your project plan
3. Study Build in UZ – now you are ready to “build” it
4. QA your study build
5. Launch & Monitor
6. Analysis & Reporting
Next time is even easier, because you’ve already done the heavy lifting!
Just repeat steps 5 & 6!
How do I start? What does the process look like?
You’re ready to benchmark. Now what?

11. STAY FOCUSED
Don’t set too many goals
or ask questions that are
not directly tied back to
helping answer your top
questions.
ROADMAP
Collect the data you need
to assist in making sound
decisions.
KNOWN PROBLEMS
Address problems you or
your customers have
already identified.
YOUR HYPOTHESIS
Design your tasks and questions to get the
answer you need to confirm or refute your
hypothesis.
INDUSTRY BEST
PRACTICE
DEFINE
Be clear and specific when you
establish your benchmarking goals.
Step 1: Project Plan | Set Your Research Goals & Objectives

12. Step 2: Study Design | Design Once, Run Multiple Times
DESIGNING A BENCHMARKING STUDY
Tasks should be consistent across all
competitors. Review each site to
confirm all tasks can be completed
across competitors.
Use the same participant profile.
CONSISTENCY
Carefully consider the usability
and business metrics you want
to track.
KEY PERFORMANCE INDICATORS (KPIs
The UserZoom Platform
makes it quick and easy to
design and build your
benchmark, then repeat/ re-
run as needed in the future.
TEMPLATES
Include the primary tasks users
do on your site or app.
TASKS

13. Step 2: Study Design | Methodology
Our methodology is unique in that it includes both task-based large sample
usability studies (50n)
A smaller Qualitative think out loud usability study (10n).
Along with a reliable way to measure and track the user experience
with Consistent UX Metrics (KPIs) via our qxScore (Quality of Experience Score).
Initial questionnaire
Task 1
Final questionnaireTask 2
Task 3
Screener

14. Step 3: Study Build in UZ | Templates, screener library

15. Step 4: QA Your Build | Preview the Participant Perspective

16. Step 5: Launch & Monitor | Start reviewing your results

17. Step 6: Analysis & Reporting | Work smarter, not harder
UserZoom Results

18. Step 6: Analysis & Reporting | Work smarter, not harder
Insights: Quantitative

19. Step 6: Analysis & Reporting | Work smarter, not harder
Insights: Qualitative
Non-Success rate of all sites for this task at 62%
Of those who were not successful:
• 42% used incomplete address information in the locator tool
• 20% had chosen a location without a physical branch
Participant ID 130:
“…It says Union Trust. Is that the name of the bank? I’m confused.”
Sample
Participant
Video

20. qxScore Calculations

21. Understanding the qxScore
WHAT’S IT
BASED ON?
This score is based on a
number plotted on a
100 point scale that
references the UZ Index
HOW’S IT
CALCULATED?
This score is comprised
of behavioral task
success data combined
with attitudinal data
WHAT WE CALL THIS
“SINGLE SCORE”?
We call this single score the quality of
experience score, better known as the
qxScore

22. Industry Best Practice | UX KPIs
BEHAVIORAL (WHAT THEY DO) ATTITUDINAL (WHAT THEY SAY)
Ease of Use
Trust & Credibility
Appearance
Loyalty, NPS
Task Success
Task Time
Page Views, Clicks
Problems & Frustrations

23. What does it look like?

24. Behavior: What they do
Task Success %, per task. Overall Task Success across all tasks is weighed @ 50% of qxScore

25. Attitude: What they say, feel
Usability (Ease of Use)
Trust
Appearance
Loyalty, NPS
Top 2 Box

26. qxScore (Quality of Experience Score) Example
UserZoom calculated the qxScore based on both behavioral and attitudinal metrics.
We then plot this qxScore on UZ Index to determine the comparative score.
Very Poor
0 to 45
Poor
46 to 60
Average
61-75
Good
76-90
Great
91-
100Experience Score (qxScore) Range:
August
2018
73
BASELINE
January
2019
90
TESTED
AGAIN AFTER
REDESIGN

27. Quality of Experience
Score Range (UZIndex):
Very Poor
>45
Poor
45 - 60
Average
61 - 75
Good
76 - 90
Great
91 - 100
Task 1 50%
Task 2 83%
Task 3 66%
Task 4 83%
Task 5 66%
Task 6 66%
Task 7 66%
Task 8 100%
Task 9 100%
Usability (ease of use) 69%
Trust 92%
Appearance 69%
Loyalty 47%
qxScore:
Longitudinal Comparison: qx ScorecardBehavior:Whattheydid(Task
Success)
Attitudes:Whattheyfeel,say
August 2018 January 2019
Quality of Experience
Score Range (UZIndex):
Very Poor
>45
Poor
45 - 60
Average
61 - 75
Good
76 - 90
Great
91 - 100
Task 1 100%
Task 2 100%
Task 3 100%
Task 4 100%
Task 5 80%
Task 6 60%
Task 7 100%
Task 8 100%
Task 9 NA NA NA NA NA
Usability (ease of use) 92%
Trust 92%
Appearance 87%
Loyalty 79%
qxScore:
Behavior:Whattheydid(Task
Success)
Attitudes:Whattheyfeel,say
9073

28. Scorecard: Task-Level Effectiveness & Efficiency (Behavioral KPIs)
Participants task success rate improved on 6 out of 8 tasks. Users had a 100% success rate for 6 tasks.
Participants’ average time spent on task was reduced by 50% or more for most tasks; average # of clicks and page views also decreased for most tasks
Task 1 Task 2 Task 3 Task 4 Task 5 Task 6 Task 7 Task 8 Task 9
Aug-18 Jan-19 Aug-18 Jan-19 Aug-18 Jan-19 Aug-18 Jan-19 Aug-18 Jan-19 Aug-18 Jan-19 Aug-18 Jan-19 Aug-18 Jan-19 Aug-18 Jan-19
Task Success* 50% 100% 83.3% 100% 66.7% 100% 83.3% 100% 66.7% 80% 66.7% 60% 66.7% 100% 100% 100% 100% NA
Time on Task
(Avg.) 1:50 0:45 2:17 1:03 3:23 1:40 1:10 0:23 2:04 1:13 4:01 0:46 1:42 0:56 0:57 1:06 0:44 NA
# of Page Views
(Avg. pages) 3.2 1 3.2 1 5.8 1 2.8 2 5.5 1 5.2 3 3.6 2 3 3 1.4 NA
# of Clicks
(Avg. clicks) 6.5 4 9.7 3 19.3 11 3.3 1 8.8 4 14 5 8 3 5.8 5 3.0 NA

29. Scorecard: Task-Level (Attitudinal KPIs)
Ease of use improved on 5 tasks; 6 tasks now have a 100% ease of use score
The number of users who reported one or more problems or frustrations declined on 6 tasks
Task 1 Task 2 Task 3 Task 4 Task 5 Task 6 Task 7 Task 8 Task 9 Task 10
Aug-18 Jan-19 Aug-18 Jan-19 Aug-18 Jan-19 Aug-18 Jan-19 Aug-18 Jan-19 Aug-18 Jan-19 Aug-18 Jan-19 Aug-18 Jan-19 Aug-18 Jan-19 Aug-18 Jan-19
Ease of
completing task
(% in Top 2 box)
NA NA 84% 100% 50% 100% 67% 100% 83% 100% 67% 40% 50% 40% 100% 100% 83% 100% 100% NA
Experienced one
or more
problems or
frustrations (%)
NA NA 17% 0% 67% 14% 50% 28% 67% 20% 33% 40% 67% 40% 0% 43% 67% 28% 33% NA

30. Competitive Example: qxScore (Quality of Experience Score)
UserZoom calculated the qxScore for the 3 site based on both behavioral and attitudinal metrics.
We then plot the qxScore on UZ Index to determine the comparative score.
Very Poor
0 to 45
Poor
46 to 60
Average
61-75
Good
76-90
Great
91-
100
Experience Score (qxScore) Range:
76BRAND 1
73BRAND 2
68BRAND 3

31. Scorecard Calculator Available on Request

32. Scorecard Calculator Available on Request

33. UX Scorecard Program

34. Business Leaders believe:
‘What cannot be measured,
cannot be managed…’

35. UX Scorecard Dashboard: Experience Score = qxScore

36. BU1 Studies (Avg of 4 studies was at 69)

37. Study 2: UX Scorecard

38. Task 2: Find X in My Account (Issues with Screenshots)

39. 10 Benchmarking Tips

40. Get started measuring today!
Tip #1

41. Stay focused!
Be clear and specific when you define
your benchmarking goals.
Tip #2

42. Once is not enough!
Keep measuring to track how the
experience changes overtime.
Tip #3

43. Keep in step with your users
Stay abreast of changes in feature
prioritization, usage, needs, and
expectations.
Tip #4

44. Your research is only as good as your
participants.
Tip #5

45. Consistency is imperative to yield
reliable longitudinal and competitive data.
Tip #6

46. Don’t miss out on the many benefits of
expanding your testing universe.
Tip #7

47. Carefully consider which
competitors you use
Tip #8

48. Plan accordingly for
larger sample sizes
Tip #9

49. Work smarter, not harder!
Tip #10

50. Who is
the lucky
winner?

51. Competitive UX Benchmark
Study: Online Banking Industry
UserZoom’s latest Competitive UX Benchmark Report
offers key recommendations to teams who are focused
on optimizing and delivering great customer
experiences within the Banking industry.
Download: info.userzoom.com/competitive-ux-banking-benchmark-
report.html

52. Q&A

53. marketing@userzoom.com US Office: +1 866-599-1550
Would you like to schedule a live
benchmarking demo or meeting with Dana?
Let us show you how we’ve helped hundreds of companies unlock
user insights with UserZoom’s research platform
SCHEDULE A CALL: USERZOOM.COM/CONTACT-US