Check out this webinar!

1. BENCHMARKING MINI-SERIES PART #1
Proving Value & Quantifying
the Impact of UX Research
through Benchmarking
Presented by: Dana Bishop, Sr. Director of UX Research

2. Quick Housekeeping
• Control panel on the side of your screen if you have any
comments during the presentation
• Time at the end for Q&A
• Today’s webinar will be recorded for future viewing
• All attendees will receive a copy of the
slides/recording
• Continue the discussion using #uzwebinar
• Stay tuned until the end for the raffle winner announcement!
Let’s make sure you’re all set up for the webinar!

3. What We’re Covering in Part #1
User Experience benchmarking and it's value
What, when, and how often you should
benchmark
Competitive UX benchmarking
How to talk to your executives about the
importance of benchmarking
Dana Bishop
Sr. Director, UX Research
Dana has 25 years of industry experience,
with a focus on UX Benchmarking and
Syndicated Research.
Email dbishop@userzoom.com

4. Benchmarking is core to UserZoom
benchmark
/noun
a standard or point of reference against which
things may be compared or assessed.
LONGITUDINAL
BENCHMARKING
COMPETITIVE
TAKE
ACTION
COLLECT
DATA
ANALYSIS

5. Business Leaders believe:
‘What cannot be measured,
cannot be managed…’

6. Benchmarking creates a baseline for
understanding the current user
experience on your website or app
Don’t wait. Start Now!

7. ✔ The first run provides a baseline to measure against in the future.
✔ Helps deliver insights and establish goals to guide you where to focus efforts to improve and
optimize the user experience.
✔ However, once is not enough! Keep testing to track how the experience changes over time.
o At regular intervals (e.g. annually, bi-annually, or quarterly).
o Retest after rolling out major updates or redesigns.
✔ Compare how design iterations impact the user experience.
✔ Track and measure changes in KPIs and other metrics over time, and what’s driving those changes.
Longitudinal UX Benchmarking
The benefits of continual measurement and improvement of your digital properties

8. Longitudinal UX Benchmarking
Differences of approximately +/- 12% are significant at the 90% confidence level
Significant
improvement

9. When designing a benchmarking study,
invest the time up front to set it up
for success now and in the future

10. Setting Your Research Goals & Objectives
STAY FOCUSED
Don’t set too many goals
or ask questions that are
not directly tied back to
helping answer your top
questions.
ROADMAP
Collect the data you need
to assist in making sound
decisions.
KNOWN PROBLEMS
Address problems you or
your customers have
already identified.
YOUR HYPOTHESIS
Design your tasks and questions to get the
answer you need to confirm or refute your
hypothesis.
INDUSTRY BEST
PRACTICE
DEFINE
Be clear and specific when you
establish your benchmarking goals.

11. Design Once, Run Multiple Times
DESIGNING A BENCHMARKING STUDY
Tasks should be consistent across all
competitors. Review each site to
confirm all tasks can be completed
across competitors.
Use the same participant profile.
CONSISTENCY
Carefully consider the usability
and business metrics you want
.to track
KEY PERFORMANCE INDICATORS (KPIs
Makes it quick and easy to
design and build your
benchmark, then repeat/
re-run as needed in the
.future
USERZOOM PLATFORM
Include the primary tasks users
do on your site or app.
A Retail benchmark might include:
1. Browse for Product
2. Search for Product
3. Purchase Process
TASKS

12. Don’t miss out on the many
benefits of expanding your
testing universe to include your
competitors

13. Competitive UX Benchmark
✓ How your site performs relative to your competitors
✓ Interpret your usability standing, feature set and more within your industry
✓ Learning from their success and failures, what works or does not for your competitors
✓ Industry best-in-class examples to emulate
✓ Sometimes benchmark with other Industries

14. Example: Study Design & Methodology
Our methodology is unique in that it includes both task-based large sample usability studies (50n) and a
smaller Qualitative think out loud usability study (10n).
Along with a reliable way to measure and track the user experience with Consistent UX Metrics (KPIs) via
our qxScore (Quality of Experience Score).
Initial questionnaire
Task 1
Final questionnaireTask 2
Task 3
Randomized

15. Learning From Your Competitors’ Successes and Failures
Task Ease: Wells Fargo fared significantly worse for ease of finding a branch
* Selected 6-7 on a 7-point scale where 1 = Very difficult and 7 = Very easy

16. Top Problems & Frustrations
Find a Branch Task: Wells Fargo
n=50

17. Insights: Qualitative Study
Wells Fargo had the highest Non-Success rate of all sites for this task at 62%
Of those who were not successful:
• 42% used incomplete address information in the locator tool
• 20% had chosen a location without a physical branch
Participant ID 130:
“…It says Union Trust. Is that the name of the bank? I’m confused.”
Sample Video
(uses correct
address, but
confused by
results)

18. The greatest insights are derived from
collecting feedback from real users

19. Your Participants
✔ Recruit participants who match your demographics (or better yet personas, if you have them).
✔ Create a screener survey that will qualify participants based on your pre-defined criteria (e.g., age, previous
experience, or other demographics, traits, or attitudes).
✔ Use the same screener each time you run your benchmarking study. Consistency is imperative to yield reliable
longitudinal data.
Existing customers and/or intended target audience.

20. How Many Participants?
✔ Need samples large enough to make projections to the population.
✔ A “significant” difference is one that we are confident we didn’t find by chance alone.
✔ Confidence increases with sample size and the accuracy of our measures.
✔ For quantitative benchmarking studies, as a rule of thumb, sample sizes of 50n or more are
recommended.
✔ An additional consideration is the inclusion of multiple segments or groups. If you are going to run any
segmentation analysis (comparing two or more different groups) you will need to increase your sample
size accordingly.
Benchmarking studies typically require a larger sample size to get statistically significant and sound results.

21. qxScore Calculations

22. Industry Best Practice | UX KPIs
BEHAVIORAL (WHAT THEY DO) ATTITUDINAL (WHAT THEY SAY)
Ease of Use
Trust & Credibility
Appearance
Loyalty, NPS
Task Success
Task Time
Page Views, Clicks
Problems & Frustrations

23. Understanding the qxScore
WHAT’S IT
BASED ON?
This score is based on a
number plotted on a
100 point scale that
references the UZ Index
HOW’S IT
CALCULATED?
This score is comprised
of behavioral task
success data combined
with attitudinal data
WHAT WE CALL THIS
“SINGLE SCORE”?
We call this single score the quality of
experience score, better known as the
qxScore

24. What does it look like?

25. qxScore (Quality of Experience Score) Example
UserZoom calculated the qxScore based on both behavioral and attitudinal metrics.
We then plot this qxScore on UZ Index to determine the comparative score.
Very Poor
0 to 45
Poor
46 to 60
Average
61-75
Good
76-90
Great
91-100Experience Score (qxScore) Range:
August
2018
73
BASELINE
January
2019
90
TESTED
AGAIN AFTER
REDESIGN

26. Quality of Experience
Score Range (UZIndex):
Very Poor
>45
Poor
45 - 60
Average
61 - 75
Good
76 - 90
Great
91 - 100
Task 1 50%
Task 2 83%
Task 3 66%
Task 4 83%
Task 5 66%
Task 6 66%
Task 7 66%
Task 8 100%
Task 9 100%
Usability (ease of use) 69%
Trust 92%
Appearance 69%
Loyalty 47%
qxScore:
Longitudinal Comparison: qx ScorecardBehavior:Whattheydid(Task
Success)
Attitudes:Whattheyfeel,say
August 2018 January 2019
Quality of Experience
Score Range (UZIndex):
Very Poor
>45
Poor
45 - 60
Average
61 - 75
Good
76 - 90
Great
91 - 100
Task 1 100%
Task 2 100%
Task 3 100%
Task 4 100%
Task 5 80%
Task 6 60%
Task 7 100%
Task 8 100%
Task 9 NA NA NA NA NA
Usability (ease of use) 92%
Trust 92%
Appearance 87%
Loyalty 79%
qxScore:
Behavior:Whattheydid(Task
Success)
Attitudes:Whattheyfeel,say
9073

27. Scorecard: Task-Level Effectiveness & Efficiency (Behavioral KPIs)
Participants task success rate improved on 6 out of 8 tasks. Users had a 100% success rate for 6 tasks.
Participants’ average time spent on task was reduced by 50% or more for most tasks; average # of clicks and page views also decreased for most tasks
Task 1 Task 2 Task 3 Task 4 Task 5 Task 6 Task 7 Task 8 Task 9
Aug-18 Jan-19 Aug-18 Jan-19 Aug-18 Jan-19 Aug-18 Jan-19 Aug-18 Jan-19 Aug-18 Jan-19 Aug-18 Jan-19 Aug-18 Jan-19 Aug-18 Jan-19
Task Success* 50% 100% 83.3% 100% 66.7% 100% 83.3% 100% 66.7% 80% 66.7% 60% 66.7% 100% 100% 100% 100% NA
Time on Task
(Avg.)
1:50 0:45 2:17 1:03 3:23 1:40 1:10 0:23 2:04 1:13 4:01 0:46 1:42 0:56 0:57 1:06 0:44 NA
# of Page Views
(Avg. pages)
3.2 1 3.2 1 5.8 1 2.8 2 5.5 1 5.2 3 3.6 2 3 3 1.4 NA
# of Clicks
(Avg. clicks)
6.5 4 9.7 3 19.3 11 3.3 1 8.8 4 14 5 8 3 5.8 5 3.0 NA

28. Scorecard: Task-Level (Attitudinal KPIs)
Ease of use improved on 5 tasks; 6 tasks now have a 100% ease of use score
The number of users who reported one or more problems or frustrations declined on 6 tasks
Task 1 Task 2 Task 3 Task 4 Task 5 Task 6 Task 7 Task 8 Task 9 Task 10
Aug-18 Jan-19 Aug-18 Jan-19 Aug-18 Jan-19 Aug-18 Jan-19 Aug-18 Jan-19 Aug-18 Jan-19 Aug-18 Jan-19 Aug-18 Jan-19 Aug-18 Jan-19 Aug-18 Jan-19
Ease of
completing task
(% in Top 2 box)
NA NA 84% 100% 50% 100% 67% 100% 83% 100% 67% 40% 50% 40% 100% 100% 83% 100% 100% NA
Experienced one
or more
problems or
frustrations (%)
NA NA 17% 0% 67% 14% 50% 28% 67% 20% 33% 40% 67% 40% 0% 43% 67% 28% 33% NA

29. Competitive Example: qxScore (Quality of Experience Score)
UserZoom calculated the qxScore for the 3 site based on both behavioral and attitudinal metrics.
We then plot the qxScore on UZ Index to determine the comparative score.
Very Poor
0 to 45
Poor
46 to 60
Average
61-75
Good
76-90
Great
91-100Experience Score (qxScore) Range:
76BRAND 1
73BRAND 2
68BRAND 3

30. UX Scorecard Dashboard: Experience Score = qxScore

31. BU1 Studies (Avg of 4 studies was at 69)

32. Study 2: UX Scorecard

33. How to talk to
your executives
about the
importance of
benchmarking
21st Century
KPI Model
Connecting
qxScore
+ Company KPIs

34. The ROI of Benchmarking

35. LOTS
OF
KPIs!

36. Level 1
Level 3
All-Enterprises - GAAP Metrics:
Revenue | Margin | Cash Flow | Quarterly Results Announcements
Product-level Metrics:
Acquisition Funnel | Customer feedback | A/B Tests| Intercept/VOC
Surveys| Twitter, FB, etc. Social metrics
Level 4
UX Metrics : Behavioral & Attitudinal
UX Metrics (task-completion, time-on-task, product NPS, SUPR-Q, etc.) |
QA data (Bug Count, back-end efficiency gains/loss, implementation
hurdles, etc.)
YOUR Enterprise’s key-metrics:
Attitudinal | Behavioral
Daily/Monthly active users | Brand-level NPS/CSAT
Transaction Volume | Churn
Level 2 The 21st
Century
Metrics
Model

37. Level 1
Level 3
All-Enterprises - GAAP Metrics:
Revenue | Margin | Cash Flow | Quarterly Results Announcements
Product-level Metrics:
Acquisition Funnel | Customer feedback | A/B Tests| Intercept/VOC
Surveys| Twitter, FB, etc. Social metrics
Level 4
UX Metrics : Behavioral & Attitudinal
UX Metrics (task-completion, time-on-task, product NPS, SUPR-Q, etc.) |
QA data (Bug Count, back-end efficiency gains/loss, implementation
hurdles, etc.)
YOUR Enterprise’s key-metrics:
Attitudinal | Behavioral
Daily/Monthly active users | Brand-level NPS/CSAT
Transaction Volume | Churn
Level 2 The 21st
Century
Metrics
Model

38. Level 1
Level 3
All-Enterprises - GAAP Metrics:
Revenue | Margin | Cash Flow | Quarterly Results Announcements
Product-level Metrics:
Acquisition Funnel | Customer feedback | A/B Tests| Intercept/VOC
Surveys| Twitter, FB, etc. Social metrics
Level 4
UX Metrics : Behavioral & Attitudinal
UX Metrics (task-completion, time-on-task, product NPS, SUPR-Q, etc.) |
QA data (Bug Count, back-end efficiency gains/loss, implementation
hurdles, etc.)
YOUR Enterprise’s key-metrics:
Attitudinal | Behavioral
Daily/Monthly active users | Brand-level NPS/CSAT
Transaction Volume | Churn
Level 2 The 21st
Century
Metrics
Model

39. Level 1
Level 3
All-Enterprises - GAAP Metrics:
Revenue | Margin | Cash Flow | Quarterly Results Announcements
Product-level Metrics:
Acquisition Funnel | Customer feedback | A/B Tests| Intercept/VOC
Surveys| Twitter, FB, etc. Social metrics
Level 4
UX Metrics : Behavioral & Attitudinal
UX Metrics (task-completion, time-on-task, product NPS, SUPR-Q, etc.) |
QA data (Bug Count, back-end efficiency gains/loss, implementation
hurdles, etc.)
YOUR Enterprise’s key-metrics:
Attitudinal | Behavioral
Daily/Monthly active users | Brand-level NPS/CSAT
Transaction Volume | Churn
Level 2 The 21st
Century
Metrics
Model

40. Level 1
Level 3
All-Enterprises - GAAP Metrics:
Revenue | Margin | Cash Flow | Quarterly Results Announcements
Product-level Metrics:
Acquisition Funnel | Customer feedback | A/B Tests| Intercept/VOC
Surveys| Twitter, FB, etc. Social metrics
Level 4
UX Metrics : Behavioral & Attitudinal
UX Metrics (task-completion, time-on-task, product NPS, SUPR-Q, etc.) |
QA data (Bug Count, back-end efficiency gains/loss, implementation
hurdles, etc.)
YOUR Enterprise’s key-metrics:
Attitudinal | Behavioral
Daily/Monthly active users | Brand-level NPS/CSAT
Transaction Volume | Churn
Level 2 The 21st
Century
Metrics
Model

41. Benchmarking Workshop
Let’s
determine
your drivers
(hypothesis)

42. Who is
the lucky
winner?

43. Join us next week! April 2nd
Planning & conducting
UX benchmarking
Collecting and tracking
data with scorecards
UserZoom’s qxScore
& Scorecard program
Get 10 expert
benchmarking tips
PLUS...another shot to qualify for the raffle!

44. Competitive UX Benchmark
Study: Online Banking Industry
UserZoom’s latest Competitive UX Benchmark Report
offers key recommendations to teams who are focused
on optimizing and delivering great customer experiences
within the Banking industry.
Download: info.userzoom.com/competitive-ux-banking-benchmark-report.html

45. Q&A

46. marketing@userzoom.com US Office: +1 866-599-1550
How would you stack up over time
or against your competition?
Let us show you how we’ve helped hundreds of companies unlock
competitive user insights with UserZoom’s research platform
SCHEDULE A CALL: USERZOOM.COM/CONTACT-US