A successful web site must balance the needs of both the business and its users. To do so effectively requires a clear strategy. The core of any good strategy is measurement. Measurement plays a vital role in any effective strategy, and ultimately in developing a great user experience. But understanding what to measure, and how, is a challenge for many organizations. This presentation and discussion will focus on how measurement is key to any good user experience strategy. Key Points: · Why measurement is the core of any good strategy · How to determine and measure business goals · The components of user experience and how to measure them

2. May 25, 1961, address to congress
…this nation should
commit itself to achieving
the goal, before this
decade is out, of landing
a man on the moon and
returning him safely to
the Earth.

3. Putting a man on the
moon is a big deal.
You’re going to need a plan.

4. STRATEGY: A long term plan or method for
achieving a particular mission (or goal).

5. The Apollo mission had eight goals.
3,683 km/h
EARTH
7
4
8
1 2
3
343,304 km
5
6
1 - 1st Burn – Insertion into Earth Orbit
2 - 2nd Burn – Trans-lunar Injection
3 - Trans-lunar Trajectory (3rd and 4th burns to adjust course not required)
- Command Space Module extracts Lunar Module
4 - Command Space Module Brakes into Lunar Orbit
5 - Lunar Module Separates and Descends to the Moon
- Lunar Module Launches from the Moon
- Lunar Module Reunites with the Command Space Module
6 - Trans-Earth Trajectory
7 - Command Module Enters Earth Orbit
8 - Command Module Water Recovery in Pacific Ocean
MOON

6. A strategy breaks up
the mission into goals:
MISSION (Strategic Objective)
GOAL
GOAL
GOAL
Tactic
STRATEGY
How?
Tactic
Tactic
Mission, Objective, Goals
and Tactics are all “What”
EXECUTION
Actually
doing it.
STRATEGY

7. Goals should be:
S M A R T
SPECIFIC
MEASURA
BLE
ATTAINABL
E
RELEVANT
TIMERELATED

8. In order to stay on course.
You need to measure often
and make adjustments.
!

9. Measurement is the key
to an effective strategy.

10. Metrics are the standard of measurement
by which efficiency, performance,
progress, or quality of a plan, process, or
product can be assessed.

11. Analytics is the discovery and
communication of meaningful
patterns in data.

12. Facts and statistics
collected together for
reference or analysis.
Data

13. Photo: Graham Lavender: http://www.flickr.com/photos/guybrariang/
What is conveyed by a
particular arrangement
or sequence of things.
Information

14. Photo: Graham Lavender: http://www.flickr.com/photos/guybrariang/
Understanding,
based on information
and experience.
Knowledge

15. Photo: http://www.sunilshinde.com/2013/07/greek-odyssey-two-weeks-in-athens-delphi-peloponnese-and-santorini.html
The ability to discern or
judge what is true, right,
or lasting
Wisdom

16. SYSTEM
DATA
METRICS
INFORMATION
ANALYSIS
KNOWLEDGE
Knowledge allows you to predict the impact of changes to the system.
Wisdom allows you to know which changes are good for your mission.

17. But what
should you
measure?

18. We need to measure two things:
1
2
Are we achieving
our goals? (Strategic)
How is the system
performing? (Tactical)

19. Just because your system is
performing, does not mean,
you are meeting your goals.

20. STRATEGIC
METRICS
(Key Performance Indicators)
Measurements that directly relate to a goal/objective.
GOAL
KPI
METRICS
# of Sales
Business
Growth
Revenue
Change
Govt. Policy
Influence
with
Decision
Makers
$ of Sales
Policy Briefs
Downloaded
Petitions Signed

21. TACTICAL METRICS
(Performance Indicators)
Visits
Completion
Rates
Readability
Ranking
Pages per
visit
Task Time
Repeat Visits
Keywords
Time on Page
Directness
Social Shares
Cost per lead
Bounce Rates
Disaster
Rates
Sentiment
Conversions
and more…

22. Measure what is
truly important,
not just what is
easy to measure.

23. Measurement
informs design

24. The Apollo mission involved a lot of
measurement, and iteration.
1234
8 10
EARTH
MOON
567 9
AS201:
AS202:
AS203:
Apollo 1:
(Apollo 2):
(Apollo 3):
(Apollo 4):
Apollo 5:
Apollo 6:
Apollo 7:
Apollo 8:
Apollo 9:
Unmanned Tests of Rocket System
Unmanned Tests of Rocket System
Unmanned Tests of Rocket System
Disaster, 3 Astronauts killed in Command Space
Module (CSM) on launch pad during launch
test
Test of unmanned CSM
Test of unmanned CSM
Test of unmanned CSM
Unpiloted, first flight of Lunar module (LM)
Unpiloted, LM test
Manned Earth Orbit Test Flight
First Manned Lunar Orbit
Test of all components and EVA (Spacewalk)
11121314151617
Apollo 10:
Dress Rehearsal – stopped short of
landing on the moon
Apollo 11:
One small step for man. One
giant leap for mankind.
Apollo 12:
Apollo 13:
Apollo 14:
Apollo 15:
Apollo 16:
Apollo 17:
2nd Lunar Landing
Aborted due to fire during Lunar
Orbit
3rd Lunar Landing
4th Lunar Landing with Lunar
Roving Vehicle (LRV)
5th Lunar Landing
6th, and last, Lunar Landing

25. Measurement is why the Command
Space Module looked like this.

26. And not like this.

27. It’s why the
lunar lander,
looked like this.

28. And not like this.

29. USER EXPERIENCE:
The positive or negative
experience of using a system
or product.

30. Image Number: SI 99-15229
Credit: Photo by Mark Avino, National Air and Space Museum, Smithsonian Institution
The User Interface of the Command Space Module
was complicated. It required hundreds of hours of
training. But it worked.

31. The User Interface of the StarTrek LCARS system
looked good, and seems simple.
But it never had to work in real life.

32. User Experience consists of the User, the System,
and the Context in which the system is used.
SYSTEM
USER
(product or
Service)
CONTEXT

33. Aspects of a good web site or application
USER
EXPERIENCE
CREDIBILITY
USABILITY
LOYALTY
• Appropriate
• Intuitive
• Satisfying
• Professiona
l
• Accessible
• Competitive
• Efficient
• Relevant
• Functional
ACQUISITION
CONVERSION
ENGAGEMENT
Desired outcomes of a good web site or application

34. Elements of User Experience
CONTENT
NAVIGATION
STRUCTU
RE
MESSAGE
User
Experience
FUNCTIO
N
VISUAL
DESIGN
INTERACTIO
N
TECHNOLO
GY

35. How do you
measure User
Experience?

36. QUALITATIVE:
Non-numerical characteristics
“why and how”
QUANTITATIVE:
Numerical characteristics
“where, when and what”

37. Focus Sessions

38. Card Sorting

39. Tree Tests

40. Web Surveys

41. Readability Tests

42. Usability Tests

43. Accessibility Tests

44. A/B Tests

45. Load Tests

46. $$$
!#!
SALES
COMPLAINTS
???
INQUIRIES
Other Metrics

47. Web Traffic

48. When should you
measure?

49. START
MEASURING
ACTUAL TRAFFIC
DESIGN TO ENABLE
MEASUREMENT
AND MEET GOALS
DETERMINE
GOALS
IMPROVE
DISCOVER
DEFINE
DEFINE HOW
YOU WILL
MEASURE
SUCCESS
DESIGN
DEVELOP
IMPLEMENT
TOOLS FOR
MEASURING
DEPLOY
USE
KNOWLEDGE
GAINED FROM
MEASURING
TO IMPROVE

50. Surveys, Focus
Sessions, Interviews,
Historical traffic,
Business metrics, etc.
Usability Tests,
A/B Tests, Focus
Sessions, etc.
Load Tests, QC
Tests, UAT, more
usability tests…
IMPROVE
DISCOVER
DEFINE
DESIGN
Card Sorting, Treejack
tests, Usability Testing
Prototypes and
wireframes, etc.
DEVELOP
DEPLOY
Usability Testing,
Load Testing,
Accessibility Testing,
Code Testing, etc.
Web Traffic, A/B
Testing, Usability
Testing, Focus
Sessions, Surveys,
Business Metrics, etc.

51. When it comes to measuring,
3 things to keep in mind.

52. Trends > Snapshots

53. Escape Velocity:
11200. m/s
10 Seconds:
20 Seconds:
30 Seconds:
40 Seconds:
50 Seconds:
60 Seconds:
3 Minutes:
24.7 m/s
56.4 m/s
45.6 m/s
14.5 m/s
20.3 m/s
27.2 m/s
283.2 m/s
2 hrs, 44 Minutes:
10423. m/s
Snapshots don’t tell you the whole story

54. It is better to be
roughly right than
exactly wrong!
- Carveth Read

55. 50% of people like Hot Tea
50% of people like Iced Tea
Based on averages, 100% of people
would like luke warm tea

56. HiPPO = Highest Paid Person’s Opinion
The more data you have,
the weaker their opinion.

57. Measurement leads
to big goals

58. Top 10 Barriers to an effective measurement
strategy
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
Lack of budget/resources
Lack of strategy
Siloed organization
Lack of understanding
Too much data
Lack of senior management buy-in
Difficulty reconciling data
IT blockages
Lack of trust in analytics
Finding staff
http://econsultancy.com/uk/reports/online-measurement-and-strategy-report
45%
31%
29%
25%
18%
18%
17%
17%
16%
12%

59. http://xkcd.com/1074/