The document discusses the problems that can arise from focusing on a single metric, referred to as "the one metric that matters". It notes that companies like Etsy and Yahoo have run into issues by focusing only on gross merchandise sales or revenue per user, respectively. The document recommends using a framework with three types of metrics: overall evaluation criteria to measure long-term business value, feature metrics to understand specific product changes, and guardrail metrics to avoid degradation. It provides examples of metrics that could be used for companies like Uber/Lyft and Twitter.

1. The lure of “the one metric that matters”
Adil Aijaz (@adilaijaz)
Co-founder & CEO
Split Software

2. Lure of One Metric That Matters
Software Engineer Sr Staff Software
Engineer
Principal Member
Technical Staff
Co-founder & CEO

3. Lure of One Metric That Matters
Split is a product decisions platform
that lets engineering & product teams
rapidly - and safely - deliver valuable software

4. Lure of One Metric That Matters
We power your product decisions

5. Lure of One Metric That Matters
… by providing a unified platform for
feature flags and experimentation

6. Lure of One Metric That Matters
Output
factory
Outcome
factory
We change the way engineering teams operate
#allaboutmeasurement

7. Lure of One Metric That Matters
Measurement is all about metrics

8. Lure of One Metric That Matters
It’s a topic that eludes many

9. Lure of One Metric That Matters
Common problems with metrics

10. Lure of One Metric That Matters
#1 Opinion driven cultures - zero metrics

11. Lure of One Metric That Matters
#2 The lure of one metric that matters

12. Lure of One Metric That Matters
#3 The tyranny of metrics - fixation with
metrics

13. Lure of One Metric That Matters
The lure of one metric that matters

14. How Etsy’s New CEO Threw
Out All of the Company’s
Key Metrics, Except One
“I observed that GMS [gross merchandise sales]
is the one metric that matters," says Silverman.
Before the company was measuring 15 or so
metrics, which justified a lot of projects that
may or may not have been worthwhile. In fact,
he asked his new team to "stack rank" the 800
product initiatives in Etsy's pipeline, all with an
eye to what could goose GMS quickly.”
Fortune.com March 5th, 2018
Lure of One Metric That Matters
Sometimes, it is explicit

15. Yahoo! Mail’s display ad
problem due to one metric
that matters:
revenue / user
Lure of One Metric That Matters
Sometimes, you just
know it

16. Sometimes, it is funny

17. Sometimes, it threatens
our democracy

18. Lure of One Metric That Matters
Why is the focus on a single
metric bad?

19. Oversimplification
can lead us
to failure

20. … and to negative externalities

21. Lure of One Metric That Matters
But why are we lured by the search
for one metric that matters?

22. Competition brings out our hunting

23. Lure of One Metric That Matters
From the top of management stack, the
entire business looks like a set of metrics

24. Lure of One Metric That Matters
Solution

25. Lure of One Metric That Matters
Apply a framework for metrics

26. Types of Metrics
Lure of One Metric That Matters

27. Lure of One Metric That Matters
● Overall Evaluation
Criteria
● Feature Metrics
● Guardrail Metrics
Three types of
metrics*
*based on work done by Microsoft Analysis & Experimentation team

28. Overall Evaluation Criteria (“OEC”)
Lure of One Metric That Matters

29. The (OEC) is the metric your product
development is designed to improve.
What is an OEC?
Lure of One Metric That Matters
measure of long-term
business value or
satisfaction for user,
rather than granular
feature specific metric.

30. ● Sensitivity - The metric should change due
to small changes in user satisfaction or
business value so that the impact of
product development is clear within a
reasonable time period.
● Directionality - If the business value
increases, the metric should move
consistently in one direction. If it
decreases, the metric should move in the
opposite direction.
● Understandability - OEC ties product
development to business value, so it
should be easily understood by business
executives.
Lure of One Metric That Matters
Properties of a
Good OEC

31. Lure of One Metric That Matters
What’s a good OEC for Uber / Lyft?

32. Example for riders - Uber / Lyft
Lure of One Metric That Matters
Revenue / User Rides / User
Distance / User Sessions / User

33. Example for riders - Uber / Lyft
Lure of One Metric That Matters
Revenue / User Rides / User
Distance / User Sessions / User

34. Lure of One Metric That Matters
What’s a good OEC for Twitter?

35. Example for consumers - Twitter
Lure of One Metric That Matters
Tweets / User Time Spent / User
Sessions / User Interactions /
User

36. Example for consumers - Twitter
Lure of One Metric That Matters
Tweets / User Time Spent / User
Sessions / User Interactions /
User

37. ● Click through rate on a button is not a
good OEC.
● One way to recognize whether you have a
good OEC is to intentionally experiment
with a bad feature you know your users
would not like.
● For advanced companies, OEC can be the
output of a model that takes multiple
input signals.
● It’s best to normalize the OEC “/customer”.
Lure of One Metric That Matters
Some rules of
thumb

38. Feature Metric
Lure of One Metric That Matters

39. What is a feature
metric?
Lure of One Metric That Matters
Like the OEC, but of
localized interest to a
specific team.
Often explains why
the OEC moved a
certain way

40. ● Click through rate on a button
can be a good feature metric.
● Should be directional and
sensitive
● Understandability is not critical.
● Tie back to business value is
not critical
Lure of One Metric That Matters
Some rules of
thumb

41. Guardrail Metrics
Lure of One Metric That Matters

42. What is a
guardrail metric?
Lure of One Metric That Matters
Metric that should not
degrade in the pursuit
of OEC

43. Examples
Lure of One Metric That Matters
● Page load time
● Ride cancellations / user
● Revenue / user
● Content quality / content
● Unsubscribe rates

44. ● One team’s OEC can be another
team’s guardrail
● Engineering system performance
is always a good guardrail
● Should be directional and
sensitive
● Understandability is not critical.
● Tie back to business value is not
critical
Lure of One Metric That Matters
Some rules of
thumb

45. Metric Framework
Lure of One Metric That Matters

46. Lure of One Metric That Matters
A framework gives a vocabulary to your team
to talk about metrics

47. Lure of One Metric That Matters
HEART Metrics*
*Based on work done by Google’s UX team

48. ● Metrics that measure user attitude
● Often subjective
● Usually measured via surveys
● Examples: NPS score, ease of use,
satisfaction scores.
Lure of One Metric That Matters
Happiness

49. ● Metrics that measure user involvement
● Frequency, intensity, or depth of
interaction over some period of time
● Usually measured by Product Analytics
tools
● Examples: daily active users, sessions /
user, number of shares / user / week
Lure of One Metric That Matters
Engagement

50. ● Metrics that measure how new users are
adopting a feature or product
● Often tied to the OEC
● Examples: # of rides taken by a Lyft rider
in their first month.
Lure of One Metric That Matters
Adoption

51. ● Metrics that measure whether users who
adopted a feature in one time period,
continue using it in subsequent
time period
● Often tied to the OEC
● Examples: # of rides taken by a Lyft rider
in their second, third, and fourth month.
Lure of One Metric That Matters
Retention

52. ● Metrics that measure user efficiency,
effectiveness, and error rate in
completing a critical task
● Often tied to the OEC
● Examples: wait time / ride measures
efficiency, , # of rides cancelled by a rider
measures effectiveness, & app crashes
measure error rate for Lyft
Lure of One Metric That Matters
Task Success

53. Thank You
Lure of One Metric That Matters