Tim Wilson's presentation at eMetrics San Francisco in April 2013 covering an approach for analysts to establish and manage a well-defined process for hypothesis discovery, prioritization, and testing to help pivot companies to be data-driven -- to orient towards a culture of validated learning.

1. Web Optimization
Process Management
Tim Wilson
VP of Measurement & Analytics, Clearhead
/emetrics eMetrics, April 2013
@tgwilson

2. /emetrics @tgwilson

3. The
analyst
as an
agent of
change
/emetrics @tgwilson

4. Our job: reorient
the business
towards validated
learning.
/emetrics @tgwilson

5. Everyone wants to
be data-driven
these days.
/emetrics @tgwilson

6. Being Data Driven
People Process
Technology
/emetrics @tgwilson

7. Being Data Driven
The Business’ perspective
People Process
Technology
/emetrics @tgwilson

8. Being Data Driven
The Analyst’s perspective
People Process
Technology
/emetrics @tgwilson

9. Being Data Driven
My perspective
People Process
Technology
/emetrics @tgwilson

10. Let’s talk about a
process revolution.
/emetrics @tgwilson

11. Every revolution
needs a mobilizing
catch-phrase.
/emetrics @tgwilson

12. A.D.A.P.T.
to
Act & Learn
/emetrics @tgwilson

13. Align on goals and KPIs
Discover hypotheses across the company
Assess the hypotheses and your approach
Prioritize the hypotheses
Test the highest priority hypotheses
Act on the results
Learn for the future
/emetrics @tgwilson

14. Align on goals and KPIs
Discover hypotheses across the company
Assess the hypotheses and your approach
Prioritize the hypotheses
Test the highest priority hypotheses
Act on the results
Learn for the future
/emetrics @tgwilson

15. Alignment means
clearly established
goals and KPIs.
/emetrics @tgwilson

16. /emetrics @tgwilson

17. /emetrics @tgwilson

18. Magic questions.
1. What are we trying to achieve?
2. How do we know if we’ve done that?
/emetrics @tgwilson

19. A Process: Goals and KPIs
Two Magic − What are we trying to achieve?
Questions − How will we know if we’ve done that?
Establish − Open-ended questions
Appropriate − “What?” and “How?” (not “Why?”)
Measures − The power of restatement
− Historical data
Establish
− Back-of-the-napkin creativity
Targets
− Bracketing
/emetrics @tgwilson

20. Online Revenue Engagement Reach
Drive online revenue through the presence of xxxx Increase overall site engagement Increase the reach of [site].com
Conversion Rate Lift +14.3% New Profiles Created 2,707 SEO Visits to xxxx 116
Visits that interacted with xxxx 7.08% Total Profiles created since launch 3,391
20
250
Visits that did not interact with xxxx 6.19% 200 15
15.0% 150
10
10.0% 100
5.0% 50 5
0.0% 0
3/10 3/26 3/10 3/26 0
3/10 3/26
Overall conversion Projected rate Sitewide
rate for the site without xxxx % Lift % Interacting w/ xxxx 13.8%
+3.19% +3.14% +1.6% Visits that interacted 36,605 Social Shares of xxxx 251
Visits that did not interact 229,125 50
Avg. Order Value Lift +2.7% 20.0% 40
Visits that interacted with xxxx $23.37 15.0%
30
20
Visits that did not interact with xxxx $22.76 10.0%
10
$40 5.0%
0
0.0% 3/10 3/26
$20
3/10 3/26
$0
3/10 3/26
Time on Site Lift +77.6% Social Referral Visits 46
Overall AOV Projected AOV Sitewide
for the site without xxxx % Lift Visits that interacted with xxxx 24:49 8
$23.04 $22.96 +0.3% Visits that did not interact with xxxx 13:59
6
30:00 4
Total Revenue Lift $8,976 25:00
20:00 2
Conversion rate lift only $7,389 15:00
0
AOV lift only $1,389 10:00 3/10 3/26
05:00
AOV lift on incremental orders $199 00:00
3/10 3/26 Refer to the glossary for a detailed definition for each metric.
Annualized Lift (ignoring seasonality) $192,727
/emetrics @tgwilson

21. Align on goals and KPIs
Discover hypotheses across the company
Assess the hypotheses and your approach
Prioritize the hypotheses
Test the highest priority hypotheses
Act on the results
Learn for the future
/emetrics @tgwilson

22. An analysis is
nothing more than
testing a hypothesis.
/emetrics @tgwilson

23. Where do
hypotheses come
from?
/emetrics @tgwilson

24. Everyone Else
at the Company
Our
Stakeholders
Analysts
/emetrics @tgwilson

25. Everyone in the
business has
hypotheses!
/emetrics @tgwilson

26. The trick is getting
them articulated
clearly.
/emetrics @tgwilson

27. What we say to marketers
“What is your
hypothesis?”
What they hear
“I’m dragging you
back to the 8th grade
science class you
hated.”
/emetrics @tgwilson

28. I believe that…
[some idea]
If I am right, then…
[some action]
/emetrics @tgwilson

29. Each idea can be
added to your
hypothesis library.
­ Google Spreadsheet
­ Excel
­ uservoice.com
­ Sharepoint
­ Lotus Notes
­ Jira
/emetrics @tgwilson

30. Align on goals and KPIs
Discover hypotheses across the company
Assess the hypotheses and your approach
Prioritize the hypotheses
Test the highest priority hypotheses
Act on the results
Learn for the future
/emetrics @tgwilson

31. To assess the effort
required, you have to
determine the
approach.
/emetrics @tgwilson

32. Flickr / justinbaeder
/emetrics @tgwilson

33. The analyst must
be aware of the full
tool belt.
/emetrics @tgwilson

34. I believe that…
[some idea]
If I am right, then…
[some action]
To test this, I will…
[tool/technology/approach]
/emetrics @tgwilson

35. What’s in your
company’s toolbelt?
Web Analytics Email Surveys
Voice of the Customer Social Media Analytics
A/B (and MV) Testing Focus Groups
Usability Testing Secondary Research
/emetrics @tgwilson

36. Align on goals and KPIs
Discover hypotheses across the company
Assess the hypotheses and your approach
Prioritize the hypotheses
Test the highest priority hypotheses
Act on the results
Learn for the future
/emetrics @tgwilson

37. It’s not a simple formula.
/emetrics @tgwilson

38. But a formula can
guide the
discussion.
/emetrics @tgwilson

39. Likelihood belief is right H M L
Likelihood change would occur H M L
Likely scale of impact H M L
Alignment w/ business goals H M L
Effort to test (approach) H M L
/emetrics @tgwilson

40. Align on goals and KPIs
Discover hypotheses across the company
Assess the hypotheses and your approach
Prioritize the hypotheses
Test the highest priority hypotheses
Act on the results
Learn for the future
/emetrics @tgwilson

41. /emetrics @tgwilson

42. Align on goals and KPIs
Discover hypotheses across the company
Assess the hypotheses and your approach
Prioritize the hypotheses
Test the highest priority hypotheses
Act on the results
Learn for the future
/emetrics @tgwilson

43. Driving action
requires effective
communication of
the results.
/emetrics @tgwilson

44. The data
does not
speak for itself.
/emetrics @tgwilson

45. /emetrics @tgwilson

46. Who needs to do
what?
/emetrics @tgwilson

47. Lead with the
business problem
statement.
/emetrics @tgwilson

48. Follow with the
answer and the
action.
/emetrics @tgwilson

49. Do not show your
work.
/emetrics @tgwilson

50. /emetrics @tgwilson

51. And then track to
action.
/emetrics @tgwilson

52. We tested…
[the idea]
As a result, we…
[took the action]
Resulting in…
[quantified result]
/emetrics @tgwilson

53. Align on goals and KPIs
Discover hypotheses across the company
Assess the hypotheses and your approach
Prioritize the hypotheses
Test the highest priority hypotheses
Act on the results
Learn for the future
/emetrics @tgwilson

54. Learning means
looking beyond the
near-term action.
/emetrics @tgwilson

55. Hypothesis tested Action taken Deeper learning
Hypothesis tested No action taken
Hypothesis tested Action taken
Hypothesis tested Action taken Deeper learning
Hypothesis tested No action taken Deeper learning
Hypothesis tested Action taken
Hypothesis tested Action taken
Hypothesis tested Action taken
/emetrics @tgwilson

56. Reorienting the
business towards
validated learning
/emetrics @tgwilson

57. Align on goals and KPIs
Discover hypotheses across the company
Assess the hypotheses and your approach
Prioritize the hypotheses
Test the highest priority hypotheses
Act on the results
Learn for the future
/emetrics @tgwilson

58. clearhead.me/emetrics
­ This presentation
­ Guide: getting to great KPIs
­ Hypothesis library template
­ Approach options self-audit template
­ Recommended reading
/emetrics @tgwilson

59. nthanksN
/emetrics @tgwilson