This document summarizes key points from a lecture on analytics. It discusses why analytics are important to quantify success and make scientific decisions. Good analytics should be simple, relevant, unambiguous and actionable. Key metrics include acquisition, activation, retention, referral and revenue (AARRR). The document also covers website metrics like traffic sources, popular pages and conversions. It discusses optimization testing and competitive intelligence gathering.
2. Lecture 8
• Introduction
• Website metrics
• Optimization
• Competitive intelligence
• Surveys
• Tools and books
From Code to Product Lecture 8 — Analytics— Slide 2 gidgreen.com/course
3. Why analytics?
• Quantify success/failure
– For yourselves
– For investors
– Against competition
• Scientific decisions
– No blind faith
– Fewer arguments
– Avoid HiPPO = highest paid person’s opinion
From Code to Product Lecture 8 — Analytics— Slide 3 gidgreen.com/course
4. Good analytics
• Simple
• Few in number
• Relevant
• Unambiguous
• Actionable
• Instant (or nearly)
• Repeatable
From Code to Product Lecture 8 — Analytics— Slide 4 gidgreen.com/course
5. AARRR — Metrics for pirates
Acquisition Site visit or app download
Activation Registration or usage
Dave McClure, 500 Startups
Retention Repeat usage
Referral Brings other people
Revenue Generate cash
From Code to Product Lecture 8 — Analytics— Slide 5 gidgreen.com/course
6. Some quotes
“What gets measured, gets managed.”
— Peter Drucker
“The only metrics that entrepreneurs
should invest energy in collecting are
those that help them make decisions.”
— Eric Ries, The Lean Startup
From Code to Product Lecture 8 — Analytics— Slide 6 gidgreen.com/course
7. In-app analytics
• Home rolled or third party
• Store usage information locally
– ‘Call home’ when online
• Privacy concerns
– Confirmation dialog?
• Complete access to device
– But you will be caught!
• Problem: slow iteration
From Code to Product Lecture 8 — Analytics— Slide 7 gidgreen.com/course
8. Web analytics
• All activity visible to site
– Users don’t expect privacy
• Web servers log requests
– Also: Javascript solutions
• Page view centric
– Other events require integration
– Coffee break?
– Events not sessions
From Code to Product Lecture 8 — Analytics— Slide 8 gidgreen.com/course
9. A web server log line
www.websudoku.com
24.186.55.113
[06/May/2012:08:13:02 -0400]
"GET / HTTP/1.1”
200
1045
"http://www.google.com/search?q=sudoku”
"Mozilla/5.0 (iPhone; CPU iPhone OS 5_1
like Mac OS X) AppleWebKit/534.46 (KHTML,
like Gecko) Mobile/9B179 Safari/7534.48.3"
From Code to Product Lecture 8 — Analytics— Slide 9 gidgreen.com/course
10. Javascript tracking code
<script type="text/javascript”>
var _gaq = _gaq || [];
_gaq.push(['_setAccount', 'UA-1165533-3']);
_gaq.push(['_trackPageview']);
(function() {
var ga = document.createElement('script'); ga.type =
'text/javascript'; ga.async = true;
ga.src = ('https:' == document.location.protocol ?
'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
var s = document.getElementsByTagName('script')[0];
s.parentNode.insertBefore(ga, s);
})();
</script>
From Code to Product Lecture 8 — Analytics— Slide 10 gidgreen.com/course
11. Metrics alternatives
Server logs Javascript Home-made
Integration None Via HTML Server code
Download + Web-based
Convenience Up to you
analyze access
Delay None Up to 24 hours Up to you
Reporting Varies Advanced Up to you
Other events Hard Via API Easy
Data leakage None Total! None
From Code to Product Lecture 4 — UI Design— Slide 11 gidgreen.com/course
12. Track web users by…
• IP address
– Given for every web request
– Good for geography
– But: proxies, classrooms, router resets
• Cookies
– Track user browser over long term
– But: clearing, multi-browsing, first request
– Customization of web server
From Code to Product Lecture 8 — Analytics— Slide 12 gidgreen.com/course
13. Track web users by…
• Log in
– Reliable for registered users
– But: anonymous users, multiple accounts
– Requires custom logging tools
• Solution: combine!
– Intelligently tie IPs, cookies and accounts
– Example: user registration
• Data always incomplete
From Code to Product Lecture 8 — Analytics— Slide 13 gidgreen.com/course
14. Lecture 8
• Introduction
• Website metrics
• Optimization
• Competitive intelligence
• Surveys
• Tools and books
From Code to Product Lecture 8 — Analytics— Slide 14 gidgreen.com/course
16. Immediate questions
• When does one visit end?
– GA: 30 minutes without activity
• What makes a visitor unique?
– GA: Tracking cookie
• How is duration calculated?
– GA: Time between first and last pages
• What makes a visitor new?
– GA: Never visited your site before
From Code to Product Lecture 8 — Analytics— Slide 16 gidgreen.com/course
17. Geography
From Code to Product Lecture 6 — BM — Advertising— Slide 17 gidgreen.com/course
20. Sources of traffic
• Type-in (no referrer)
– Includes browser bookmarks
• Search engines
– Navigational search = type-in
• Referrals
– Website links or social media
• Paid advertising
• Email campaigns
From Code to Product Lecture 8 — Analytics— Slide 20 gidgreen.com/course
21. The multitouch problem
• There’s history before the referrer
– Who deserves the credit, e.g. affiliates
• So who gets the credit?
– Last click (standard)
– First click (unrealistic)
– Even split
– Split weighted to last
• Real question: what gives best ROI?
From Code to Product Lecture 8 — Analytics— Slide 21 gidgreen.com/course
22. Search engine queries
Also: internal site search
From Code to Product Lecture 8 — Analytics— Slide 22 gidgreen.com/course
26. Conversion funnel
Source: www.searchenginejournal.com
From Code to Product Lecture 8 — Analytics— Slide 26 gidgreen.com/course
27. Sampling methods
• Popular site => lots of data
– Burden to collect, slow to analyze
• Don’t record all events
– Choose important pages
– Random subset of visitors
– Random subset of pageviews
• Sub-sample when analyzing
– By page or visitor
From Code to Product Lecture 8 — Analytics— Slide 27 gidgreen.com/course
28. Staleness due to changes in…
• Content
• User familiarity
• Search engine rankings
• Market
• Technology
• Cookies
– Long-term analysis would be great!
From Code to Product Lecture 8 — Analytics— Slide 28 gidgreen.com/course
29. Lecture 8
• Introduction
• Website metrics
• Optimization
• Competitive intelligence
• Surveys
• Tools and books
From Code to Product Lecture 8 — Analytics— Slide 29 gidgreen.com/course
30. Optimization
• You don’t know how users behave
– Example: show price early on?
• Small changes => big results
– But which small changes?
• Use a scientific methodology
– Easy to set up
– Easy to get report
– Statistical significance
From Code to Product Lecture 8 — Analytics— Slide 30 gidgreen.com/course
31. Wording example
you_should_follow_me_on_twitter.html
Source: http://www.dustincurtis.com/
From Code to Product Lecture 8 — Analytics— Slide 31 gidgreen.com/course
32. A/B testing
• Two parallel variations
– Current vs challenger
• Assign randomly and evenly
– What about previous visitors?
– Repeat requests within a session?
• Set test length in advance
– Length of time or number of visits
• Chi-squared (or similar) test
From Code to Product Lecture 8 — Analytics— Slide 32 gidgreen.com/course
33. Contingency table
Product Not
purchased purchased
9 575
13 563
From Code to Product Lecture 8 — Analytics— Slide 33 gidgreen.com/course
34. Multivariate testing
Source: http://www.getelastic.com/testing-
part-1/
From Code to Product Lecture 8 — Analytics— Slide 34 gidgreen.com/course
35. Multivariate testing
• Best to use third-party tool
• Full factorial vs partial factorial
– Certainty vs efficiency
From Code to Product Lecture 8 — Analytics— Slide 35 gidgreen.com/course
36. Optimization pitfalls
• Preconception driven
– Too many similar tests
– Checking before it’s done
• Wrong goal
– e.g. started vs completed purchases
• Unfair test
– Different time periods
– New vs returning users
From Code to Product Lecture 8 — Analytics— Slide 36 gidgreen.com/course
37. More complex tests
• Non-binary outcomes
– Size of purchase, length of stay
• Cohort / longitudinal tests
• Whole-site multivariate testing
• Pricing
– How to prevent a riot?
• Spot diminishing returns
– Focus on registration, payment, etc…
From Code to Product Lecture 8 — Analytics— Slide 37 gidgreen.com/course
38. Lecture 8
• Introduction
• Website metrics
• Optimization
• Competitive intelligence
• Surveys
• Tools and books
From Code to Product Lecture 8 — Analytics— Slide 38 gidgreen.com/course
49. Revenue
Also: UK
private
companies
From Code to Product Lecture 8 — Analytics— Slide 49 gidgreen.com/course
50. Revenue
$200k
From Code to Product Lecture 8 — Analytics— Slide 50 gidgreen.com/course
51. Lecture 8
• Introduction
• Website metrics
• Optimization
• Competitive intelligence
• Surveys
• Tools and books
From Code to Product Lecture 8 — Analytics— Slide 51 gidgreen.com/course
52. Why surveys?
• Customer feedback en masse
– Initiated by you (email/web)
– Avoid vocal minority
• Understand market
– Job descriptions
– Size of company
– Use of product
• How did you find me?
From Code to Product Lecture 8 — Analytics— Slide 52 gidgreen.com/course
53. Why surveys?
• Help with strategic decisions
– Premium offerings
– Major new versions
• Customer satisfaction
– Quantify word of mouth
• Understand abandonment
– But hard to motivate response
• Open-ended feedback
From Code to Product Lecture 8 — Analytics— Slide 53 gidgreen.com/course
54. Sources of bias
• Non-response bias
– Busy customer ≠ bad customer
• Response bias
– Word questions objectively
• Predictions vs facts
– Would you pay? How much?
• Snapshot in time
– Lots of data vs ongoing data
From Code to Product Lecture 8 — Analytics— Slide 54 gidgreen.com/course
55. Good survey design
• Keep it short!
– Focus on objectives
• Minimize burden on user
– Easy questions, especially at start
– Multiple choice
• Make it feel anonymous
– Social desirability bias
• Free text at end
From Code to Product Lecture 8 — Analytics— Slide 55 gidgreen.com/course
56. Bad questions
When did you last go online and buy something?
Would you buy our superior product?
Are you willing to pay for things online?
If we created a reliable and bug-free product which
had all of the features that you requested in
response to the questions in this survey, would you
be willing to pay us $10 per month for it?
What are you looking for?
From Code to Product Lecture 8 — Analytics— Slide 56 gidgreen.com/course
57. Lecture 8
• Introduction
• Website metrics
• Optimization
• Competitive intelligence
• Surveys
• Tools and books
From Code to Product Lecture 8 — Analytics— Slide 57 gidgreen.com/course