Aimed at product managers, this talk is about how to best use data to inform product decisions.

1. Using data to inform
product decisions
Why we do need data to inform product development
ProductTank Cape Town - 26 March ‘15

2. Imagine this ...

3. ... or this

4. Outline
Why
Driven vs Informed
What
5 things to be mindful of

5. Why do we need data?

6. Why do we need data?

7. What data do we need?
It’s about asking the right questions, you fool!!

8. Data & the product lifecycle

9. What do we want?
Assumption: “Sitting on a tractor all day isn’t the best use of
my time”
Assumption: My users want to spend less time on the
tractor so that they can spend more time on other tasks
Question: Is there a market need for driverless tractors?

10. What do we want?
Hypothesis: We believe this is true if the users of our
MVP spend 20% more time on the farm
Approaches: One Single Metric, prototypes, MVP,
observations, market research, diary studies

11. What do we want?

12. What do we want?

13. How should it work?

14. How should it work?
Question: How can we encourage people to discover and
configure multiple cars?
Assumption: People will be encouraged to explore multiple
cars if they see nice images of cars similar to the one they
have just configured

15. How should it work?
Hypothesis: We believe that adding images will drive car
discovery. We know this is true if there’s a 30% increase in
the average number of cars configured per person by end
of May ’15
Approaches: A/B and MVT, behavioural plan & KPIs,
prototypes and usability testing
Approaches: A/B and MVT, behavioural plan & KPIs,
prototypes and usability testing

16. “Blank slate”

17. “Blank slate”
One Single Metric: Percentage of users per variant who
configure another car
Design and sample size: Minimum of 200 conversions per
page to reach “statistical significance”
BUT: I can’t learn everything through this experiment!

18. How is it working?

19. How is it working?
Question: Is our product / feature meeting the hypothesis?
Assumption: We believe that this feature will be used by
50% of our first time car buyers in the UK within the first
month after release
Question: What is our strongest market or user segment?

20. How is it working?
Hypothesis: We know that our assumption is correct
if we see a 20% increase (on the current benchmark) in the
number of UK first time car buyers purchasing a car through
our site
Approaches: Usage tracking, user testing, product
retrospectives and refine or reject hypothesis
Identify opportunities for product improvement or
reasons for discontinuation

21. How is it working?

22. Gathering the right data

23. What can quant data tell us?

24. What can quant data tell us?

25. Analytics

26. What can qual data tell us?
Qualitative data can help us:
!
Understand the why behind quantitative data
!
Get insight into what people think and feel
!
Learn about a product idea or prototype
!
!

27. What can qual data tell us?

28. What can qual data tell us?

29. What can qual data tell us?

30. Data driven

31. Data driven
A/B or multi-variate test continuously
!
Focus on the “One Metric That Matters”
!
Build hypothesis around key KPI
!
Optimise your product based on data
!
Are we making a noticeable difference?

32. BUT...What data cannot tell
Is it a good product idea?
!
Metrics do not always offer you the full
picture
!
Data is one of the factors that feed into a
decision
!
We typically do not own all product
decisions

33. Data informed

34. Data informed
Data
Users
Intuition
Competition
Technology
Brand
Strategy
Business
Regulation
Time

35. Data informed
Data is one of the factors to consider
!
Focus on the questions that you want
answered
!
You cannot replace intuition or creative
ideas with data
!
Assess impact on relevant areas

36. 5 things to be mindful of
Focus on asking the right questions
!
Data can’t replace intuition
!
Be clear on hypothesis, sample size and timings
!
Build and launch with data in mind
!
Listen to the data and act accordingly!

37. SO ...
Embrace the data, don’t fear it!

38. Thanks!
marcabrahamlondon@gmail.com
!
marcabraham.wordpress.com
!
@MAA1
!
!
!

39. Related links
http://svpg.com/assessing-product-opportunities/
!
http://www.romanpichler.com/blog/goal-oriented-agile-product-roadmap/
http://vimeo.com/14999991
http://www.realityisagame.com/archives/390/wooga-follows-zynga-in-metrics-driven-game-design/
http://marcabraham.wordpress.com/2013/05/03/book-review-lean-analytics/
http://www.kaushik.net/avinash/web-analytics-101-definitions-goals-metrics-kpis-dimensions-
targets/
http://marcabraham.wordpress.com/2013/09/09/some-considerations-regarding-data-driven-
design/
http://insideintercom.io/the-problem-with-data-driven-decisions/
http://www.webdesignerdepot.com/2013/05/the-perils-of-ab-testing/
http://andrewchen.co/2008/09/08/how-to-measure-if-users-love-your-product-using-cohorts-and-
revisit-rates/
http://codeascraft.com/2012/06/21/building-websites-with-science/

40. Related links
!
https://marcabraham.wordpress.com/2015/03/05/what-is-guerrilla-testing/
http://www.slideshare.net/LilyDart/guerrilla-testing-for-content
https://marcabraham.wordpress.com/2015/02/14/learning-more-about-running-ab-tests/
https://marcabraham.wordpress.com/2015/02/04/book-review-thinking-with-data/
https://marcabraham.wordpress.com/2015/01/26/book-review-designing-for-behavior-change/
http://www.simplypsychology.org/qualitative-quantitative.html
http://data.heapanalytics.com/dont-stop-your-ab-tests-part-way-through/
https://marcabraham.wordpress.com/2014/12/22/book-review-web-metrics/
!
!
!