Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.

How to Think Product Analytics in PM Interviews by Amazon Sr PM

Main takeaways:
- Knowing what metrics to measure and how to measure them are key skills for a Product Manager. Interviewers are always going to gauge this aspect.
- How should we think about setting Product Metrics for every situation? How should we think about measuring these?
- What are the strengths and limitations of A/B testing. When can you use it and when should you rely on other methods? What are the different methods for measuring metrics and when to employ those.

  • Inicia sesión para ver los comentarios

How to Think Product Analytics in PM Interviews by Amazon Sr PM

  1. 1. www.productschool.com How to Think Product Analytics in PM Interviews by Amazon Sr PM
  2. 2. Join 35,000+Product Managers on Free Resources Discover great job opportunities Job Portal prdct.school/PSJobPortalprdct.school/events-slack
  3. 3. C O U R S E S Product Management Learn the skills you need to land a Product Manager job
  4. 4. C O U R S E S Coding for Managers Build a website and gain the technical knowledge to lead software engineers
  5. 5. C O U R S E S Data Analytics for Managers Learn the skills to understand web analytics, SQL and machine learning concepts
  6. 6. C O U R S E S Learn how to acquire more users and convert them into clients Digital Marketing for Managers
  7. 7. C O U R S E S UX Design for Managers Gain a deeper understanding of your users and deliver an exceptional end-to- end experience
  8. 8. C O U R S E S For experienced Product Managers looking to gain strategic skills needed for top leadership roles Product Leadership
  9. 9. C O U R S E S Corporate Training Level up your team’s Product Management skills
  10. 10. Vivek Pandey T O N I G H T ’ S S P E A K E R
  11. 11. Thinking Product Analytics Vivek Pandey
  12. 12. 1. Who is this guy? 2. Why should I listen to him? 3. Does he have dog pictures? (Oh yes!)
  13. 13. Vivek Pandey
  14. 14. What will we talk about today? Analytics + Product Mgmt ● Role of analytics in Product Management ● What does the interviewer *really* want to know? Analytics in PM Interviews ● Structure of questions and why they are asked ● Opportunity sizing Measurement & Metrics ● Funnel Design, A/B tests
  15. 15. What will we NOT talk about today? Product design ● Design an alarm clock for cats Fermi problems ● How many golf balls will fit into the Space Needle? Product strategy ● Should Google launch a LinkedIn competitor?
  16. 16. Part.01 Analytics as a part of Product Management
  17. 17. What does a Product Manager do?
  18. 18. What does a Product Manager do? Ideates Design new products, services, features, and improvements Prioritizes Which ideas to build and implement? Persuades Convince leadership, engineering, and other stakeholders Executes Build and launch products.Track, and manage performance
  19. 19. What is the interviewer looking for? Can you identify the levers that can be controlled? 05 Can you use data to persuade others?03 Can you correctly interpret data to make decisions?02 Can you identify what data is relevant?01 Are you realistic about what data can be collected? 04
  20. 20. Part.02 Analytics Questions and how to think about them
  21. 21. I group Product analyses into four buckets Success definition and measurement03 Prioritization02 Opportunity sizing01 Diagnosing issues04
  22. 22. Example Questions Bucket Typical Question Structure Opportunity sizing ● Should we launch product/ feature X? ● How many Italian restaurants are in Seattle? Prioritization ● Given X amount of time/money, which ideas should be prioritized? ● Of all the features we discussed, which one would you build? Success definition and measurement ● Pick one metric to manage the feature you designed? ● How will you measure success after you launch your idea? Diagnosing issues ● Ad revenue dropped by 20%; how will you identify the issue? ● Why are ‘Product Page Views’ down?
  23. 23. Product analysis: Opportunity sizing Identify purchase price05 Identify addressable segments03 Identify the base population02 Clarify scope01 Identify purchase frequency04
  24. 24. Let’s run through an example: Should we launch a bicycle-based food delivery service in Seattle?
  25. 25. There are two parts to the answer Also to consider (not covered): ● Will this be profitable? ○ Competitors ○ Achievable market share ○ Unit economics ○ Does this align with the company’s strategy? ○ Will you find enough riders/delivery-people? What is the size of the opportunity?
  26. 26. Going through the product analysis steps: In this case, we are only talking about Seattle. Clarify sources of revenue. Delivery fee? Commissions? Clarify scope01
  27. 27. Who has access to your service? Base your calculations on a geographical area, industry, or demographic. What’s your unit of measure for your customers? Individuals? Households? Which measure makes most sense for a service like this? Identify the base population02
  28. 28. What’s your unit of measure for your customers? Individuals? Households? Which measure makes most sense for a service like this? Identify the base population02 Answer: # of Households = Population of Seattle / Average Household Size so.. 800K Individuals / 2.5 Individuals per household = 320K Households
  29. 29. This is the part where you show your creativity! ● How are you identifying your segments? By geography? By income? ○ How big is the serviceable area? What are the constraints? ○ How many households are in serviceable areas? ○ How many households order in? How many can afford to? ● So… what fraction of the base population is addressable? Identify addressable segments03
  30. 30. So… what fraction of the base population is addressable? Answer: Bicycle speed is a limitation, so we are starting with tight population and restaurant clusters. Let’s assume serviceable areas are just Downtown, Ballard, Green Lake, and Capitol Hill. I will assume top 50% households by income are ordering in. Adjusting for all these factors, about 30K households are serviceable. Identify addressable segments03
  31. 31. How many times will people use your service? ● Pick a reasonable unit of measure: Weekly? Monthly? ○ For some services it gets very intuitive. ● What assumptions are you making? ○ What are you basing them on? ● Will different segments have a different purchase frequency? ○ Would singles order in more often that families with kids? Note: For physical products, are you considering replacement sales? Identify purchase frequency04
  32. 32. How many times will people use your service? Answer: Let’s assume people who order in are doing so 3 times a month on average. So, now the service will be used 30K times 3 = 90K times per month. Identify purchase frequency04
  33. 33. What is the unit price? ● Pick a reasonable price. ○ Base it on industry standards, if similar products exits. ○ Base it on the next best option, it it is something new to world. Identify purchase price05
  34. 34. What is the unit price? Answer: $5 Per delivery sounds reasonable? (Based on UberEats, Amazon Restaurants etc.) Identify purchase price05
  35. 35. Total Households = 320K Addressable Households = 30K Monthly Purchase Frequency = 3 Revenue Per Delivery = $5 Total Monthly Revenue = 30K * 3 * 5 = $450K Monthly Or $450K *12 = $5.4M annually Bringing it together:
  36. 36. $5.4M per year ● Competitors (Uber Eats, Amazon Restaurants, DoorDash etc.) ● What is our competitive advantage? ● What is the achievable Market Share ● Unit Economics per delivery ● Does this align with the company’s strategy? ● Risks? So, should we do it? ¯_(ツ)_/¯ What is the size of the opportunity? Will this be profitable?
  37. 37. Summary: Opportunity sizing Identify purchase price05 Identify addressable segments03 Identify the base population02 Clarify scope01 Identify purchase frequency04 Of course, discuss competition, seasonality, regional difference, profitable vs. unprofitable segments etc. to complete your answer.
  38. 38. Prioritization, Success Metrics, and Issue Diagnosis Questions around these topics - ● Of all the ideas we discussed, which one would you implement first? ● If you had to pick one metric to manage this feature, which one would it be? ● How will you measure performance of the feature you described? ● Revenue/visits/deliveries dropped by 20%; how will you identify the issue? All these questions have one common theme.
  39. 39. How well do you understand ‘the funnel’? Total Site Visitors 100% View Item Detail Page 70% Hit ‘Add to Cart’ button 20% Go to ‘Checkout’ 7% Complete Purchase 3% This example depicts a simple eCommerce website’s funnel
  40. 40. Inputs and outputs Total Site Visitors Inputs: Marketing SEO Seasonality 100% Visit Item Detail Page Inputs: # Items shown Relevance Price UX quality 70% Hit ‘Buy Button’ Inputs: Product description # of Images Product ratings and reviews 20% Go to ‘Checkout’ Inputs: Calls to action Promotions 7% Complete Purchase Inputs: Card on file # of fields to complete Registered user vs guest 3%
  41. 41. PMs design features to improve inputs Total Site Visitors Inputs: Marketing SEO Seasonality 100% Visit Item Detail Page Inputs: # Items shown Relevance Price UX quality 70% Hit ‘Buy Button’ Inputs: Product description # of Images Product ratings and reviews 20% Go to ‘Checkout’ Inputs: Calls to action Promotions 7% Complete Purchase Inputs: Card on file # of fields to complete Registered user vs guest 3%
  42. 42. Prioritize for impact View Item Detail Page 70% View Item Detail Page 80% Feature 1 Improves Detail Page Views from 70% to 80% Complete Purchase 3% Complete Purchase 3.5% Feature 2 Improves Purchase Completion from 3% to 3.5% R evenue +14% R evenue +17%
  43. 43. Total Site Visitors Breakdown: 50% Mobile 50% Web 100% Visit Item Detail Page Breakdown: 60% Mobile 80% Web 70% Hit ‘Buy Button’ Breakdown: 15% Mobile 25% Web 20% Go to ‘Checkout’ Breakdown: 10% Mobile 4% Web 7% Complete Purchase Breakdown: 4% Mobile 2% Web 3% Troubleshooting - identify the problematic step and narrow down possible causes. The funnel varies by segment, source of traffic etc.
  44. 44. Part.03 A/B tests
  45. 45. Control Treatment A Bvs. • A ‘Control’ group of users is presented with an experience ‘A’ that is unchanged from the status quo. • A ‘Treatment’ group is presented with an alternate experience ‘B’. • The behavior of these two groups is compared over time. A/B Test Primer B Select a test Metric Calculate Sample Size Run Test Analyze Results
  46. 46. Common mistakes with A/B tests Picking the wrong test metric01 Financial metrics such as ‘Revenue per order’, ‘Profit’ etc. have a very high natural degree of variance. These make lousy candidates for A/B testing. When variance is high, you need a huge sample size. Tests can run for years! Speed is key, nobody has time for this. Tip - Pick low variance metrics such as ‘Orders per user’, ‘Click thru rate’ etc. Click thrus and conversion are more telling anyway.
  47. 47. Common mistakes with A/B tests Misunderstanding ‘Statistical Significance’02 You do not run a test until it shows statistical significance. You run an A/B test until you hit the required sample size and then you check the results. At that point - 1. There is a statistically significant difference between Control and Treatment. OR 1. There is no statistically significant difference between Control and Treatment. Tip - Use Evan Miller’s A/B testing blog for calculators.
  48. 48. Common mistakes with A/B tests ‘Peeking’ at results to draw ‘early data’03 Do not try to read into results until Sample Size is reached. We plotted the results of 100 experiments. Even though ~60% of them showed significance at some point, only ONE had a statistically significant difference after sample sizes were reached.
  49. 49. Common mistakes with A/B tests Ignoring ‘seasonality’04 On a high traffic website (think Netflix, Google, Amazon), I may reach sample size in a few hours. But does that give me the full picture? Users may behave differently over weekends. Make sure you run the test long enough to capture nuances.
  50. 50. When you can’t use A/B tests…. ● Legal issues ● Speed / Urgency ● Content based features A pre/post analysis is a messy but acceptable last-resort alternative. i.e. measure performance before and after the change, accounting for other impacts.
  51. 51. “If you can’t measure it, you can’t manage it” - Peter Drucker
  52. 52. “If you can’t measure it, you can’t manage it” - Peter Drucker - William Edwards Deming
  53. 53. “It is wrong to suppose that if you can’t measure it, you can’t manage it – a costly myth.” - William Edwards Deming, (The New Economics, Page 35)
  54. 54. Thank you!!!
  55. 55. www.productschool.com Part-time Product Management, Coding, Data Analytics, Digital Marketing, UX Design, Product Leadership courses and Corporate Training

×