Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.

Wayfair's Data Science Team and Case Study: Uplift Modeling

8.049 visualizaciones

Publicado el

Jen Wang, Wayfair Data Science Team, Projects, and Case Study -- Uplift Modeling for Driving Incremental Revenue in Display Remarketing

Publicado en: Tecnología
  • Inicia sesión para ver los comentarios

Wayfair's Data Science Team and Case Study: Uplift Modeling

  1. 1. Northeastern University, 4 April 2018 Jen Wang Wayfair Data Science Team, Projects, and Case Study -- Uplift Modeling for Driving Incremental Revenue in Display Remarketing
  2. 2. 2 Wayfair: e-Commerce Tech Company Our typical customer: 35 to 65 year old woman with annual household income of $50k to $250k; comScore median household income of $82k
  3. 3. 3 A Clear Online Leader in Home Goods OtherDirect Retail
  4. 4. 4 Main goals of seminar 1. Data Science Team in Wayfair – How Data Science work is organized at Wayfair, and the different types of projects we work on 2. Marketing Data Science – How Data Science projects are aligned against different points of the marketing funnel 3. Case Studies in MKT DS – Uplift Modeling for Driving Incremental Revenue in Display Remarketing
  5. 5. 5 Data Science Team in Wayfair
  6. 6. 6 Venn Diagram for Wayfair Data Science Commonalities across companies Trade-offs Research Application Engineering • Develop & apply machine learning algorithms to find answers to business problems • Great range of algorithm complexity (from linear / logistic regression to deep learning), but always need sufficient, “big” data to get good results • Standard set of technical tools, from R / Python for scripting to Spark for big data processing • Innovate by creating new algorithms / approaches to solve problems • Typically “bet big”, but doesn’t always pay off • Innovate by efficiently adapting existing approaches to solve problems • Can sometimes lead to more incremental progress, tricky to build for long-term • Typically use business rules & simpler algorithms • Focus on robustness & scalability first, then modeling Business Problem Solving Engineering Research / Modeling / ML Warning: No Unicorns!!! Modeling • Build “right” model first, then whittle away to get in form ready for production • Need to be mindful of 80/20
  7. 7. 7 Data Science Groups at Wayfair DS Infrastructure DS Operation Catalog Optimization NLP / CNN Competitive Intelligence NLP DS Marketing Customer Scoring & Bidding B2B Uplift Model Text Mining E.g. E.g. Business Problem Solving Engineering Research / Modeling / ML Business Problem Solving Engineering Research / Modeling / ML DS Product Recommenda- -tion System Visual Search Reinforcement learning / CF E.g. CNN Business Problem Solving Engineering Research / Modeling / ML
  8. 8. 8 Marketing Data Science: Business Problems
  9. 9. 9 The key objective of Marketing Data Science is aligned to maximize return on marketing investment by optimizing budget allocation, channel strategy and customer journey touchpoint MKT Channel A E.g. TV MKT Channel B E.g. Search MKT Channel X E.g. Retargeting MKT Budget Maximize MROI = 𝑅𝑒𝑣𝑒𝑛𝑢𝑒 𝐴𝑑 𝐶𝑜𝑠𝑡 By… 1. Guide the right budget allocation across channels in our marketing portfolio 2. Provide channel level tactical guidance towards delivering the right message to the right customer through the right channel 3. Lineup the marketing treatments in the right sequence and right cadence along the customer life journey Customer Life Time Value $ T1 T2 TN Systematic View of Marketing Marketing DS Objective
  10. 10. 10 (1/5) Example of Marketing Data Science Project MKT Channel A E.g. TV MKT Channel B E.g. Search MKT Channel X E.g. Retargeting MKT Budget 1. Guide the right budget allocation across channels in our marketing portfolio Q for DS: How would you measure the revenue contributed by different channels? Customer Life Time Value $ T1 T2 TN Order Attribution base on Incremental Value
  11. 11. 11 (2/5) Example of Marketing Data Science Project MKT Channel A E.g. TV MKT Channel B E.g. Search MKT Channel X E.g. Retargeting MKT Budget 2. Provide channel level tactical guidance towards delivering the right message to the right customer through the right channel Q for DS: • How to decide which TV channel should be invested with more ads? Customer Life Time Value $ T1 TV Targeting T2 TN
  12. 12. 12 (3/5) Example of Marketing Data Science Project MKT Channel A E.g. TV MKT Channel B E.g. Search MKT Channel X E.g. Retargeting MKT Budget 2. Provide channel level tactical guidance towards delivering the right message to the right customer through the right channel Q for DS: • How much we should bid on each google keyword? Customer Life Time Value $ T1 Keyword Bidding T2 TN
  13. 13. 13 (4/5) Example of Marketing Data Science Project MKT Channel A E.g. TV MKT Channel B E.g. Search MKT Channel X E.g. Retargeting MKT Budget 2. Provide channel level tactical guidance towards delivering the right message to the right customer through the right channel Q for DS: • How much we should bid the ads on each customer? Customer Life Time Value $ T1 Display Ads T2 TN
  14. 14. 14 (5/5) Example of Marketing Data Science Project MKT Channel A E.g. TV MKT Channel B E.g. Search MKT Channel X E.g. Retargeting MKT Budget 3. Schedule the marketing treatments in the right sequence and right cadence along the customer life journey Q for DS: • How often we should send the marketing emails to customers? Customer Life Time Value $ T1 T2 TN
  15. 15. 15 Case Study: Uplift Modeling to Drive Incremental Revenue in Display Remarketing
  16. 16. 16 Customer Scoring Uplift modeling for prediction of incremental revenue as base bid for each customer 𝒚= Ad-inventory Scoring Click-through rate prediction as bid modifier across Internet Data Science Solutions 𝑦1 = P(buy | Ad) - P(buy | no Ad) y2 = $Rev(buy | Ad) Expected incr. Rev Uplift Base Bid Case Study – Uplift Modeling in Display Remarketing Display Remarketing Why is it challenging? • Billions of bidding opportunities across Internet per day • Real-time bidding • Customer-level prediction • Causal effects of Ad targeting? *numbers are Illustrative only Final Bid! X 𝐶𝑇𝑅 = 𝐶𝑖𝑐𝑘𝑠 𝐼𝑚𝑝𝑟𝑒𝑠𝑠𝑖𝑜𝑛𝑠 Bid Modifier
  17. 17. 17 Modeling and Evaluation • Random Targeting to Collect Data • Uplift Modeling • Score (Predicted Uplift) = P(buy | Ad) - P(buy | no Ad) • Uplift = Test CVR - Control CVR Control Group: PSA Test Group: Ads Seen Case Study – Uplift Modeling in Display Remarketing Background • A method for modeling and predicting causal effects • Target most incremental (or persuadable) customers • Obama Camp persuaded millions of voters with Uplift Modeling in 2012 Persuadables Sure Things Lost Causes Sleeping Dogs Will Convert if Not Treated WillConvertifTreated No Yes NoYes NumberofIncrementalCustomers 10 20 30 40 50 60 70 80 90 100 20 30 1040 01020 Number of Customers Targeted & Random Targeting Perfect Uplift Model Good Uplift Model Perfect Conversion Model Good Conversion Model
  18. 18. 18 Jen Wang’s Journey to Data Science Ph.D. in (Biophysical) Chemistry Postdoc in Drug Design Health Data Science Fellow Marketing Data Scientist
  19. 19. 19 Wayfair: We Are Hiring! • Wayfair DS Career: https://www.wayfaircareers.com/ • Wayfair DS Blog: http://tech.wayfair.com/category/data-science/ If you are interested in Wayfair data science... Happy to answer any question you have… • LinkedIn: https://www.linkedin.com/in/jenzhenwang/

×