Large scale-ctr-prediction lessons-learned-florian-hartl
•
2 recomendaciones•957 vistas
Florian Hartl
Recomendados
Recomendados
Más contenido relacionado
La actualidad más candente
La actualidad más candente (18)
Destacado
Destacado (19)
Similar a Large scale-ctr-prediction lessons-learned-florian-hartl
Similar a Large scale-ctr-prediction lessons-learned-florian-hartl (20)
Más de PyData
Más de PyData (20)
Último
Último (20)
Large scale-ctr-prediction lessons-learned-florian-hartl
- 1. Florian Hartl florianh@yelp.com Large Scale CTR Prediction Lessons Learned
- 2. Yelp’s Mission Connecting people with great local businesses.
- 3. 92M 3272%108M Yelp Stats As of Q2 2016
- 4. CTR Prediction CTR: Click-Through Rate pCTR: predicted CTR Question How likely is the user to click on the ad? Why Proxy for relevance 5.5% 0.8% 9.2% ?
- 5. Logistic Regression with thousands of features, trained and tested on millions of samples. Current pCTR Model Kuvasz
- 6. pCTR Model History (CC) from Flickr: "Wednesday Freedom 11" by Parker Knight (CC) from Flickr: "Icelandig sheepdog" by Thomas Quine(CC) from Flickr: by Craige Moore French Brittany Icelandic Sheepdog Jindo Kuvasz
- 7. Lessons Learned (CC) from Flickr: "WEL" by luckyno3
- 9. (CC) from Flickr: "The huge crossing" by Miroslav Petrasko Infrastructure (CC) from Flickr: "KOGI and WEL" by luckyno3
- 11. user feedbackservice logs Log at source of online prediction → Prevents downstream modifications of data Logging
- 13. data logs prediction verification Assert validity of logged data Verification model
- 15. data model logs prediction verification fast scalable Make offline training iterations fast & scalable Automation is key → end-to-end pipeline → automated visualizations Tools: mrjob, Spark Iterations
- 16. Offline Training at Yelp merge logs sampling feature extraction model training evaluation mrjob AWS EMR daily scheduled pipeline kicked off manually mrjob AWS EMR Spark mrjob AWS EMR mrjob AWS EMR mrjob AWS EMR new features (CC) from Flickr: "Cloud" by Jason Pratt
- 17. Lessons Learned Infrastructure Log at source of online prediction Verify predictions Make offline iterations fast & scalable
- 18. Model Comprehension (CC) from Flickr: "Bella" by Maureen Lee
- 20. Focus on a single metric (but don't trust it blindly) Evaluation data model prediction verification evaluation fast scalable
- 21. Our Metric
- 22. Focus on a single metric (but don't trust it blindly) Create helpful visualizations Tools: Zeppelin Evaluation data model prediction verification evaluation fast scalable
- 23. Visualizations ... feature 1 feature 2 feature 3 ... feature contribution Feature contributions sd(feature) * coef Feature value vs. CTR count feature value CTR
- 24. user feedbackservice onlineoffline data model logs prediction verification evaluation fast scalable
- 25. logs Beware of biased training data → offline != online → pCTR threshold Thresholds user feedbackservice
- 26. pCTR Threshold CTR pCTR Model 1 Good CTR pCTR Model 2 Bad CTR pCTR Model 3 Good
- 27. pCTR Threshold time training data Model 1 Model 2 Model 3 Model 4 Idea: Frequent retraining Better: Deliberate sampling of bad ads CTR pCTR
- 28. Online Evaluation CTR pCTR Model 1 Good CTR pCTR Model 2 Bad CTR pCTR Model 3 Good
- 29. Online Evaluation CTR pCTR Model 1 Good CTR pCTR Model 2 Bad CTR pCTR Model 3 Good
- 30. Combined Rescoring new modelcurrent model online offline
- 31. Combined Rescoring new modelcurrent model online offline evaluation
- 32. Lessons Learned Infrastructure Log at source of online prediction Verify predictions Make offline iterations fast & scalable Model Comprehension Evaluate, evaluate, evaluate Be aware of threshold effects
- 33. user feedbackservice onlineoffline data model logs prediction verification evaluation fast scalable
- 35. simplicity rule-based approach simple models Occam's razor appropriate metric documentation "Simple Made Easy"
- 36. user feedbackservice onlineoffline data model logs prediction verification evaluation fast scalable well documented fast scalable well documented simplicity
- 37. user feedbackservice onlineoffline data model logs prediction verification evaluation fast scalable well documented fast scalable well documented simplicity
- 38. Lessons Learned Above all, keep it simple. Infrastructure Log at source of online prediction Verify predictions Make offline iterations fast & scalable Model Comprehension Evaluate, evaluate, evaluate Be aware of threshold effects