SlideShare una empresa de Scribd logo
1 de 51
Watch the video with slide 
synchronization on InfoQ.com! 
http://www.infoq.com/presentations 
/machine-learning-netflix-2014 
InfoQ.com: News & Community Site 
• 750,000 unique visitors/month 
• Published in 4 languages (English, Chinese, Japanese and Brazilian 
Portuguese) 
• Post content from our QCon conferences 
• News 15-20 / week 
• Articles 3-4 / week 
• Presentations (videos) 12-15 / week 
• Interviews 2-3 / week 
• Books 1 / month
Presented at QCon New York 
www.qconnewyork.com 
Purpose of QCon 
- to empower software development by facilitating the spread of 
knowledge and innovation 
Strategy 
- practitioner-driven conference designed for YOU: influencers of 
change and innovation in your teams 
- speakers and topics driving the evolution and innovation 
- connecting and catalyzing the influencers and innovators 
Highlights 
- attended by more than 12,000 delegates since 2007 
- held in 9 cities worldwide
Machine Learning At 
Netflix Scale 
Aish Fenton 
Manager - Research Engineering 
@aishfenton
Everything is a 
recommendation
4
Top Picks for Aish
Movies based on books
Because you watched Bob’s Burgers
Rank based on your taste 
Rank based on your taste
75% of plays come 
from homepage
Back Story…
What we were interested in: 
▪High quality recommendations 
Proxy question: 
▪Accuracy in predicted rating 
▪Improve by 10% = $1million! 
predicted 
actual
Top two results still used in production! 
SVD RBMs
>
2006 2013
• > 44M members 
• > 40 countries 
• > 5B hours in Q3 2013 
• Log 100B events/day 
• 31.62% of peak US downstream traffic
Data and Models
▪> 40M subscribers 
▪Ratings: ~5M/day 
▪Searches: >3M/day 
▪Plays: > 50M/day 
▪Streamed hours: 
o 5B hours in Q3 2013 
Geo Info 
Time 
Impressions 
Ratings 
Device Info 
Metadata 
Social 
Plays 
Demographics 
Member Behavior
Latent User Vector 
Aish House of Cards 
Latent Item Vector
House of Cards 
m1 
m2 
m3 
House of Cards 
3.53 
M 
u1 u2 u3 
Aish Aish 
U R
Mean Rating My Bias 
Movie Bias 
Interaction
3.55 = 2.50 + -1.5 + 1.2 + pq 
Mean Rating My Bias 
Movie Bias 
Interaction 
My rating for 
House of Cards
House of Cards 
3.53 
R 
U 
M 
u1 u2 u3 
m1 
m2 
m3 
Aish 
1.34 
2.35 
Time 
t1 t2 t3 Time 
T
▪Matrix/Tensor Factorization 
▪Regression models (Logistic, Linear, Elastic nets) 
▪Factorization Machines 
▪Restricted Boltzmann Machines 
▪Markov Chains & other graph models 
▪Clustering / Topic Models 
▪Neural Networks 
▪Association Rules 
▪GBDT/RF 
▪…
Improvement Over Baseline 
Popularity 
+ Ratings 
+ More Features & Optimized Models 
0% 
50% 
100% 
150% 
200% 
250% 
300%
Anatomy of a 
Machine Learning 
Platform
Problem 
Data 
Experiment 
Offline 
Test / 
Metrics 
Produce 
Model
Offline 
Near-line 
Online 
Event 
Distribution 
UI Clients 
Online 
Algs 
Model 
Trainer 
Pre-compute 
AB Test 
Metrics 
API Layer 
Monitoring 
Hadoop / Data Warehouse 
Experimentation 
Platform 
S3 / HDFS 
Offline 
Metrics 
Query Tools 
Models 
Models
Offline 
Near-line 
Online 
Event 
Distribution 
UI Clients 
Online 
Algs 
Model 
Trainer 
Pre-compute 
AB Test 
Metrics 
API Layer 
Monitoring 
Hadoop / Data Warehouse 
Experimentation 
Platform 
S3 / HDFS 
Offline 
Metrics 
Query Tools 
Models 
Models
Many different types of data… 
▪App Logs 
▪User Actions 
▪Ratings 
▪Plays 
▪Queue Adds 
▪Algo Actions 
▪Impressions (Presentation Bias) 
▪Context 
▪Device Info 
▪User Demographics 
▪Social 
▪Time 
▪…
Offline 
Near-line 
Online 
Event 
Distribution 
UI Clients 
Online 
Algs 
Model 
Trainer 
Pre-compute 
AB Test 
Metrics 
API Layer 
Monitoring 
Hadoop / Data Warehouse 
Experimentation 
Platform 
S3 / HDFS 
Offline 
Metrics 
Query Tools 
Models 
Models 
Embedded 
Embedded
Weights 
Real-time popularity of movie
Example: Neural Network Training
θ 
Input Hidden Layer Output
Input Hidden Layers Output
Neural Network Training 
1,536 cores 
G2 Instances 
$0.60 p/h
But… things can go astray
Offline 
Near-line 
Online 
Event 
Distribution 
UI Clients 
Online 
Algs 
Model 
Trainer 
Pre-compute 
AB Test 
Metrics 
API Layer 
Monitoring 
Hadoop / Data Warehouse 
Experimentation 
Platform 
S3 / HDFS 
Offline 
Metrics 
Query Tools 
Models 
Models
M 
U R 
Pre-compute 
Online u1 u2 u3
Offline 
Near-line 
Publish new model 
Online 
Event 
Distribution 
UI Clients 
Online 
Algs 
Model 
Trainer 
Pre-compute 
AB Test 
Metrics 
API Layer 
for Aish 
Monitoring 
Hadoop / Data Warehouse 
Experimentation 
Platform 
S3 / HDFS 
Offline 
Metrics 
Query Tools 
Models 
Models 
Aish played HoC
Aish Fenton 
@aishfenton 
https://www.linkedin.com/profile/view?id=47917219
Watch the video with slide synchronization on 
InfoQ.com! 
http://www.infoq.com/presentations/machine-learning- 
netflix-2014

Más contenido relacionado

Más de C4Media

Más de C4Media (20)

Does Java Need Inline Types? What Project Valhalla Can Bring to Java
Does Java Need Inline Types? What Project Valhalla Can Bring to JavaDoes Java Need Inline Types? What Project Valhalla Can Bring to Java
Does Java Need Inline Types? What Project Valhalla Can Bring to Java
 
Service Meshes- The Ultimate Guide
Service Meshes- The Ultimate GuideService Meshes- The Ultimate Guide
Service Meshes- The Ultimate Guide
 
Shifting Left with Cloud Native CI/CD
Shifting Left with Cloud Native CI/CDShifting Left with Cloud Native CI/CD
Shifting Left with Cloud Native CI/CD
 
CI/CD for Machine Learning
CI/CD for Machine LearningCI/CD for Machine Learning
CI/CD for Machine Learning
 
Fault Tolerance at Speed
Fault Tolerance at SpeedFault Tolerance at Speed
Fault Tolerance at Speed
 
Architectures That Scale Deep - Regaining Control in Deep Systems
Architectures That Scale Deep - Regaining Control in Deep SystemsArchitectures That Scale Deep - Regaining Control in Deep Systems
Architectures That Scale Deep - Regaining Control in Deep Systems
 
ML in the Browser: Interactive Experiences with Tensorflow.js
ML in the Browser: Interactive Experiences with Tensorflow.jsML in the Browser: Interactive Experiences with Tensorflow.js
ML in the Browser: Interactive Experiences with Tensorflow.js
 
Build Your Own WebAssembly Compiler
Build Your Own WebAssembly CompilerBuild Your Own WebAssembly Compiler
Build Your Own WebAssembly Compiler
 
User & Device Identity for Microservices @ Netflix Scale
User & Device Identity for Microservices @ Netflix ScaleUser & Device Identity for Microservices @ Netflix Scale
User & Device Identity for Microservices @ Netflix Scale
 
Scaling Patterns for Netflix's Edge
Scaling Patterns for Netflix's EdgeScaling Patterns for Netflix's Edge
Scaling Patterns for Netflix's Edge
 
Make Your Electron App Feel at Home Everywhere
Make Your Electron App Feel at Home EverywhereMake Your Electron App Feel at Home Everywhere
Make Your Electron App Feel at Home Everywhere
 
The Talk You've Been Await-ing For
The Talk You've Been Await-ing ForThe Talk You've Been Await-ing For
The Talk You've Been Await-ing For
 
Future of Data Engineering
Future of Data EngineeringFuture of Data Engineering
Future of Data Engineering
 
Automated Testing for Terraform, Docker, Packer, Kubernetes, and More
Automated Testing for Terraform, Docker, Packer, Kubernetes, and MoreAutomated Testing for Terraform, Docker, Packer, Kubernetes, and More
Automated Testing for Terraform, Docker, Packer, Kubernetes, and More
 
Navigating Complexity: High-performance Delivery and Discovery Teams
Navigating Complexity: High-performance Delivery and Discovery TeamsNavigating Complexity: High-performance Delivery and Discovery Teams
Navigating Complexity: High-performance Delivery and Discovery Teams
 
High Performance Cooperative Distributed Systems in Adtech
High Performance Cooperative Distributed Systems in AdtechHigh Performance Cooperative Distributed Systems in Adtech
High Performance Cooperative Distributed Systems in Adtech
 
Rust's Journey to Async/await
Rust's Journey to Async/awaitRust's Journey to Async/await
Rust's Journey to Async/await
 
Opportunities and Pitfalls of Event-Driven Utopia
Opportunities and Pitfalls of Event-Driven UtopiaOpportunities and Pitfalls of Event-Driven Utopia
Opportunities and Pitfalls of Event-Driven Utopia
 
Datadog: a Real-Time Metrics Database for One Quadrillion Points/Day
Datadog: a Real-Time Metrics Database for One Quadrillion Points/DayDatadog: a Real-Time Metrics Database for One Quadrillion Points/Day
Datadog: a Real-Time Metrics Database for One Quadrillion Points/Day
 
Are We Really Cloud-Native?
Are We Really Cloud-Native?Are We Really Cloud-Native?
Are We Really Cloud-Native?
 

Último

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Safe Software
 
Architecting Cloud Native Applications
Architecting Cloud Native ApplicationsArchitecting Cloud Native Applications
Architecting Cloud Native Applications
WSO2
 
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
?#DUbAI#??##{{(☎️+971_581248768%)**%*]'#abortion pills for sale in dubai@
 
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Victor Rentea
 

Último (20)

Platformless Horizons for Digital Adaptability
Platformless Horizons for Digital AdaptabilityPlatformless Horizons for Digital Adaptability
Platformless Horizons for Digital Adaptability
 
Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...
Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...
Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...
 
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
 
Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...
 
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
 
Artificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : UncertaintyArtificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : Uncertainty
 
Architecting Cloud Native Applications
Architecting Cloud Native ApplicationsArchitecting Cloud Native Applications
Architecting Cloud Native Applications
 
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
 
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdfRising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
 
Boost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfBoost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdf
 
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
 
MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024
 
AWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of TerraformAWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of Terraform
 
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
 
Mcleodganj Call Girls 🥰 8617370543 Service Offer VIP Hot Model
Mcleodganj Call Girls 🥰 8617370543 Service Offer VIP Hot ModelMcleodganj Call Girls 🥰 8617370543 Service Offer VIP Hot Model
Mcleodganj Call Girls 🥰 8617370543 Service Offer VIP Hot Model
 
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWEREMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
 
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data DiscoveryTrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
 
Six Myths about Ontologies: The Basics of Formal Ontology
Six Myths about Ontologies: The Basics of Formal OntologySix Myths about Ontologies: The Basics of Formal Ontology
Six Myths about Ontologies: The Basics of Formal Ontology
 
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot TakeoffStrategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
 
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, AdobeApidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
 

Machine Learning at Netflix Scale

  • 1.
  • 2. Watch the video with slide synchronization on InfoQ.com! http://www.infoq.com/presentations /machine-learning-netflix-2014 InfoQ.com: News & Community Site • 750,000 unique visitors/month • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • News 15-20 / week • Articles 3-4 / week • Presentations (videos) 12-15 / week • Interviews 2-3 / week • Books 1 / month
  • 3. Presented at QCon New York www.qconnewyork.com Purpose of QCon - to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide
  • 4. Machine Learning At Netflix Scale Aish Fenton Manager - Research Engineering @aishfenton
  • 5. Everything is a recommendation
  • 6. 4
  • 9. Because you watched Bob’s Burgers
  • 10.
  • 11. Rank based on your taste Rank based on your taste
  • 12. 75% of plays come from homepage
  • 14.
  • 15. What we were interested in: ▪High quality recommendations Proxy question: ▪Accuracy in predicted rating ▪Improve by 10% = $1million! predicted actual
  • 16. Top two results still used in production! SVD RBMs
  • 17. >
  • 19. • > 44M members • > 40 countries • > 5B hours in Q3 2013 • Log 100B events/day • 31.62% of peak US downstream traffic
  • 21. ▪> 40M subscribers ▪Ratings: ~5M/day ▪Searches: >3M/day ▪Plays: > 50M/day ▪Streamed hours: o 5B hours in Q3 2013 Geo Info Time Impressions Ratings Device Info Metadata Social Plays Demographics Member Behavior
  • 22. Latent User Vector Aish House of Cards Latent Item Vector
  • 23. House of Cards m1 m2 m3 House of Cards 3.53 M u1 u2 u3 Aish Aish U R
  • 24.
  • 25. Mean Rating My Bias Movie Bias Interaction
  • 26. 3.55 = 2.50 + -1.5 + 1.2 + pq Mean Rating My Bias Movie Bias Interaction My rating for House of Cards
  • 27. House of Cards 3.53 R U M u1 u2 u3 m1 m2 m3 Aish 1.34 2.35 Time t1 t2 t3 Time T
  • 28. ▪Matrix/Tensor Factorization ▪Regression models (Logistic, Linear, Elastic nets) ▪Factorization Machines ▪Restricted Boltzmann Machines ▪Markov Chains & other graph models ▪Clustering / Topic Models ▪Neural Networks ▪Association Rules ▪GBDT/RF ▪…
  • 29. Improvement Over Baseline Popularity + Ratings + More Features & Optimized Models 0% 50% 100% 150% 200% 250% 300%
  • 30. Anatomy of a Machine Learning Platform
  • 31. Problem Data Experiment Offline Test / Metrics Produce Model
  • 32. Offline Near-line Online Event Distribution UI Clients Online Algs Model Trainer Pre-compute AB Test Metrics API Layer Monitoring Hadoop / Data Warehouse Experimentation Platform S3 / HDFS Offline Metrics Query Tools Models Models
  • 33. Offline Near-line Online Event Distribution UI Clients Online Algs Model Trainer Pre-compute AB Test Metrics API Layer Monitoring Hadoop / Data Warehouse Experimentation Platform S3 / HDFS Offline Metrics Query Tools Models Models
  • 34. Many different types of data… ▪App Logs ▪User Actions ▪Ratings ▪Plays ▪Queue Adds ▪Algo Actions ▪Impressions (Presentation Bias) ▪Context ▪Device Info ▪User Demographics ▪Social ▪Time ▪…
  • 35.
  • 36.
  • 37. Offline Near-line Online Event Distribution UI Clients Online Algs Model Trainer Pre-compute AB Test Metrics API Layer Monitoring Hadoop / Data Warehouse Experimentation Platform S3 / HDFS Offline Metrics Query Tools Models Models Embedded Embedded
  • 40.
  • 41. θ Input Hidden Layer Output
  • 43. Neural Network Training 1,536 cores G2 Instances $0.60 p/h
  • 44. But… things can go astray
  • 45.
  • 46.
  • 47. Offline Near-line Online Event Distribution UI Clients Online Algs Model Trainer Pre-compute AB Test Metrics API Layer Monitoring Hadoop / Data Warehouse Experimentation Platform S3 / HDFS Offline Metrics Query Tools Models Models
  • 48. M U R Pre-compute Online u1 u2 u3
  • 49. Offline Near-line Publish new model Online Event Distribution UI Clients Online Algs Model Trainer Pre-compute AB Test Metrics API Layer for Aish Monitoring Hadoop / Data Warehouse Experimentation Platform S3 / HDFS Offline Metrics Query Tools Models Models Aish played HoC
  • 50. Aish Fenton @aishfenton https://www.linkedin.com/profile/view?id=47917219
  • 51. Watch the video with slide synchronization on InfoQ.com! http://www.infoq.com/presentations/machine-learning- netflix-2014