This document provides recommendations for building machine learning software from the perspective of Netflix's experience.
The first recommendation is to be flexible about where and when computation happens by distributing components across offline, nearline, and online systems. The second is to think about distribution starting from the outermost levels of the problem by parallelizing across subsets of data, hyperparameters, and machines. The third recommendation is to design application software for experimentation by sharing components between experiment and production code. The fourth recommendation is to make algorithms and models extensible and modular by providing reusable building blocks. The fifth recommendation is to describe input and output transformations with models. The sixth recommendation is to not rely solely on metrics for testing and instead implement unit testing of code.
4. 4
Netflix Scale
> 69M members
> 50 countries
> 1000 device types
> 3B hours/month
36.4% of peak US
downstream traffic
5. 5
Goal
Help members find content to watch and enjoy
to maximize member satisfaction and retention
6. 6
Everything is a Recommendation
Rows
Ranking
Over 80% of what
people watch
comes from our
recommendations
Recommendations
are driven by
Machine Learning
13. 13
System Architecture
Offline: Process data
Batch learning
Nearline: Process events
Model evaluation
Online learning
Asynchronous
Online: Process requests
Real-time
Netflix.Hermes
Netflix.Manhattan
Nearline
Computation
Models
Online
Data Service
Offline Data
Model
training
Online
Computation
Event Distribution
User Event
Queue
Algorithm
Service
UI Client
Member
Query results
Recommendations
NEARLINE
Machine
Learning
Algorithm
Machine
Learning
Algorithm
Offline
Computation Machine
Learning
Algorithm
Play, Rate,
Browse...
OFFLINE
ONLINE
More details on Netflix Techblog
14. 14
Where to place components?
Example: Matrix Factorization
Offline:
Collect sample of play data
Run batch learning algorithm like
SGD to produce factorization
Publish video factors
Nearline:
Solve user factors
Compute user-video dot products
Store scores in cache
Online:
Presentation-context filtering
Serve recommendations
Netflix.Hermes
Netflix.Manhattan
Nearline
Computation
Models
Online
Data Service
Offline Data
Model
training
Online
Computation
Event Distribution
User Event
Queue
Algorithm
Service
UI Client
Member
Query results
Recommendations
NEARLINE
Machine
Learning
Algorithm
Machine
Learning
Algorithm
Offline
Computation Machine
Learning
Algorithm
Play, Rate,
Browse...
OFFLINE
ONLINE
V
sij=uivj Aui=b
sij
X≈UVt
X
sij>t
16. 16
Three levels of Learning Distribution/Parallelization
1. For each subset of the population (e.g.
region)
Want independently trained and tuned models
2. For each combination of (hyper)parameters
Simple: Grid search
Better: Bayesian optimization using Gaussian
Processes
3. For each subset of the training data
Distribute over machines (e.g. ADMM)
Multi-core parallelism (e.g. HogWild)
Or… use GPUs
17. 17
Example: Training Neural Networks
Level 1: Machines in different
AWS regions
Level 2: Machines in same AWS
region
Spearmint or MOE for parameter
optimization
Mesos, etc. for coordination
Level 3: Highly optimized, parallel
CUDA code on GPUs
19. 19
Example development process
Idea Data
Offline
Modeling
(R, Python,
MATLAB, …)
Iterate
Implement in
production
system (Java,
C++, …)
Data
discrepancies
Missing post-
processing
logic
Performance
issues
Actual
output
Experimentation environment
Production environment
(A/B test) Code
discrepancies
Final
model
20. 20
Shared Engine
Avoid dual implementations
Experiment
code
Production
code
ProductionExperiment • Models
• Features
• Algorithms
• …
21. 21
Solution: Share and lean towards production
Developing machine learning is iterative
Need a short pipeline to rapidly try ideas
Want to see output of complete system
So make the application easy to experiment with
Share components between online, nearline, and offline
Use the real code whenever possible
Have well-defined interfaces and formats to allow you to go
off-the-beaten-path
23. 23
Make algorithms and models extensible and modular
Algorithms often need to be tailored for a
specific application
Treating an algorithm as a black box is
limiting
Better to make algorithms extensible and
modular to allow for customization
Separate models and algorithms
Many algorithms can learn the same model
(i.e. linear binary classifier)
Many algorithms can be trained on the same
types of data
Support composing algorithms
Data
Parameters
Data
Model
Parameters
Model
Algorithm
Vs.
24. 24
Provide building blocks
Don’t start from scratch
Linear algebra: Vectors, Matrices, …
Statistics: Distributions, tests, …
Models, features, metrics, ensembles, …
Loss, distance, kernel, … functions
Optimization, inference, …
Layers, activation functions, …
Initializers, stopping criteria, …
…
Domain-specific components
Build abstractions on
familiar concepts
Make the software put
them together
25. 25
Example: Tailoring Random Forests
Using Cognitive Foundry: http://github.com/algorithmfoundry/Foundry
Use a custom
tree split
Customize to
run it for an
hour
Report a
custom metric
each iteration
Inspect the
ensemble
30. 30
Machine Learning and Testing
Temptation: Use validation metrics to test software
When things work and metrics go up this seems great
When metrics don’t improve was it the
code
data
metric
idea
…?
31. 31
Reality of Testing
Machine learning code involves intricate math and logic
Rounding issues, corner cases, …
Is that a + or -? (The math or paper could be wrong.)
Solution: Unit test
Testing of metric code is especially important
Test the whole system: Just unit testing is not enough
At a minimum, compare output for unexpected changes across
versions
33. 33
Two ways to solve computational problems
Know
solution
Write code
Compile
code
Test code Deploy code
Know
relevant
data
Develop
algorithmic
approach
Train model
on data using
algorithm
Validate
model with
metrics
Deploy
model
Software Development
Machine Learning
(steps may involve Software Development)
34. 34
Take-aways for building machine learning software
Building machine learning is an iterative process
Make experimentation easy
Take a holistic view of application where you are placing
learning
Design your algorithms to be modular
Look for the easy places to parallelize first
Testing can be hard but is worthwhile
35. 35
Thank You Justin Basilico
jbasilico@netflix.com
@JustinBasilico
We’re hiring