Hyperparameter tuning and optimization is a powerful tool in the area of AutoML, for both traditional statistical learning models as well as for deep learning. There are many existing tools to help drive this process, including both blackbox and whitebox tuning. In this talk, we'll start with a brief survey of the most popular techniques for hyperparameter tuning (e.g., grid search, random search, Bayesian optimization, and parzen estimators) and then discuss the open source tools which implement each of these techniques. Finally, we will discuss how we can leverage MLflow with these tools and techniques to analyze how our search is performing and to productionize the best models.
Speaker: Joseph Bradley
3. TEAM
About Databricks
Started Spark project (now Apache Spark) at UC Berkeley in 2009
PRODUCT
Unified Analytics Platform
MISSION
Making Big Data Simple
Try for free today.
databricks.com
4. Hyperparameters
• Express high-level concepts, such as statistical assumptions
• Are fixed before training or are hard to learn from data
• Affect objective, test time performance, computational cost
E.g.:
• Linear Regression: regularization, # iterations of optimization
• Neural Network: learning rate, # hidden layers
5. Tuning hyperparameters
E.g.: Fitting a
polynomial
Common goals:
• More flexible modeling process
• Reduced generalization error
• Faster training
• Plug & play ML
6. Challenges in tuning
Curse of dimensionality
Non-convex optimization
Computational cost
Unintuitive hyperparameters
8. Tuning in the Data Science workflow
Training Data Test Data
ML Model
9. Tuning in the Data Science workflow
Training
Data
Validation
Data
Test Data
Final
ML Model
ML Model 1
ML Model 2
ML Model 3
10. Tuning in the Data Science workflow
ML Model
Featurization
Model family
selection
Hyperparameter
tuning
“AutoML” includes hyperparameter tuning.
11. This talk
Popular methods for hyperparameter tuning
• Overview of methods
• Comparing methods
• Open-source tools
Tuning in practice with MLflow
• Instrument tuning
• Analyze results
• Productionize models
Beyond this talk
12. This talk
Popular methods for hyperparameter tuning
• Overview of methods
• Comparing methods
• Open-source tools
Tuning in practice with MLflow
• Instrument tuning
• Analyze results
• Productionize models
Beyond this talk
13. Overview of tuning methods
• Manual search
• Grid search
• Random search
• Population-based algorithms
• Bayesian algorithms
14. Manual search
Select hyperparameter settings to try based on human intuition.
2 hyperparameters:
• [0, ..., 5]
• {A, B, ..., F}
Expert knowledge tells us to try:
(2,C), (2,D), (2,E), (3,C), (3,D), (3,E)
A B C D E F
0
1
2
3
4
5
15. Grid Search
Try points on a grid defined by ranges and step sizes
X-axis: {A,...,F}
Y-axis: 0-5, step = 1
A B C D E F
0
1
2
3
4
5
16. A B C D E F
0
1
2
3
4
5
Random Search
Sample from distributions over ranges
X-axis: Uniform({A,...,F})
Y-axis: Uniform([0,5])
17. Start with random search, then iterate:
• Use the previous “generation” to
inform the next generation
• E.g., sample from best performers &
then perturb them
Population Based Algorithms
A B C D E F
0
1
2
3
4
5
18. Start with random search, then iterate:
• Use the previous “generation” to
inform the next generation
• E.g., sample from best performers &
then perturb them
Population Based Algorithms
A B C D E F
0
1
2
3
4
5
19. Start with random search, then iterate:
• Use the previous “generation” to
inform the next generation
• E.g., sample from best performers &
then perturb them
Population Based Algorithms
A B C D E F
0
1
2
3
4
5
20. Model the loss function:
Hyperparameters à loss
Iteratively search space, trading off
between exploration and exploitation
A B C D E F
0
1
2
3
4
5
Bayesian Optimization
36. Comparing tuning methods
Iterative /
adaptive?
# evaluations
for P params
Model of
param space
Grid search No O(c^P) none
Random search No O(k) none
Population-based Yes O(k) implicit
Bayesian Yes O(k) explicit
38. This talk
Popular methods for hyperparameter tuning
• Overview of methods
• Comparing methods
• Open-source tools
Tuning in practice with MLflow
• Instrument tuning
• Analyze results
• Productionize models
Beyond this talk
39. This talk
Popular methods for hyperparameter tuning
• Overview of methods
• Comparing methods
• Open-source tools
Tuning in practice with MLflow
• Instrument tuning
• Analyze results
• Productionize models
Beyond this talk
41. Organizing with
Training Data Validation Data Test Data
Final ML ModelML Model 1
ML Model 2
ML Model 3
Experiment
Main run
Child runs
42. Instrumenting tuning with
What to track in a run for a model
• Hyperparameters: all vs. ones being tuned
• Metric(s): training & validation, loss & objective, multiple objectives
• Tags: provenance, simple metadata
• Artifacts: serialized model, large metadata
Tip: Tune full pipeline, not 1 model.
43. Analyzing how tuning performs
Questions to answer
• Am I tuning the right hyperparameters?
• Am I exploring the right parts of the search space?
• Do I need to do another round of tuning?
Examining results
• Simple case: visualize param vs metric
• Challenges: multiple params and metrics, iterative experimentation
44. Moving models to production
Repeatable experiments via MLflow Projects
• Code checkpoints
• Environments
Model serialization via MLflow Models
• Flavors: TensorFlow, Keras, Spark, MLeap, ...
Deployment to prediction services
• Azure ML, AWS Sagemaker, Spark UDF
45. Auto-tracking MLlib with
Training Data Validation Data Test Data
Final ML ModelML Model 1
ML Model 2
ML Model 3
Experiment
Main run
Child runs
In Databricks
• CrossValidator &
TrainValidationSplit
• 1 run per setting of
hyperparameters
• Avg metrics for CV folds(demo)
46. This talk
Popular methods for hyperparameter tuning
• Overview of methods
• Comparing methods
• Open-source tools
Tuning in practice with MLflow
• Instrument tuning
• Analyze results
• Productionize models
Beyond this talk
47. This talk
Popular methods for hyperparameter tuning
• Overview of methods
• Comparing methods
• Open-source tools
Tuning in practice with MLflow
• Instrument tuning
• Analyze results
• Productionize models
Beyond this talk
48. Advanced topics
Efficient tuning
• Parallelizing hyperparameter
search
• Early stopping
• Transfer learning
Fancy tuning
• Multi-metric optimization
• Conditional/awkward
parameter spaces
Check out Maneesh Bhide’s talk:
"Advanced Hyperparameter
Optimization for Deep Learning"
to hear about early stopping,
multi-metric, & conditionals
Thursday @ 3:30pm, Room 3014
49. Advanced topics
Efficient tuning: Parallelizing
hyperparameter search
Challenge in analyzing results:
multiple parameters or
multiple metrics
Hyperopt + Apache Spark
+ MLflow integration
• Hyperopt: general tuning
library for ML in Python
• Spark integration:
parallelize model tuning in
batches
• MLflow integration: track
runs, analogous to MLlib +
MLflow integration
(demo)
50. Getting started
MLflow: http://mlflow.org
MLlib tuning
• Databricks auto-tracking with MLflow in private preview now, public
preview mid-May
Hyperopt
• Distributed tuning via Apache Spark: working to open-source the code
• Databricks auto-tracking with MLflow in public preview mid-May
51. Thank You!
Questions?
AMA @ DevLounge Theater
Thursday @ 10:30-11am
Thanks to Maneesh Bhide for
material for this talk!