Best Practices for Hyperparameter Optimization: All machine learning and artificial intelligence pipelines – from reinforcement agents to deep neural nets – have tunable hyperparameters. Optimizing these hyperparameters provides tremendous performance gains, but only if the optimization is done correctly. This presentation will discuss topics including selecting performance criteria, why you should always use cross validation, and choosing between state of the art optimization methods.

1. Best Practices for Hyperparameter
Optimization
Alexandra Johnson
@alexandraj777

2. Example: Beating Vegas
Scott Clark. Using Model Tuning to Beat Vegas.

3. Terminology
● Optimization
● Hyperparameter
Optimization
● Hyperparameter
tuning
● Model tuning

4. Tune the Whole Pipeline

5. Optimize all Parameters at Once
TensorFlow Playground

6. Include Feature Parameters

7. Include Feature Parameters

8. Choosing a Metric
● Balance long-term
and short-term goals
● Question underlying
assumptions
● Example from
Microsoft

9. Composite Metric
Example: Lifetime Value
clicks*wclicks + likes*wlikes + views*wviews

10. Choose Multiple Metrics
● Balance competing
metrics
● Explore entire result
space
Image from PhD Comics

11. Avoiding Overfitting

12. Get A Suggestion

13. Shuffle and Split Data

14. Train the Model

15. Test the Performance

16. Repeat
Shuffle Train Evaluate

17. Report An Observation

18. Repeat the Entire Process
Shuffle Train Evaluate

19. Optimization Methods

20. Hand Tuning
● Hand tuning is time
consuming and
expensive
● Algorithms can
quickly and cheaply
beat expert tuning

21. Grid Search Random Search Bayesian Optimization
Alternatives to Hand Tuning

22. No Grid Search
Hyper-
parameters
Model
Evaluations
2 100
3 1,000
4 10,000
5 100,000

23. No Random Search
● Theoretically more
effective than grid
search
● Large variance in
results
● No intelligence

24. Bayesian Optimization
● Explore/exploit
● Ideal for "expensive"
optimization
● No requirements on:
convex,
differentiable,
continuous

25. Alternatives to Bayesian Optimization
Genetic algorithms
Particle-based methods
Convex optimizers
Simulated annealing
To name a few...

26. Takeaways
●Optimize the entire pipeline
●Ensure generalization
●Use Bayesian optimization

27. Thank You!
blog.sigopt.com
sigopt.com/research