▸ Machine Learning / Deep Learning models require to set the value of many hyperparameters ▸ Common examples: regularization coefficients, dropout rate, or number of neurons per layer in a Neural Network ▸ Instead of relying on some "expert advice", this presentation shows how to automatically find optimal hyperparameters ▸ Exhaustive Search, Monte Carlo Search, Bayesian Optimization, and Evolutionary Algorithms are explained with concrete examples

2. Francesco Casalegno – Hyperparameter Optimization
2

3. Francesco Casalegno – Hyperparameter Optimization
●
○
→
→
→
○
→ …
→
→ …
●
○
○
3

4. Francesco Casalegno – Hyperparameter Optimization
●
…
→
○
○
○
●
○
x F
○ F →
○ F(x) 4

5. 5

6. Francesco Casalegno – Hyperparameter Optimization
●
●
○ →
○ →
● GridSearchCV
6

7. Francesco Casalegno – Hyperparameter Optimization
●
●
→
●
●
●
→
●
→
7

8. 8

9. Francesco Casalegno – Hyperparameter Optimization
● O(N - 1/2
)
N
○
●
● RandomizedSearchCV
9

10. Francesco Casalegno – Hyperparameter Optimization
●
●
●
●
●
●
→
10

11. 11

12. Francesco Casalegno – Hyperparameter Optimization
●
○ F(x) F(x)
○
○ F(x)
○
●
○ F(x)
○ F(x)
○ …
○ … →
12

13. Francesco Casalegno – Hyperparameter Optimization
● F(x)
○ x = [x1
… xN
], [F(x1
) … F(xN
)]
○
■
■
→
13

14. Francesco Casalegno – Hyperparameter Optimization
●
○ x F(x) →
○ →
●
○
○
○
○ … 14

15. Francesco Casalegno – Hyperparameter Optimization
● BayesSearchCV
●
15

16. Francesco Casalegno – Hyperparameter Optimization
●
●
●
→
● σ ρ
●
●
→
→
→
16

17. 17

18. Francesco Casalegno – Hyperparameter Optimization
●
M x1
… xM
F(x)
Prob x is selected F(x)
child = 1 … M
■ x
■
18

19. Francesco Casalegno – Hyperparameter Optimization
● TPOTClassifier TPOTRegressor tpot
→ eaMuPlusLambda() DEAP
● tpot
19

20. Francesco Casalegno – Hyperparameter Optimization
●
●
●
→
●
●
●
→
●
→
→
● M
20

21. Francesco Casalegno – Hyperparameter Optimization
●
●
●
●
●
●
21