AE: A domain-agnostic platform for adaptive experimentation MLSE NeurIPS 2018 Systems for ML 論文読む会

1. AE: A domain-agnostic
platform for adaptive
experimentation
2019-02-08@ MLSE NuerIPS2018 Sys for ML

2. Yuta Kashino ( )
BakFoo, Inc. CEO
Astro Physics /Observational Cosmology
Zope / Python
Realtime Data Platform for Enterprise
PoC/Prototyping

3. AE: A domain-agnostic platform
for adaptive experimentation
Eytan Bakshy, Lili Dworkin, Brian Karrer, Konstantin Kashin, Benjamin Letham,
Ashwin Murthy, Shaun Singh
Facebook

4. AE:
• Google Vizier / Ray.Tune
ML + Python
• Vizier: http://bit.ly/2DZoh4I
• Ray.Tune: http://bit.ly/2DZ4D8U
• FB (OSS
• FB Learner Flow http://bit.ly/2DXHGD5 …
• sequential experiment: BO,

5. AE:
1.
1. DNN Web UI A/B / HHVM JIT
2. HHVM JIT: PHP C++/LLVM http://bit.ly/2DYOqQO
2. human-in-the-loop
3. A/B
4.

6. 1. : BO
• Bayesian Optimization
•
• Spearmint http://bit.ly/2E0pM2p
• BayesOpt http://bit.ly/2DZ1mGB
• MOE http://bit.ly/2DZOCiS
• Hyperopt http://bit.ly/2DYIGXx
• Optuna https://optuna.org/
• Ray.Tune http://bit.ly/2DZ4D8U
• SigOpt https://sigopt.com/
• Vizier http://bit.ly/2DZoh4I
• Kaggle
Hyperopt

7. 1.seqential expriment
• sequential expriment:
• box 1992 http://bit.ly/2E0qPiR
• Bayesian Optimization
Bayesian Optimization

8. 1. :BO
•
• Gaussian
Process(GP)
• A/B
• x: parameter config.
• x*: param.
• f:
• Expected Improvement(EI)
• EI
http://bit.ly/2GA7Opj
EI(x) =
∫
∞
−∞
max(0,y − y*)p(y|x)dy
y = f(x), y* = f(x*)

9. 1.Hyperopt
• BO : Tree Parzen Estimator (TPE)
• PFN http://bit.ly/2GtAm3s
•
•
• fmin

10. 1.AE: Core API

11. 2.AE: human-in-the-loop

12. 3.AE:
• Bayesian Optimization
• Constrained Bayesian Optimization with Noisy Experiments http://bit.ly/2E1pJTZ
• Expected Improvement
• botorch
• PyTorch/GPyTorch
•

13. 4.AE:
• Runner Batch
• Online/Offline
• MySQL
• AnalysisPlan Batch
• AnalysisPlan
• AnalysisPlan

14. AE: FB
• HHVM JIT
• PlanOut UI A/B
• DNN
•
• FB Video
• FB
http://bit.ly/2GwMAbI

15. • AE
Facebook

16. • AE Runner: Python shell
• Ray + Tune
• DNN Kubernetes
• Kubeflow + Seldon
• Polyaxon
• 2018 ML
http://bit.ly/2GwocH8
http://bit.ly/2GuCxEa

17. BakFoo, Inc.
• Software 2.0 http://bit.ly/2DZ4z96
• Sofware 2.0 = ML + Software 1.0

18. BakFoo, Inc.
• Software
• Kubernetes
• FaaS +
• FaaS: AWS Lambda, Google Functions …
• kubeflow polyaxon
• DB/
• Distributed Functions ( )
•

19. Questions
kashino@bakfoo.com
@yutakashino

20. BakFoo, Inc.
NHK NMAPS: +

21. BakFoo, Inc.
PyConJP 2015
Python

22. BakFoo, Inc.

23. BakFoo, Inc.
: SNS +