This webinar, hosted by SigOpt co-founder and CEO Scott Clark, explains how advanced features can help you achieve your modeling goals. These features include metric definition and multimetric optimization, conditional parameters, and multitask optimization for long training cycles.
4. SigOpt. Confidential.
Your firewall
Training
Data
AI, ML, DL,
Simulation Model
Model Evaluation
or Backtest
Testing
Data
New
Configurations
Objective
Metric
Better
Results
EXPERIMENT INSIGHTS
Track, organize, analyze and
reproduce any model
ENTERPRISE PLATFORM
Built to fit any stack and scale
with your needs
OPTIMIZATION ENGINE
Explore and exploit with a
variety of techniques
RESTAPI
Configuration
Parameters or
Hyperparameters
Your data
and models
stay private
Iterative, automated optimization
Integrates
with any
modeling
stack
5. SigOpt. Confidential.
$300B+
in assets under management
Current SigOpt algorithmic
trading customers represent
$500B+
in market capitalization
Current SigOpt enterprise customers
across six industries represent
14. SigOpt. Confidential.
Benefits
Learn fast, fail fast
Give yourself the best chance at finding good use
cases while avoiding false negatives
Connect outputs to outcomes
Define, select and iterate on your metrics
with end-to-end evaluation
Find the global maximum
Early non-optimized decisions in the process limit
your ability to maximize performance
Boost productivity
Automate modeling tasks so modelers spend
more time applying their expertise
18. SigOpt. Confidential.
How it works: Multimetric optimization (with thresholds)
● Define two metrics instead of one
● Optimize against both metrics
automatically and simultaneously
● Set thresholds on each individual metric to
reflect business or modeling needs
● Compare a Pareto frontier of best model
configurations that balance these two
metrics
● Relevant docs
● Relevant blog post
19. SigOpt. Confidential.
Potential applications of multimetric optimization
Balance Competing Objectives Define and Select Metrics Connect Metrics to Outcomes
https://sigopt.com/blog/intro-to-multicriteria
-optimization/
https://sigopt.com/blog/multimetric-updates-
in-the-experiment-insights-dashboard/
https://sigopt.com/blog/metric-thresholds-a
-new-feature-to-supercharge-multimetric-
optimization/
20. Use Case: Balancing Speed & Accuracy in Deep Learning
Multimetric Use Case 1
● Category: Time Series
● Task: Sequence Classification
● Model: CNN
● Data: Diatom Images
● Analysis: Accuracy-Time Tradeoff
● Result: Similar accuracy, 33% the inference time
Multimetric Use Case 2
● Category: NLP
● Task: Sentiment Analysis
● Model: CNN
● Data: Rotten Tomatoes
● Analysis: Accuracy-Time Tradeoff
● Result: ~2% in accuracy versus 50% of training time
Learn more
https://devblogs.nvidia.com/sigopt-deep-learning-
hyperparameter-optimization/
21. SigOpt. Confidential.
Experiment Design for Sequence Classification
Data
● Diatom Images
● Source: UCR Time Series Classification
Model
● Convolutional Neural Network
● Source: Wang et al. (paper, code)
● Tensorflow via Keras
Metrics
● Inference Time
● Accuracy
HPO Methods (Implemented via SigOpt)
● Random Search
● Bayesian Optimization
Note: Experiment code available here
23. SigOpt. Confidential.
Result: Bayesian outperforms random search
● Both methods were executed
via the SigOpt API
● Bayesian optimization required
90% fewer training runs than
random search
● Bayesian optimization found
85.7% of the combined Pareto
frontier of optimal model
configurations—almost 6x as
many choices
10x random
26. SigOpt. Confidential.
How it works: Conditional parameters
Take into account the conditionality of
certain parameter types in the
optimization process
● Establish conditionality between
various parameters
● Use this conditionality to improve
the Bayesian optimization process
● Boost results from the hyper-
parameter optimization process
● Example: Architecture parameters
for deep learning models
● Example: Parameter types for SGD
variants (to the right)
● Relevant docs
27. Use Case: Effective and Efficient NLP Optimization
Use Case
● Category: NLP
● Task: Question Answering
● Model: MemN2N
● Data: bAbI
● Analysis: Performance benchmark
● Result: 4.84% gain, 30% the cost
Learn more
https://devblogs.nvidia.com/optimizing-end-to-end-memory-
networks-using-sigopt-gpus/
28. SigOpt. Confidential.
Design: Question answering data and memory networks
Data Model
Sources:
Facebook AI Research (FAIR) bAbI dataset: https://research.fb.com/downloads/babi/
Sukhbaatar et al.: https://arxiv.org/abs/1503.08895
31. SigOpt. Confidential.
Result: Highly cost efficient accuracy gains
Comparison across random search versus Bayesian optimization with conditionals
SigOpt is
18.5x as
efficient
33. SigOpt. Confidential.33
How it works: Multitask Optimization
Partial Full
● Introduce a variety of cheap
and expensive tasks in a
hyperparameter optimization
experiment
● Use cheaper tasks earlier
(explore) in the tuning process
to inform more expensive tasks
later (exploit)
● In the process, reduce the full
time required to tune an
expensive model
● Relevant docs
Sources:
Matthias Poloczek, Jialei Wang, Peter I. Frazier: https://arxiv.org/abs/1603.00389
Aaron Klein, Frank Hutter, et al.: https://arxiv.org/abs/1605.07079
34. SigOpt. Confidential.
How it works: Combine multitask with parallelization
Your firewall
New
Configurations
Objective
Metric
Better
Results
EXPERIMENT INSIGHTS
Track, organize, analyze and
reproduce any model
ENTERPRISE PLATFORM
Built to fit any stack and scale
with your needs
OPTIMIZATION ENGINE
Explore and exploit with a
variety of techniques
RESTAPI
Configuration
Parameters or
Hyperparameters
WorkerWorker
Worker Worker
35. Use Case: Image Classification on a Budget
Use Case
● Category: Computer Vision
● Task: Image Classification
● Model: CNN
● Data: Stanford Cars
● Analysis: Architecture Comparison
● Result: 2.4% accuracy gain for much cheaper model
Learn more
https://mlconf.com/blog/insights-for-building-high-performing-
image-classification-models/
41. SigOpt. Confidential.
Fine-tuning the smaller
network significantly
outperforms feature
extraction on a bigger
network
Results: Optimizing and tuning the full network outperforms
41
Multitask optimization
drives significant
performance gains
+3.92%
+1.58%
42. SigOpt. Confidential.42
Insight: Multitask efficiency at the hyperparameter level
Example: Learning rate accuracy and values by cost of task over time
Progression of observations over time Accuracy and value for each observation Parameter importance analysis
43. SigOpt. Confidential.
Insight: Parallelization further accelerates wall-clock time
43
928 total hours to optimize ResNet 18
220 observations per experiment
20 p2.xlarge AWS ec2 instances
45 hour actual wall-clock time
44. SigOpt. Confidential.
Implication: Fine-tuning significantly outperforms
Cost Breakdown for Multitask Optimization
Cost efficiency Feature Extractor ResNet 50 Fine-Tuning ResNet 18
Hours per training 4.08 4.2
Observations 220 220
Number of Runs 1 1
Total compute hours 898 924
Cost per GPU-hour $0.90 $0.90
% Improvement 1.58% 3.92%
Total compute cost $808 $832
cost ($) per %
improvement $509 $20
Similar Compute Cost
Fine-Tuning Significantly
More Efficient and Effective
Similar Wall-Clock Time
45. SigOpt. Confidential.
Implication: Multiple benefits from multitask
45
Tuning ResNet-18
Cost efficiency Multitask Bayesian Random
Hours per training 4.2 4.2 4.2
Observations 220 646 646
Number of Runs 1 1 20
Total compute hours 924 2,713 54,264
Cost per GPU-hour $0.90 $0.90 $0.90
Total compute cost $832 $2,442 $48,838
Time to optimize Multitask Bayesian Random
Total compute hours 924 2,713 54,264
# of Machines 20 20 20
Wall-clock time (hrs) 46 136 2,713
1.7% the cost of
random search to
achieve similar
performance
58x faster
wall-clock time
to optimize with
multitask versus
random search
46. SigOpt. Confidential.
Techniques
1. Metric definition: multimetric optimization
Read the blog here.
2. Model search: conditional parameters
Read the blog here.
3. Long training cycles: multitask optimization
Read the blog here.
47. SigOpt. Confidential.
Try our solution
Sign up at
sigopt.com/try-it
today.
Register with code: 1SFSPON25
https://sanfrancisco.theaisummit.com
September 25-26, 2019
Register with code: SIGOPT20
https://twimlcon.com/
October 1-2, 2019
Download eBook
https://twimlai.com/announcing-our-
ai-platforms-series-and-ebooks/