1.
1CONFIDENTIAL INFORMATION OF NEXOSIS.
Automating Time Series Model
Selection with Decision Theory
Ryan West
MLconf Atlanta 2017

2.
2CONFIDENTIAL INFORMATION OF NEXOSIS.
K-Folds
Cross-Validation
Rolling Time Series
Cross-Validation
https://en.wikipedia.org/wiki/Cross-validation_(statistics)
https://robjhyndman.com/hyndsight/tscv/

3.
3CONFIDENTIAL INFORMATION OF NEXOSIS.
Problem Visualized Total Test
Set
Model 1
Forecast
Model 2
Forecast
Model M
Forecast
Test Set
Subset 1
Test Set
Subset
N
Test Set
Subset 1
Test Set
Subset
N
Test Set
Subset 1
Test Set
Subset
N
.......
....... ..............
Minimize:
Maximize:

4.
4CONFIDENTIAL INFORMATION OF NEXOSIS.
Formulation
o minimax(x1, x2, …, xM)
o xi = a variable of N possible values
o error metric calculated with N test sets and forecast of model i
min s
subject to:
s ≥ x1
s ≥ x2
…
s ≥ xM
Equivalent to:

5.
5CONFIDENTIAL INFORMATION OF NEXOSIS.
Alternative Problem Total Test
Set
Test Set
Subset 1
Test Set
Subset 2
Test Set
Subset N
Model 1
Forecast
Model M
Forecast
Model 1
Forecast
Model M
Forecast
Model 1
Forecast
Model M
Forecast
.......
.....................
Maximize:
Minimize:

6.
6CONFIDENTIAL INFORMATION OF NEXOSIS.
Alternative Formulation
o maximin(x1, x2, …, xN)
o xi = a variable of M possible values
o error metric calculated with the forecasts of M models and test set i
Equivalent to: max s
subject to:
s ≤ x1
s ≤ x2
…
s ≤ xN

7.
7CONFIDENTIAL INFORMATION OF NEXOSIS.
Experiment
o 856 time series of daily retail sales data
o 7 exogenous variables per time series
o e.g. promotions, holidays, indicator variables of store open or closed
o 38 possible models
o Testing forecast accuracy of different model selection techniques

8.
8CONFIDENTIAL INFORMATION OF NEXOSIS.
Model Selection Techniques
o Selection using ensembling
o Single test set for model selection
o Additional holdout set
o Selection based on maximin of error metric
o Multiple test sets for model selection
o Additional holdout set
o Selection based on minimizing error metric
o Single test set for model selection
o Additional holdout set

9.
9CONFIDENTIAL INFORMATION OF NEXOSIS.
Partial Autocorrelation
o Strongly seasonal time series

10.
10CONFIDENTIAL INFORMATION OF NEXOSIS.
Error Metric Visualization (MAE)

11.
11CONFIDENTIAL INFORMATION OF NEXOSIS.
Error Metric Visualization (RMSE)

12.
12CONFIDENTIAL INFORMATION OF NEXOSIS.
Error Metric Visualization (RMSPE)

13.
13CONFIDENTIAL INFORMATION OF NEXOSIS.
Error Metric Visualization (sMAPE)

14.
14CONFIDENTIAL INFORMATION OF NEXOSIS.
Forecast Accuracy
Average RMSPE*
on Holdout Set
Model Selection Technique Feature
Engineering
0.382 Minimizing RMSPE on test set No
0.223 Naïve median weekly seasonal
predictions
No
0.215 Maximin of RMSPE on test set subsets Yes
0.204 Minimizing RMSPE on test set Yes
0.191 Ensemble Averaging Yes
*RMSPE = Root Mean Squared Percentage Error

15.
15CONFIDENTIAL INFORMATION OF NEXOSIS.
Thank You!