There is one thing you can count on with every model - model performance will deteriorate. During this session, Analytics Consultant Marsha Silverman shows you how you can monitor model performance to catch deterioration before it is too late. We cover: How to identify model deterioration When should a model be re-calibrated When should a model be redeveloped Watch the recording of this webinar here: https://www.experian.com/business-information/webinar-sip-and-solve-ep-13-signs-you-need-to-adjust-model

1. 1 © Experian 25/10/2019
Sip and Solve
Signs you need to adjust your
decision model
Featuring:
Marsha Silverman
Lead Analytic Consultant
Experian Commercial Decision Sciences

2. 2 © Experian Sip and Solve: Signs you need to adjust your decision model
What is a model?
An engine or mathematical formula that distinguishes the relationships between “random
information” and an event.
• Used to guide a business area to make decisions at key points in the customer life cycle.
• Helps give the likelihood of particular ‘outcome’ for individual accounts or transaction.
• Based on the historic behavior of similar types of accounts.
• Examples of use cases:
• Will an account go delinquent within the next 12 months?
• What is the likelihood a customer will respond to a marketing offer?
• Who is more likely to pay its delinquent debt?

3. 3 © Experian
K-S or Kolmogorov-Smirnov measures performance of classification models
• It is a measure of the degree of separation between the positive and negative distributions.
• In most classification models, the K-S will fall between 0 and 100, where the higher the value, the better the model is at separating
the positive from negative cases.
GINI is similar to KS and measures the area under the curve
Bad Capture rate measures how many “bads” are captured in the bottom percentage of the population
Cumulative%bads
Cumulative % goods
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
How do we know if a model “works”?

4. 4 © Experian Sip and Solve: Signs you need to adjust your decision model
Why do models deteriorate?
One of the chief assumptions in modelling is that the future will be a lot like the past.
However,
• Change is inevitable
• Change is continuous
• Businesses change how they conduct business
• There are changes in external factors such as the economy, politics, regulations, etc.
Therefore,
• Scorecards will degrade over time as a byproduct of other changes
• If changes are gradual, so, too, will be the process of degradation

5. 5 © Experian
Starting population – Accounts opened in Q1 2018
Jan – Mar 2018
Observation period
Jan – Mar 2019
12-month rolling performance window from opening
Performance definitions
Bad
Businesses with C/O or
91+ delinquency
Good
Businesses without Bad
statuses
End of performance period
Accounts would be retro-scored as of the observation point to determine how well the score
predicts the actual behavior experienced during the performance window.
Sip and Solve: Signs you need to adjust your decision model
Performing periodic score validations will show if a model deteriorates
• For example, if a model was developed using new accounts from Q1 2017, a validation may be
done on accounts from Q1 2018
How do we know if a model still “works”?

6. 6 © Experian Sip and Solve: Signs you need to adjust your decision model
Comparing current year (CY) to prior year (PY) demonstrates if deterioration has occurred
Experian performs annual score validations

7. 7 © Experian Sip and Solve: Signs you need to adjust your decision model
A population shift can mask the effectiveness of a model. Assessing the population stability index (PSI) can
be helpful to determine if the model still works but the population has shifted.
The Population Stability Index (PSI) quantifies shifts in score distributions for two populations. Each score
distribution is separated into buckets and the percent of records in each bucket is calculated.
The PSI is calculated as: PSI=Sum of { LN (percent/percent2)*(percent-percent2) } over all buckets.
Results are interpreted with the following parameters:
PSI > 0.25 – Significant change in population
PSI between 0.1 and 0.25 – Some minor changes
PSI<= 0.1 – No significant changes in the population
Population stability

8. 8 © Experian Sip and Solve: Signs you need to adjust your decision model
If a model has been found to have deterioration and/or there is a large shift in population running
through the model:
• Models may be recalibrated using the same predictors but with different co-efficients
• Models may be fully redeveloped on a new sample population
What should be done?

9. 9 © Experian 25/10/2019
Thank you!
Have questions or want to learn more about when to adjust your model?
• Send us a message in the “Chat” function or via bit.ly/sip-solve-chat
Upcoming Sip and Solves -
• Join the list to stay in the loop: bit.ly/sip-and-solve