Presentation slide deck in Odin conference in Oslo September 19 2018.

1. Applying Machine Learning
to Product Risk Prediction
@ Sparebank1
Odin 2018
Minh Nguyen

2. Agenda
►Motivation
►Problem definition
►Implementation
►Lesson learned & Take-aways
2

3. 3
Terminologies
Source: www.newgenapps.com/blog/machine-learning-vs-predictive-analytics

4. Motivation
► Machine Learning (ML) has been widely adopted in most industries.
► But not fully exploited in Software industry, especially Development&Test.
► Huge amount of data collected and available in SDLC.
► Significant correlation between the attributes of Change and Risk.
► Frequently used Risk-based test technique can be time consuming,
subjective assessments and limited by human memory.
4
To build ML models to predict Product Risk with real training
dataset from a development team in SB1

5. Problem: Risk Prediction
5
User
Change
Risk
Score
ML Prediction Model
Training
Dataset
Manual Risk
Assessment
Optionally
combined with
Test
Operation
Defect
Requirement
Development
Improved Test Planning
and Prioritization

6. 6
Data sources
Development
Requirement /
Change
Test
Operation
Statistics
Defect
Zephyr
Remedy/
Splunk
Confluence
Jira
Jira

7. Dataset for Supervised Training
Data extracted from Jira on Change:
► Summary
► Description
► Severity
► Priority
► Component
Complexity
ML-model
Derived from:
Defects
Incidents
Calculation
Risk Score
Label

8. ML-Classification
8
Complexity
Score
(3-High, 2-Normal
and 1-Low)
Natural Language Processing
Naive Bayes ML algoritme
Summary
&
Description
249 observations in training dataset:
Bag of Words
(incl. 1420 words)
1. Cleaning
2. Stop-words
3. Stemming

9. Risk Score Calculation
9
DEFECT
For a given
Change found
in test phase
INCIDENT
for a given
Component
reported from Prod
i = 1 &
y in (critical, major,
normal, trivial)
Max #
of defects
Defect(i) * Severity(i) * Weight(y)
Incident(i) * Severity(i) * Weight(y)
i = 1 &
y in (critical, major,
normal, trivial)
Max #
of incidents
RISK SCORE
For a given
Change

10. 10
Risk Prediction with Regression
Change
Multiple Linear Regression
Risk Score
123 observations in labeled training dataset
ML-Classified
ML-Predicted

11. Measure success of prediction
How to demonstrate improvement of following metrics:
1. Test Effectiveness = No. of defect found / No. of Test Execution
2. MTBI – Mean time between Incident for a given Component
3. Test Productivity = Number of testing hours per Change
11

12. Choice of ML Technology
12
• Easy-to-use and quick enabler.
• Simulation with ML-algorithms.
• Lisence based – expensive.
• Cloud-based – «GDPR».
• Development (convenient packages).
• Flexibility and better understanding.
• Freeware.
• Local data governance.

13. Lesson learned
13
Problem
Definition
Data
Collection
Data
Pre-processing
Training
& Test
Deployment
& Learning
 Time-consuming.
 Data access.
 Data quality.
 Missing values.
 Inconsistent linkages.
 NLP – Norwegian.
 Out-of-the box
• ML-algorithms
• Performance metrics
 Limited dataset.
 Unprecise labeling.
 Seamlessly integrated
to current process.
 Self-learning cycle.

14. 14
Inspiration…
Problem
Prediction/
Classification
Answer to a
specific question
Clustering
Understand &
Innsight
Strategy
Explore
Exploit
Reinforcement
Unsupervised
Supervised
ML-
algorithms
Labeled
past data
Unlabeled
past data
Possibilities,
Reward

15. Thank you for listening
15
Minh Nguyen
+47 982 28 460
minh.nguyen@sogeti.no
www.linkedin.com/in/minhng67/