What needs to be considered when doing process improvements in high maturity environments. An example of improving the efficiency of code reviews.

1. Portugal
Process Improvement in High Maturity
Environments
Improving the Efficiency of Code Reviews
André Ferreira
Software Quality Assurance Engineer
Critical Software S.A.
2013-10-18

2. Outline
© 2013 CMMI Portugal Conference Series – All Rights reserved.
1. SPI success - evaluation and measurement
2. CMMI High Maturity Processes Areas - a framework to drive
SPI initiatives
3. SPI initiative at Critical Software – Improving the efficiency of
Code Reviews
4. Recommendations/guidelines to improve the probability of
success of SPI initiatives.
2

3. SPI success: reports on SPI Success.
Medium
Improvement
# Data
points
Lowest
Improvement
Highest
Improvement
Cost
© 2013 CMMI Portugal Conference Series – All Rights reserved.
Performance
category
34%
29
3%
87%
Schedule
50%
22
2%
95%
Productivity
61%
20
11%
329%
Quality
48%
34
2%
132%
Customer
Satisfaction
14%
7
-4%
55%
Return on
Investment
[4.0:1]
22
[1.7:1]
[27.7:1]
Performance Results of CMMI-Based Process Improvement –Technical Report
CMU/SEI-2006-TR-004 ESC-TR-2006-004
3

4. SPI success: evaluation of SPI Success - I
© 2013 CMMI Portugal Conference Series – All Rights reserved.
Evaluation and validity of SPI success is questionable.
Hard to establish a clear relation of SPI initiative and SPI
results. Results are not questionable by themselves, but a clear
understanding of how success relates to SPI is unclear, mainly as result of:
1.
2.
3.
4.
5.
Confounding factors – not considering relevant factors, validity threat
Incomplete context descriptions – context of SPI not fully described
Evaluation validity – threats to validity not discussed
Measurement validity – measurement definition inconsistencies
Measurement scope – missing the big picture
Unterkalmsteiner et.al. (2011). Evaluation and Measurement of Software Process Improvement A Systematic Literature Review. IEEE Transactions on Software Engineering
4

5. © 2013 CMMI Portugal Conference Series – All Rights reserved.
SPI success: evaluation of SPI Success - I
Causation
established
Y(SPI sucess) = f ( f1, f2, f,3)
5

6. © 2013 CMMI Portugal Conference Series – All Rights reserved.
SPI success: evaluation of SPI Success - I
Causation
established
Presence of
confounding factors
Y(SPI sucess) = f ( f1, f2, f,3)
Y’(SPI sucess) = f’ (?, ?, f1, f2, f3, ?)
6

7. SPI success: evaluation of SPI Success - II
Evaluation strategies for SPI initiatives:
© 2013 CMMI Portugal Conference Series – All Rights reserved.
Name
Percentage of studies
Pre-post Comparison
49%
Statistical Analysis
15%
Pre-post Comparisons & Survey
7%
Statistical Process Control
5%
Coat benefit analysis
3%
Not identified
14%
Unterkalmsteiner et.al. (2011). Evaluation and Measurement of Software Process Improvement A Systematic Literature Review. IEEE Transactions on Software Engineeringl
7

8. Process
Management
Project
Management
© 2013 CMMI Portugal Conference Series – All Rights reserved.
CMMI High Maturity Framework Overview
Support
BO
OPM
PPBs
Project QPPOs
OPP
QPM
OIP
QPPOs
CAR
Defects
Performance
Issues
Corrective Actions
8

9. © 2013 CMMI Portugal Conference Series – All Rights reserved.
SPI Initiative
Improving the Efficiency of
Code Reviews
9

10. SPI Initiative – Opportunity Identification
OPM.SP 1.1/OPM.SP1.2
© 2013 CMMI Portugal Conference Series – All Rights reserved.
1. Annual Review Business Objectives


Business objective: Improve organizational profit margins
QPPOs: Improve project profit margins – do the
same/better for less
2. Feedback from projects
 Issues with review procedure – cumbersome - to much
time managing review records.
 Lead to establish a performance baseline of total
appraisal cost for code reviews.
10

11. SPI Initiative – Opportunity Identification
© 2013 CMMI Portugal Conference Series – All Rights reserved.
OPM.SP1.2/OPM.SP 1.3
 Analysis of performance data
 Feedback from projects was confirmed to be an
issue – statistical analysis of a control chart.
 Code review is a - fat process!
 Opportunity identified:
 Decrease appraisal cost of performing code
review procedure by:
 Improving efficiency (less effort) but control
impact on effectiveness
11

12. SPI Initiative – Potential Improvement I
OPM.SP 1.3
High
Cost
© 2013 CMMI Portugal Conference Series – All Rights reserved.
Cost of Quality (CoQ) Management model
Total cost of
quality
(I) Internal
and (E)
External
failure costs
(A) Appraisal and
(P) Preventive
costs
Low
100%
defective
Quality
100%
good
12

13. SPI Initiative – Potential Improvement I
OPM.SP 1.3
High
Cost
© 2013 CMMI Portugal Conference Series – All Rights reserved.
Cost of Quality (CoQ) Management model
Total cost of
quality
(I) Internal
and (E)
External
failure costs
(A) Appraisal and
(P) Preventive
costs
Low
100%
defective
Quality
100%
good
13

14. SPI Initiative – Potential Improvement I
OPM.SP 1.3
Process Management
© 2013 CMMI Portugal Conference Series – All Rights reserved.
Rationale of impact on QPPOs
EBT
BO
Project
Gross
Margin
>= Target
QPPOs
Processes
Verificati
on/ Code
reviews
Rework
14

15. SPI Initiative – Potential Improvement I
OPM.SP 1.3
Process Management
© 2013 CMMI Portugal Conference Series – All Rights reserved.
Rationale of impact on QPPOs
BO
Project
Gross
Margin
>= Target
QPPOs
Processes
Decrease on overall CoQ will
have a positive impact in the
ability of projects to meet
Project Gross Margin (QPPO)
EBT
Verificati
on/ Code
reviews
Rework
15

16. SPI Initiative – Potential Improvement II
OPM.SP 1.3
High
Cost
© 2013 CMMI Portugal Conference Series – All Rights reserved.
Hypothesis1: Less effort spent performing code reviews will
contribute to a reduction in total appraisal cost.
Total cost of
quality
Internal and
external failure
costs
Appraisal and
preventive
costs
Low
100%
defective
Quality
100%
good
16

17. SPI Initiative – Potential Improvement II
OPM.SP 1.3
High
Cost
© 2013 CMMI Portugal Conference Series – All Rights reserved.
Hypothesis1: Less effort spent performing code reviews will
contribute to a reduction in total appraisal cost.
Total cost of
quality
Internal and
external failure
costs
Appraisal and
preventive
costs
Low
100%
defective
Quality
100%
good
17

18. SPI Initiative – Potential Improvement II
OPM.SP 1.3
High
Cost
© 2013 CMMI Portugal Conference Series – All Rights reserved.
Hypothesis1: Less effort spent performing code reviews will
contribute to a reduction in total appraisal cost.
Total cost of
quality
Internal and
external failure
costs
Appraisal and
preventive
costs
Low
100%
defective
Quality
100%
good
18

19. SPI Initiative – Potential Improvement II
OPM.SP 1.3
© 2013 CMMI Portugal Conference Series – All Rights reserved.
Hypothesis1: Less effort spent performing each code review will
contribute to a reduction in total appraisal cost
 Rationale of estimation of potential benefit
 (1) Assuming a 10x decrease on managing
review records -> x hours saved per review
 (2) y number of reviews performed each year
Total benefit (TH1)
 TH1(hours) = x*y
 TH1(€) = ~4K€/year
19

20. SPI Initiative – Potential Improvement III
OPM.SP 1.3
© 2013 CMMI Portugal Conference Series – All Rights reserved.
Hypothesis 2: Less effort per code review can be used to
perform additional code reviews (re-invest in Appraisal Cost)


Motivate an increase of # code reviews in projects with little
investment in code reviews
Motivate re-investment of effort saved to perform additional
code reviews (increase code coverage)
...with the final objective of decrease Total Cost of Quality by
decreasing considerably the Internal and External Failure
costs.
20

21. SPI Initiative – Potential Improvement III
OPM.SP 1.3
High
Cost
© 2013 CMMI Portugal Conference Series – All Rights reserved.
Hypothesis 2: Less effort per code review can be used to
perform additional code reviews (Increase Appraisal Costs)
Total cost of
quality
Internal and
external failure
costs
Appraisal and
preventive
costs
Low
100%
defective
Quality
100%
good
21

22. SPI Initiative – Potential Improvement III
OPM.SP 1.3
High
Cost
© 2013 CMMI Portugal Conference Series – All Rights reserved.
Hypothesis 2: Less effort per code review can be used to
perform additional code reviews (Increase Appraisal Costs)
Total cost of
quality
Internal and
external failure
costs
Appraisal and
preventive
costs
Low
100%
defective
Quality
100%
good
22

23. SPI Initiative – Potential Improvement III
OPM.SP 1.3
High
Cost
© 2013 CMMI Portugal Conference Series – All Rights reserved.
Hypothesis 2: Less effort per code review can be used to
perform additional code reviews (Increase Appraisal Costs)
Total cost of
quality
Internal and
external failure
costs
Appraisal and
preventive
costs
Low
100%
defective
Quality
100%
good
23

24. SPI Initiative – Potential Improvement III
OPM.SP 1.3
 Estimation of potential benefit
© 2013 CMMI Portugal Conference Series – All Rights reserved.



Based on performance baselines it takes 5x more to fix a
defect at ST when compared to code reviews
Projects who invest in code reviews, on average, have less
defects/LOC found in ST (corroborates the model) with a
typical distribution of [74%/24%] distribution of code reviews
VS. ST defects
Projects who invest less in code reviews have [5%/94%]
defect distribution
In the scenario of these projects with lower investment in code review
shifting to a bigger investment in code reviews, having an approximation
to a ratio of [74%/24/] we estimated potential savings of 70K€ for a year
timeframe.
TH2(€) = 70K€
24

25. SPI Initiative – ROI
© 2013 CMMI Portugal Conference Series – All Rights reserved.
OPM.SP 1.3
 Estimation of potential benefit
 Hypothesis 1
 Tool license cost: 3.6K
 Management of SPI initiative:
Deployment + Training: 3K€
 ROI [ 0.4:1]
 Hypothesis 2
 Tool license cost: 3.6K
 Management of SPI initiative:
Deployment + Training: 3K€
 ROI [ 9.6:1]
25

26. SPI Initiative – Elicit Improvements
© 2013 CMMI Portugal Conference Series – All Rights reserved.
OPM.SP2.1/SP2.2
 A target was given to identify approaches that could
decrease in 10x the time to manage code review
records
 Several tools were identified and analyzed in a DAR
initiative – ‘smoke test’.
 Crucible came as the best option as it enabled to set-up
a review in less than 10 minutes among other relevant
factors related to infrastructure.
26

27. SPI Initiative – Pilot
© 2013 CMMI Portugal Conference Series – All Rights reserved.
OPM.SP2.3/SP2.4
 Criteria of success in supporting H1.
 Decrease review setup/manage records is
consistent below 10m/code review
 Impact on defect detection ability (def/kloc) is
non negative
 Pilot lasted 3 months and involved 5 project and a
workshop with SPA area members.
27

28. SPI Initiative – Pilot Results
OPM.SP2.4
© 2013 CMMI Portugal Conference Series – All Rights reserved.
 Performance baseline with pilot results was created.

Review setup/manage records was under 10m
resulting in a 12x decrease on overall effort – change
in performance tested with statistical
significance.

Defect defection rate has not affected:
when compared to existing performance baseline
(44 defect/KLOC)
Overall, Crucible supported H1 and therefore a base to
approach H2 with a higher degree of confidence.
28

29. SPI Initiative – Deployment
OPM.SP3.1/SP3.2/SP3.3
© 2013 CMMI Portugal Conference Series – All Rights reserved.
 Phase currently in progress
– Evaluation of ROI will be over a 1 year period
– Monitoring cost of quality
– For the second year end beyond the NPV will
replace ROI for Cost/Benefit analysis of SPI
initiative
29

30. SPI Recommendations
© 2013 CMMI Portugal Conference Series – All Rights reserved.
Seek to establish your governance model (GM)



Use/create a GM that supports the rationale of your
network of Objectives vs. Processes
Start small, measure the least as possible, focus on
making the model reliable.
Only with a minimal statistical confidence on cause and
effect you are able to estimate ROI of SPI
Y(Repeatable SPI sucess) = f (f1,f2,f3)
Y’(Risky SPI initiative) = f‘(?, ?, f1, f2, ?)
30

31. © 2013 CMMI Portugal Conference Series – All Rights reserved.
You can contact me at:
al-ferreira@criticalsoftware.com
or
csw-quality@criticalsoftware.com
Questions?
31