This document summarizes a presentation on problems organizations face when implementing CMMI and achieving high maturity levels. It discusses a case study of three organizations that achieved CMMI level 5 maturity. Through analysis of the case study and literature, common problems were identified in areas such as entry conditions, process definition, metrics definition, and tool setup. Recommendations are provided to address each problem, such as gradually maturing lower levels before pursuing high maturity and involving experts in defining meaningful metrics. The presentation concludes future research could evaluate the quality of CMMI implementations.

1. Portugal
Problems of CMMI® Implementations
Recommendations to Avoid them
and Achieve High Maturity
Isabel Lopes Margarido
PhD Student Researcher
Faculdade de Engenharia da Universidade do Porto
João Pascoal Faria
FEUP/INESC
Marco Vieira
FCTUC
Raul Moreira Vidal
FEUP
2013-10-18, Lisboa

2. © 2013 CMMI Portugal Conference Series – All Rights reserved.
Isabel Lopes Margarido, 18th of October 2013, Lisbon
agenda
introduction
methodology
problems and recommendations
conclusions
2

3. © 2013 CMMI Portugal Conference Series – All Rights reserved.
Isabel Lopes Margarido, 18th of October 2013, Lisbon
introduction
method
problems
conclusions
3

4. © 2013 CMMI Portugal Conference Series – All Rights reserved.
Isabel Lopes Margarido, 18th of October 2013, Lisbon
introduction
method
problems
conclusions
4

5. © 2013 CMMI Portugal Conference Series – All Rights reserved.
Isabel Lopes Margarido, 18th of October 2013, Lisbon
introduction
method
problems
conclusions
objectives
gain additional
knowledge about CMMI
5

6. introduction
method
problems
conclusions
© 2013 CMMI Portugal Conference Series – All Rights reserved.
Isabel Lopes Margarido, 18th of October 2013, Lisbon
objectives
understand problems
and difficulties
gain additional
knowledge about CMMI
6

7. introduction
method
problems
conclusions
© 2013 CMMI Portugal Conference Series – All Rights reserved.
Isabel Lopes Margarido, 18th of October 2013, Lisbon
objectives
learn solutions to avoid
them
understand problems
and difficulties
gain additional
knowledge about CMMI
7

8. introduction
method
problems
conclusions
typically organisations using CMMI improve performance
© 2013 CMMI Portugal Conference Series – All Rights reserved.
Isabel Lopes Margarido, 18th of October 2013, Lisbon
motivation
many programs failed in CMMI level 5 organisations
SEI was concerned with
high maturity
8

9. introduction
method
problems
conclusions
© 2013 CMMI Portugal Conference Series – All Rights reserved.
Isabel Lopes Margarido, 18th of October 2013, Lisbon
high maturity levels dependencies
dependent on lower maturity levels
maturity level (ML) 2 M&A


9

10. introduction
method
problems
conclusions
© 2013 CMMI Portugal Conference Series – All Rights reserved.
Isabel Lopes Margarido, 18th of October 2013, Lisbon
high maturity levels dependencies
dependent on lower maturity levels
maturity level (ML) 2 M&A
building blocks for ML4: QPM, OPP (PPM and PPB)

10

11. introduction
method
problems
conclusions
© 2013 CMMI Portugal Conference Series – All Rights reserved.
Isabel Lopes Margarido, 18th of October 2013, Lisbon
high maturity levels dependencies
dependent on lower maturity levels
maturity level (ML) 2 M&A
building blocks for ML4: QPM, OPP (PPM and PPB)
knowledge base for quantitative continuous improvement at
ML5 (CAR, OPM)
11

12. introduction
method
problems
conclusions
© 2013 CMMI Portugal Conference Series – All Rights reserved.
Isabel Lopes Margarido, 18th of October 2013, Lisbon
case study (published in QUATIC, Lopes Margarido et al., 2012)
3 organisations appraised at CMMI
Dev ML5
– 2 organisations, 1 business unit
real problems and difficulties from industry
– documents, tools, interviews
– how CMMI implementation was conducted and the
processes were defined
– what processes, metrics and tools were developed
– how people were actually using them
12

13. introduction
method
problems
conclusions
© 2013 CMMI Portugal Conference Series – All Rights reserved.
Isabel Lopes Margarido, 18th of October 2013, Lisbon
analysis
analysed an SEI report (McCurley and
Goldenson, 2010)
verified which problems were common
to the case study, literature and SEI survey
verified which of the recommendations were sustained by
literature and SEI survey
13

14. introduction
method
problems
conclusions
© 2013 CMMI Portugal Conference Series – All Rights reserved.
Isabel Lopes Margarido, 18th of October 2013, Lisbon
problems (P) and recommendations (R)
entry conditions
P1: avoid underestimation
R1: plan: maturing levels, analysing and understanding HML,
building and maturing PPB and PPM
P2: don’t begin the house by the roof
R2: ML 2 and 3 need to mature
P3: understand quantitative nature of level 4
R3: involve statistician
R4: six sigma
R5: review goals
14

15. introduction
method
problems
conclusions
© 2013 CMMI Portugal Conference Series – All Rights reserved.
Isabel Lopes Margarido, 18th of October 2013, Lisbon
process definition and implementation
P4: copied processes
R6: reflect organisation culture
R7: involve experts and process users
P5: multicultural environments
R8: share processes, lessons learnt
P6: impose processes
R8 and R9: goals specific for business units related to
organisation business goals
R10: indicators at different report levels
15

16. introduction
method
problems
conclusions
© 2013 CMMI Portugal Conference Series – All Rights reserved.
Isabel Lopes Margarido, 18th of October 2013, Lisbon
process definition and implementation
P7: dissemination problems
R11: commitment from entire organisation
R12: training contents, specialised training
R13: coaching and monitoring
P8: lack of institutionalisation
R14: top management set goals for gradual
institutionalisation, monitor and reward
R13 and R15: give time for metrics and processes to mature
16

17. introduction
method
problems
conclusions
© 2013 CMMI Portugal Conference Series – All Rights reserved.
Isabel Lopes Margarido, 18th of October 2013, Lisbon
metrics definition
P9: meaningless uncorrelated metrics
P10: metrics definition (collect and analyse data)
R16: use goal question metric or equivalent
R17: unambiguous, repeatable, understandable
R18: size metrics according with work product
R19: interpret in context
R20: variables normalisation
17

18. introduction
method
problems
conclusions
© 2013 CMMI Portugal Conference Series – All Rights reserved.
Isabel Lopes Margarido, 18th of October 2013, Lisbon
metrics definition
P11: first data uncorrelated
P12: metrics categorisation
P13: baselines not applicable
R21: several cycles
R22: let PPM and PPB become stable
R23: categorise data
R24: aggregate normalised data
18

19. introduction
method
problems
conclusions
© 2013 CMMI Portugal Conference Series – All Rights reserved.
Isabel Lopes Margarido, 18th of October 2013, Lisbon
metrics usage
P14: abusive elimination of outliers
R25: quarantine
R26: maintain data points that are unique but recurrent
P15: not all projects are measurable
R27: specific based measures normalised to get derived
measures; R14
19

20. introduction
method
problems
conclusions
© 2013 CMMI Portugal Conference Series – All Rights reserved.
Isabel Lopes Margarido, 18th of October 2013, Lisbon
metrics usage
P16: effort estimates
R28: use expert judgment when needed
R29: any related historical data
R30: iterative planning, real time sampling
P17: people behaviour
R12, R13 and R31: personal data not used to evaluate
people
20

21. introduction
method
problems
conclusions
© 2013 CMMI Portugal Conference Series – All Rights reserved.
Isabel Lopes Margarido, 18th of October 2013, Lisbon
tools setup
P18: tools setup
R32: give time for tool setup
R33: do not use data collected when defects affect metrics
P19: overhead
P20: tools requirements
R34: only collect necessary data
R35: automated and unperceptive data collection
R36: discipline people, change mentality
21

22. introduction
method
problems
conclusions
© 2013 CMMI Portugal Conference Series – All Rights reserved.
Isabel Lopes Margarido, 18th of October 2013, Lisbon
problems analysis
59% found in the case studies, literature and survey
several problems shared between the studied organisations
37,5% found in literature were found in the case studies
53,3% found in the survey were found in the case studies
 16,7% of those also found in literature
22

23. introduction
method
problems
conclusions
© 2013 CMMI Portugal Conference Series – All Rights reserved.
Isabel Lopes Margarido, 18th of October 2013, Lisbon
problems analysis
59% found in the case studies, literature and survey
several problems shared between the studied organisations
37,5% found in literature were found in the case studies
53,3% found in the survey were found in the case studies
 16,7% of those also found in literature
“the percentage of problems shared in more than one
organisation/source indicates that they can occur when
implementing HML, so organisations should be aware of
them.” (Lopes Margarido et al., 2013, submitted for
publication to SQP)
23

24. introduction
method
problems
conclusions
© 2013 CMMI Portugal Conference Series – All Rights reserved.
Isabel Lopes Margarido, 18th of October 2013, Lisbon
summary and future research
wide variety of implementation methods
variance of performance results
SCAMPI:
– evaluates a sample
– does not evaluate performance
future research
– framework to evaluate quality of implementation of
CMMI practices
– definition published in PROFES 2012 (Lopes Margarido
et al., 2012)
24

25. © 2013 CMMI Portugal Conference Series – All Rights reserved.
Isabel Lopes Margarido, 18th of October 2013, Lisbon
introduction
method
problems
conclusions
think about...
25

26. © 2013 CMMI Portugal Conference Series – All Rights reserved.
Isabel Lopes Margarido, 18th of October 2013, Lisbon
questions
http://paginas.fe.up.pt/~pro09003
partially sponsored by
26

27. © 2013 CMMI Portugal Conference Series – All Rights reserved.
Isabel Lopes Margarido, 18th of October 2013, Lisbon
references
• M. Campo. “Why Maturity Level 5?”, Crosstalk, January/February
2012. 15-18.
• C. Hollenbach and D. Smith. (2002) A portrait of a CMMI level 4
effort Systems Engineering. 52-61.
• P. Leeson, "Why the CMMI® does not work," presented at the
SEPG Europe, Prague, Czech Republic, 2009.
• R. Radice, "Statistical Process Control in Level 4 and Level 5
Software Organizations Worldwide," presented at the Software
Technology Conference, 2000.
• M. Schaeffer, "DoD Systems Engineering and CMMI," presented
at the CMMI Technology Conference and User Group, 2004.
27

28. © 2013 CMMI Portugal Conference Series – All Rights reserved.
Isabel Lopes Margarido, 18th of October 2013, Lisbon
acronyms
CMMI – Capability Maturity Model Integration
M&A – Measurement and Analysis
ML – Maturity Level
OPM – Organisational Performance Management
OPP – Organisational Process Performance
P – Problem
PPB – Process Performance Baselines
PPM – Process Performance Models
QPM – Quantitative Project Management
R - Recommendation
SEI – Software Engineering Institute
SQP – Software Quality Professional
28

29. © 2013 CMMI Portugal Conference Series – All Rights reserved.
Isabel Lopes Margarido, 18th of October 2013, Lisbon
images
http://api.ning.com/files/hpf*xOTebDs- F23o6FETZ3j*3sNiONWjfXjTJCzprPjU5bS1
WJoGgWBjMPIOiQkm3SbZ41ijncrJ4K2aT-6dM9QURwHK3led/Dissemination2.jpg -26-06-2010
http://blog.pmtech.com.br/wp-content/uploads/Square-Paradox.jpg – 29-04-2011
http://www.signsexpressshop.co.uk/prodpics/1103.gif – 29-04-2011
Benjamin Haas/Shutterstock, http://cynthiayildirim.posterous.com/how-can-we-measure-the-size-of-theuniverse – 29-04-2011
http://ryanstephensmarketing.com/blog/wp-content/uploads/2009/10/one_size_fits_all.JPG
http://evolvingwe.com/wp-content/uploads/2010/11/image3.png – 29-04-2011
http://www.screenhog.com/sketch/LightbulbIdea.jpg – 21-04-2010
http://igraduatedwhatnow.files.wordpress.com/2009/11/thank_you_small.jpg – 02-05-2010
http://www.articulate.com/rapid-elearning/wp-content/uploads/2008/08/summary-objectives450.gif –
adapted, 25-05-2011
http://www.braxtechconsulting.com/Portals/22771/images/cmmi_certificate.jpg.png - 10-04-2012
http://erikhatch.org/wp-content/uploads/2012/02/no_time_1_.jpg - 31-08-2012
http://blog.inktechnologies.com/wp-content/uploads/2012/06/Statistics.jpg - 31-08-2012
http://images.yourdictionary.com/images/definitions/lg/erase.jpg -31-08-2012
https://earlychildcare.files.wordpress.com/2009/09/child-misbehaving.jpg - 31-08-2012
http://www.thaiworldview.com/jpg/img024.jpg - 31-08-2012
http://image.shutterstock.com/display_pic_with_logo/5880/5880,1268065242,15/stock-photobackground-concept-wordcloud-illustration-of-scientific-method-research-glowing-light-48226942.jpg 02-09-2012
29