The importance of Parametric estimation, performance measurement and benchmarking.

1. Software Estimation and
Performance Measurement

2. Harold van Heeringen
Software Cost Engineer, Sogeti Nederland B.V.
Sizing, Estimating & Control
ISBSG president
NESMA board
COSMIC IAC, representing the Netherlands
Sogeti Stockholm
April 19 2013
Software Estimation and
Performance Measurement

3. 3
Overview
Software Projects
Software Project Estimation
Functional Size Measurement
Estimating & Performance Measurement
ISBSG repository
Trends in the market
Conclusions

4. 4
Software projects
Software project industry: low maturity
Low estimation maturity
No or little formal estimation processes
No or little use of historical data
Lots of schedule and cost overruns
Standish Chaos reports: Most projects fail or are at least
unsuccessful
Low customer satisfaction rates
In Europe: only slightly higher than the financial sector

5. 5
Software project estimation
Most of the projects are estimated by ‘experts’
Bottom up, task by task effort estimation
Usually very optimistic (>30%)
Experts estimate, but other people (juniors) do the job
Forgotten activities (e.g. testscript reviews)
No feedback loop with past projects: experts don’t learn from past estimates
and actuals
No scenario’s: duration, team size, etc.
Not objective, transparent, verifiable and repeatable
Not defendable!
‘Easy’ to push back by stakeholders
No risk assessment (distribution of the estimate)

6. 6
Software Cost engineering
Not a real profession yet
Consultant software metrics
Estimation officer
Bid specialist
Parametric estimates
Functional size measurement  size of the software
Productivity rates from historical data or industry data
Parametric estimation tools
Objective, repeatable, transparent and verifiable
Defendable!!
‘Impossible’ to push back by stakeholders
Risk assesment (distribution of the estimate)

7. 7
Functional Size Measurement
Function Point Analysis (NESMA, IFPUG or COSMIC)
Measure the functional user requirements – size in function points;
ISO standards – objective, verifiable, repeatable;
Strong relation between functional size and project effort needed;
More function points means more functionality (e.g. more value);
the user or client understands that more functionality means a higher price
What to do with the results?
Project effort/duration/cost estimation
Project benchmarking/performance measurement
Use in Request for Proposal management (answer price/FP questions)
What about historical data?
Company data (preferably for estimation)
Industry data (for benchmarking)

8. 8
Software Estimation
Measure the Size of the functional user requirements
Determine the right productivity figures, based on own history data
or industry project data
Use Parametric tools to estimate effort, duration, team size, quality
- non linear relations
Scenario analysis
- different durations
- different team size
- varying influence of non-functional requirements
- etcetera

9. 9
Software equation (Putnam)
Size/productivity
= Effort 1/3
* duration4/3
Effort
Duration
Plan A: 6 months, 4.500 hours
Plan B: 7 months, 2.400 hours

10. 10
Duration
Plan A
Duration: 6 months
Effort: 4.500 hours
Max. team size: 5,8 fte
MTTD: 1,764 days
Plan B
Duration: 7 months
Effort: 2.400 hours
Max. team size: 2,7 fte
MTTD: 2,816 days
Which duration have the experts in mind??
Effort(hours)Same project, different duration

11. 11
Short demo of QSM SLIM

12. 12
Cost of wrong estimates
Non-linear extra costs
-Planning errors
-team enlargement more expensive, not faster
-Extra management attention / overhead
-Stress: More defects, lower maintainability !!
Linear extra kosten
Extra hours will be used

13. 13
Sogeti SEC
Sizing, Estimating & Control
Certified (COSMIC) Function Point Analysts
Metrics consultants
Responsible for metrics part of a quotation.
Size: FPA/COSMIC
Estimation: SEER-SEM / QSM / Sogeti tool / ISBSG
Product: Methodical Estimation Report (scenario’s)
Pricing: EUR/FP
Quality: Defects/FP
Centers of Excellence:
MS.Net, Java, Oracle, mobile, drupal, sharepoint, BI, etc.

14. 14
Estimating & Performance Measurement
PLAN
Estimate
AdministrateEvaluate
Adjust &
Report
Size measurement: FPA
Historical data
Estimation tools
Finetune Estimation model
Analyse productivity,
Report productivity
Start: Estimate request
Start: Project completed
Results:
- Parametric Estimation
- Expert Estimation
Result:
-Management report,
-Adjusted model
Result:
-Growing project DB,
-Performance measurement
-Updated expert knowledge
ACT
CHECK DO
Start: Project start
Continuous data collection
• effort hours registration
• defect registration
• change measurement
• project characteristics
Result: Project data
Data collection and
administration
• Collect project data
• Measure size
• Benchmark the project
Start: Periodically
Expert Estimate

15. 15
Performance Measurement
Measure the performance of completed projects
- Size in function points delivered
- Defects found in various test phases
- Hours spent per main activity
- Compare actuals to plan
Derive the main metrics for the project, e.g.
- Productivity (hours/FP)
- Cost (EUR/FP)
- Quality (Defects/FP)
- Schedule / Budget slippage
Store and Benchmark the data – internal and external

16. 16
Historical data: ISBSG repositories
International Software Benchmarking Standards Group
Independent and not-for-profit
Members are non-profit organizations, like IFPUG and NESMA
Grows and exploits two repositories of software data:
New development projects and enhancements (> 6000 projects)
Maintenance and support (> 1000 applications)
Everybody can submit project data
DCQ on the site
Anonymous
Free benchmark report in return
Special reports, Practical Project Estimation book, Compendium
Portal to access the project data

17. 17
Trends in performance
PI vs Effective FP
100 150 200 250 300 350 400 450 500 550 600
Effective FP
5
10
15
20
25
PI
Java Totaal QSM Business 4GL ISBSG Avg. Line Style
Needed for management to control and manage
Needed to express to the market in proposals
Needed to make sure we keep improving

18. 18
Trend information

19. 19
Trends
Supplier Performance Measurement
- Reality check of supplier’s quotations based on industry data
- Agreement on performance improvements: hours/FP or EUR/FP
Request for Proposals
- Output based pricing
- Price per function point asked
- Often cheapest one selected – which is usually the wrong choice
Increasing professionalism in project estimation
- Basis of Estimate – Recommended practice by AACE for software services
- Tools, research and better functional sizing methods (e.g. COSMIC)
- More people understand the importance of parametric estimates

20. 20
Conclusions
Estimation & Performance Process is important!
Increasing maturity of software realization
More accurate estimates, based on history data  defendable
estimates;
‘Professional’ image communicated to client;
Benchmarking  learning organization  improvement;
Performance Measurement  manage by actual performance data
Data is needed when answering metrics questions in RFP / RFI
Sogeti – international collaboration is important
Shared Best Practices
Sogeti SEC wishes to help you when possible!

21. @haroldveendam
Harold.van.heeringen@sogeti.nl
Harold van Heeringen
Senior Consultant Software Metrics /Software Cost Engineer
Sogeti Sizing, Estimating & Control (SEC)
President ISBSG (International Software Benchmarking Standards Group (www.isbsg.org))
Board member NESMA (Netherlands Software Metrics Association (www.nesma.nl))
IAC member COSMIC (www.cosmicon.com)