1. A Guide to Effective Benchmarking
of Applications Development
October 30, 2012
4:00 PM - 5:30 PM
Slide: 1
12/27/2012
Webinar Sponsored by Computer Aid, Inc.

2. Pam Morris
CEO
Total Metrics
Pam.Morris@totalmetrics.com
Hosted by:
Michael Milutis
Director of Marketing
Computer Aid, Inc. (CAI)
Michael_Milutis@compaid.com
Slide: 2
12/27/2012
Webinar Sponsored by Computer Aid, Inc.

3. About Total Metrics
• “Supporting organizations in their quest to optimize the cost effectiveness and efficiency of their
software delivery”
• Internationally based - Certified IFPUG and COSMIC Function Point and Metrics Consultants
and Trainers – Europe, UK, USA, Australia
• Provide metrics related tools, procedures, consulting and training - Thought leader in functional
size measurement and developers of :
• SCOPE Project Sizing Software™ – Function point Counting, Portfolio Metrics
Management, Industry Benchmarking Analytics.
• FP Outline™ Wizard for Estimating Application and Project
Functional Size
• For organizations serious about measuring Functional Size
• Multi-lingual Korean, Chinese, Japanese, Portuguese,
Spanish , German, Dutch, Italian, French, English etc.
• NEW Release FREEdownloads see WWW.totalmetrics.com
Slide: 3
12/27/2012
Webinar Sponsored by Computer Aid, Inc.

4. CAI Achieves IT Operational Excellence
Slide: 4
12/27/2012
Webinar Sponsored by Computer Aid, Inc.

5. PDU Credits Available for this Webinar
• The PMI has accredited this webinar with PDUs
• You will be eligible to receive 1.0 PDU credits
• Your PDU email will be sent to you within 24 hours
Slide: 5
12/27/2012
Webinar Sponsored by Computer Aid, Inc.

6. Online Webinar Recordings
NOW AVAILABLE
• Anytime Access
• Hundreds of Topics
Visit:
www.ITMPI.org/library
Slide: 6
12/27/2012
Webinar Sponsored by Computer Aid, Inc.

7. Enjoy the benefits of ITMPI Membership
JOIN TODAY!
• UNLIMITED Free Webinar Recordings
• UNLIMITED Free PDU Credits
• Hundreds of Topics
Visit:
www.ITMPI.org/subscribe
Slide: 7
12/27/2012
Webinar Sponsored by Computer Aid, Inc.

8. Overview
• What is a Benchmark?
• Why Benchmark?
• What are the Risks?
• Ensuring Success - Terms of Reference
Slide: 8
12/27/2012
Webinar Sponsored by Computer Aid, Inc.

9. Benchmarking Definitions
“Benchmarking is the process of continuously comparing and measuring an
organisation with business leaders anywhere in the world to gain information
which will help the organisation take action to improve its performance.”
American Productivity and Quality Centre
Benchmarking Management Guide
Cambridge, Mass., 1993, Productivity Press
“Activity of comparing objects of interest to each other or against a benchmark
to evaluate characteristic(s)”
ISO /IEC FCD 29155-1, Software and Systems Engineering – IT Project Performance Benchmarking
Framework – Part 1: Concepts & Definitions
“Benchmarking is the process of determining a metric baseline for an
organisational or functional unit for the purpose of comparison”
Total Metrics – Glossary of Terms
Slide: 9
12/27/2012
Webinar Sponsored by Computer Aid, Inc.

10. Types of Benchmarks
• Internal Benchmark
– Benchmark against a group of commonly usable and authorised
benchmarks based upon data sourced from within the target
organisation.
– Procedure of comparing and measuring one‟s own performance
within an organisation
• External Benchmark
– Benchmark against a group of commonly usable benchmarks based
upon data sourced from external organisations and issued by
external repository providers.
– Procedure of comparing and measuring an organisation with other
organisations
Slide: 10
12/27/2012
Webinar Sponsored by Computer Aid, Inc.

11. Why Benchmark?
Answer: To be able to make informed decisions!
– Are we more or less effective than our competitors? (Should we offshore?)
– Are we more or less effective than industry? (Should we upgrade our skills?
Technology? Or should we outsource?)
– Is our IT consuming more money than it should ? (Should we change
suppliers?)
– Are we improving at the rate required by our contract? (Do we need to
prepare for likely penalties?)
– Are we achieving ROI on investments in tools and technology? (Do we need
to review our IT investment strategy?)
– Are all organization units performing at the same level? (Which units do we
keep or cut?)
Slide: 11
12/27/2012
Webinar Sponsored by Computer Aid, Inc.

12. What are the Risks?
Answer: Incorrect strategic decisions based on invalid
numbers!
» Critical to get the Benchmarking Right!
Slide: 12
12/27/2012
Webinar Sponsored by Computer Aid, Inc.

13. How do we to ensure Valid Benchmark Results?
Answer: Establish rigorous ‘Terms of Reference’ and
get stakeholder agreement on the Benchmark’s:
• Strategic Intent
• Type
• Performance Metrics
• Standards for Measures
• Scope
• Frequency
• Peers for Comparison
• Report Requirements
• Dispute Resolution Process
Slide: 13
12/27/2012
Webinar Sponsored by Computer Aid, Inc.

14. Strategic Intent
Common Understanding: Why are we Benchmarking?
• Strategy is determined by:
– What results are required
– When - within what time frame
– How results will be used
Slide: 14
12/27/2012
Webinar Sponsored by Computer Aid, Inc.

15. Common purposes for Benchmarks - External
• Client initiated Benchmarks
– Establish current IT efficiency and effectiveness compared to Industry
norms – should we look at outsourcing?
– Evaluate supplier performance against Industry standards – should we
change suppliers?
– Set competitive range for performance baseline and targets – how can we
monitor that we are getting value for money from our suppliers?
• Supplier initiated Benchmarks
– Establish competitive position for productivity, pricing and service levels –
how can we ensure that we are in a competitive bidding position?
Slide: 15
12/27/2012
Webinar Sponsored by Computer Aid, Inc.

16. Common purposes for Benchmarks - Internal
• In-house initiated Benchmarks
– Which operational units are the most efficient? - Should we review
where work is allocated ?
– Which operational have “Best Practice” processes? - How can we
implement practices in all units?
– Did we achieve the planned benefits of implementing new tools,
techniques or technologies? - Can we justify the expenditure?
– How consistent is our development productivity rate? - Can we be
confident in using it to estimate planned projects?
– Is our M&S support productivity improving? - Are we doing more
with less staff than Industry norms?
Slide: 16
12/27/2012
Webinar Sponsored by Computer Aid, Inc.

17. Example : M&S Support efficiency improvement
Slide: 17
12/27/2012
Webinar Sponsored by Computer Aid, Inc.

18. Comparisons – Internal? External? Both?
– Internal
• To demonstrate improvement trends over time for the
organisations internal processes, or
– External
• To compare internal results with external independent
organisational units, or Industry as a whole. Eg. ISBSG
Slide: 18
12/27/2012
Webinar Sponsored by Computer Aid, Inc.

19. What will we measure?
• Cost-effectiveness of the process – are we getting „value‟ for
money invested?
• Efficiency of the process – how „productively‟ is our software
being developed or maintained?
• Speed of Delivery – how „quickly‟ can we deliver software
product or „solve a problem‟?
• Quality of the product – how „good‟ is the software product or
service we deliver?
• Quality of the process – how much time and money was wasted
in „rework‟?
• Customer Satisfaction – how well does the delivery of our
software products and related services meet and or exceed their
customer‟s expectations?
Slide: 19
12/27/2012
Webinar Sponsored by Computer Aid, Inc.

20. Comparable Measures?
• Establish Rules for Comparison:
• Units of measurement – Day? Hrs?
• Measurement scope – which processes , whose effort, which
FPs?
• Data accuracy
• Validation requirements
Slide: 20
12/27/2012
Webinar Sponsored by Computer Aid, Inc.

21. Industry Data Collection Standards
• Glossary of Terms
• Data Collection Standards
• Development and Enhancement (D&E)
• Maintenance and Support (M&S)
• ISO/IEC 29155. Systems and software engineering - Information
technology project performance benchmarking framework
Slide: 21
12/27/2012
Webinar Sponsored by Computer Aid, Inc.

22. Industry Benchmarking Standards
IT project performance benchmarking framework overview
International standard (ISO/IEC 29155-1. Systems and software engineering -- Information technology project performance benchmarking
framework -- Part 1: Concepts and definitions. )
Slide: 22
12/27/2012
Webinar Sponsored by Computer Aid, Inc.

23. Scope - What is ‘in’ and what is ‘out’
• Comparing “apples to apples”
• Decide on the inclusion and exclusion criteria for projects and
applications into the Benchmarking Set
• Delivery Options
• Types of Requirements
• Resourcing Profiles
• Technology Profiles
• Size Profiles
Slide: 23
12/27/2012
Webinar Sponsored by Computer Aid, Inc.

24. Scope - What is ‘in’ and what is ‘out’
• Aim for homogeneous data sets for comparison
• User Base
• Functional Domains
• Project Classifications
Slide: 24
12/27/2012
Webinar Sponsored by Computer Aid, Inc.

25. How often should we compare?
Collect data and compare at a rate that is useful and meaningful!
• Demonstrating Trends
• Allocating Projects to Benchmarking Periods
• Usefulness of the Result
• Statistical Validity of the Result
Slide: 25
12/27/2012
Webinar Sponsored by Computer Aid, Inc.

26. Comparable External data
• Organisational Type
• Different User Priorities
• Quality of the External Dataset
• Filtering of Submission Data in an External Dataset
Slide: 26
12/27/2012
Webinar Sponsored by Computer Aid, Inc.

27. Benchmarking Report
• Structure and content
• Level of granularity
• Presentation technology
• Confidentiality and Audience
• Review Process and Acceptance criteria
• Feedback process
Slide: 27
12/27/2012
Webinar Sponsored by Computer Aid, Inc.

28. Dispute Resolution Process
• Who is responsible for:
– resolution costs
– liability payments
– consequential loss
• Who will arbitrate?
Slide: 28
12/27/2012
Webinar Sponsored by Computer Aid, Inc.

29. Summary
• Benchmarking results provide important input into tactical and
strategic decisions
• Benchmarking can be done relatively cost effectively using Industry
data
• BUT ….Careful upfront planning is the most important step!
– Get the planning right and
– The rest is easy (…..well easier!!)
Slide: 29
12/27/2012
Webinar Sponsored by Computer Aid, Inc.

30. Other Considerations
• Submitting to the ISBSG Data Repository
– Maintenance and Support
– Development and Enhancement
• Benchmarking Internally
• Benchmarking against ISBSG
Slide: 30
12/27/2012
Webinar Sponsored by Computer Aid, Inc.

31. Thank You!
EMAIL
Admin@Totalmetrics.com
Total Metrics
TO GET THE FULL PAPER ON
Level 1, 191 Riversdale Road
WHICH THIS PRESENTATION
Hawthorn
WAS BASED.
Victoria 3122 Australia
Phone +61 (3) 9882 7611
Admin@Totalmetrics.com
www.totalmetrics.com
Slide: 31
12/27/2012
Webinar Sponsored by Computer Aid, Inc.

32. Questions?
Slide: 32
12/27/2012
Webinar Sponsored by Computer Aid, Inc.

33. CAI Sponsors Proudly Sponsors
The IT Metrics & Productivity Institute
• IT and Software Knowledge Center: WWW.ITMPI.ORG
• Weekly Webinars Hosted by Industry Leaders: WWW.ITMPI.ORG / WEBINARS
• Access Webinar Recordings Anytime at WWW.ITMPI.ORG / LIBRARY
• Enjoy the Benefits of ITMPI Membership at WWW.ITMPI.ORG / SUBSCRIBE
• Follow Us on TWITTER at WWW.TWITTER.COM / ITMPI
• Follow Us on FACEBOOK at WWW.ITMPI.ORG/ FACEBOOK
• Join Our Network on LINKED IN at WWW.ITMPI.ORG/ LINKEDIN
• Find Out About Our CONFERENCES at WWW.ITMPI.ORG/ EVENTS
Slide: 33
12/27/2012
Webinar Sponsored by Computer Aid, Inc.

34. Subscribe to the AITS Newsletter
• Daily articles in your mailbox on improving the management of IT.
Subscribe at www.AITS.org
• Listen to our “Thought Radio” podcasts on iTunes or at
www.AITS.org
Slide: 34
12/27/2012
Webinar Sponsored by Computer Aid, Inc.

35. Pam Morris
CEO
Total Metrics
Pam.Morris@totalmetrics.com
Hosted by:
Michael Milutis
Director of Marketing
Computer Aid, Inc. (CAI)
Michael_Milutis@compaid.com
Slide: 35
12/27/2012
Webinar Sponsored by Computer Aid, Inc.