Más contenido relacionado La actualidad más candente (20) Similar a Benchmark METRICS THAT MATTER October 4 2012 (20) Benchmark METRICS THAT MATTER October 4 20121. METRICS THAT MATTER
Challenge Your Current Thinking!
(a.k.a. Selecting Valuable Metrics Instead of
Vanity Metrics)
Presented by: Betty Schaar & Jeff Roth
1
10/5/2012 ©2012 BenchmarkQA, Inc. All rights reserved.
2. METRICS THAT MATTER
The Software Quality Experts
“The only metrics that entrepreneurs should invest energy in
collecting are those that help them make decisions. Unfortunately,
the majority of data available in off-the-shelf analytics packages
are what I call ‘Vanity Metrics.’ They might make you feel good,
but they don’t offer clear guidance for what to do.”
Eric Ries
2
10/5/2012 ©2012 BenchmarkQA, Inc. All rights reserved.
3. MEASUREMENT VS. METRIC
Measurement - The value of a dimension,
The Software Quality Experts
quantity, or capacity obtained by collecting
project data.
A single measurement does not provide support
for decision-making.
Metric - A comparison, ratio, or plot of a series
of measurements or an algorithm of two or
more measurements.
A metric or group of metrics can be used to
make decisions and manage a project.
3
10/5/2012 ©2012 BenchmarkQA, Inc. All rights reserved.
4. ISSUES WITH METRICS
vs. Valuable
The Software Quality Experts
> Relative vs. Absolute
> Not enough context
> Systems don’t support data collection
> Not enough historical data to be valuable
> Fear causes “skewage” of data
> Outliers cause “skewage” of data
> The numbers can lie!
(or can at least be manipulated)
4
10/5/2012 ©2012 BenchmarkQA, Inc. All rights reserved.
6. CONTEXT TYPES
The Software Quality Experts
Environment/Technologies
SDLC/Methodology
Team/Organization
In Relation to
Other Information
Type of Project
6
10/5/2012 ©2012 BenchmarkQA, Inc. All rights reserved.
7. CONTEXT – SDLC/METHODOLOGY
The Software Quality Experts
> Waterfall
> Agile
> Multiple phases/releases
> Pilot or prototype initially
> External constraints/factors
impacting delivery
7
10/5/2012 ©2012 BenchmarkQA, Inc. All rights reserved.
9. CONTEXT - TYPE OF PROJECT
The Software Quality Experts
> In-house
New development
Maintenance
> COTS
> Outsourced
> SaaS
> New delivery platform, e.g.
mobile
9
10/5/2012 ©2012 BenchmarkQA, Inc. All rights reserved.
10. CONTEXT - TEAM/ORGANIZATION
The Software Quality Experts
> Co-located team vs.
distributed team
> Internal vs. external
application users
10
10/5/2012 ©2012 BenchmarkQA, Inc. All rights reserved.
11. CONTEXT – ENVIRONMENT/TECHNOLOGIES
The Software Quality Experts
> Regulated vs. non-regulated
> Mature vs. leading-edge
technologies
11
10/5/2012 ©2012 BenchmarkQA, Inc. All rights reserved.
12. CONTEXT – IN RELATION TO OTHER INFO
> We have no open Severity 1 defects.
The Software Quality Experts
Can we release?
∙ What if I told you we also have 250 Severity 2
defects?
> We ran 2,000 test cases.
Can we release?
∙ What if I told you we ran out of time to test
the most recently added feature?
> We covered all of the critical
requirements.
Can we release?
∙ What if I told you we only had time to run
positive tests?
∙ What if I told you we still have 2 open Critical
Severity defects?
12
10/5/2012 ©2012 BenchmarkQA, Inc. All rights reserved.
13. CONTEXT – DON’T OVERTHINK IT
The Software Quality Experts
> SDLC/methodology
> Project type
> Team/organization
> Environment/ technologies
> Metrics in relation to other information
13
10/5/2012 ©2012 BenchmarkQA, Inc. All rights reserved.
15. ME
> QA Lead
The Software Quality Experts
Test case building progress by tester
Test execution progress by tester
Defect close rate
Test coverage ME
> Test Analysts
Tests to build
Feature coverage
Test to execute
Defects written by me
Defects assigned to me
15
10/5/2012 ©2012 BenchmarkQA, Inc. All rights reserved.
16. YOU
Development Lead
The Software Quality Experts
Customer
ME
YOU YOU
YOU
YOU
Business
Analyst YOU
Project
YOU
Manager Project
YOU YOU
QA Manager Sponsor
Product
Developer
Owner
16
10/5/2012 ©2012 BenchmarkQA, Inc. All rights reserved.
17. US
> Project team
The Software Quality Experts
> Scrum team YOU ME
> QA team
YOU
YOU
YOU
YOU
17
10/5/2012 ©2012 BenchmarkQA, Inc. All rights reserved.
18. THEM
> Enterprise
The Software Quality Experts
> Execute Suite / C-level THEM
> Divisional Managers THEM THEM
> Customers
ME
18
10/5/2012 ©2012 BenchmarkQA, Inc. All rights reserved.
19. USE METRICS TO ANSWER A QUESTION
10/5/2012 ©2012 BenchmarkQA, Inc. All rights reserved.
20. ARE WE DONE YET?
> Building Tests
The Software Quality Experts
# of Total Tests vs. # Written/Approved
Test Coverage Completeness
> Executing Tests
# of Test Passed/Failed/Blocked/Not Run
Automation vs. Manual
Velocity/Rate of Execution
20
10/5/2012 ©2012 BenchmarkQA, Inc. All rights reserved.
22. UAT PROCESS EXECUTION DASHBOARD
The Software Quality Experts
22
10/5/2012 ©2012 BenchmarkQA, Inc. All rights reserved.
23. QUALITY CENTER TIP – VELOCITY OF TEST
Setup Result
The Software Quality Experts
23
10/5/2012 ©2012 BenchmarkQA, Inc. All rights reserved.
24. ARE WE DONE YET?
> Sufficient Quality Feature Critical High Medium Low Total
Ready for
Retest
The Software Quality Experts
General 0 0 0 0 0 0
# of Active Feature 1 0 0 0 0 0 0
Defects by Feature 2
Feature 3
0
0
1
0
1
0
0
0
2
0
2
1
Severity Feature 4
Feature 5
0
0
0
1
3
2
1
0
4
3
3
5
Quality Bar Feature 6
Feature 7
0
3
0
1
0
1
0
0
0
5
0
0
Feature 8 0 0 0 0 0 0
Feature 9 2 3 5 0 10 0
Feature 10 0 1 0 0 1 0
QUALITY BAR Feature 11
Feature 12 - General
2
0
3
1
5
0
2
0
12
1
1
0
Feature 12: Sub 1 0 1 0 0 1 0
65% Feature 12: Sub 2 0 0 0 0 0 0
Feature 12: Sub 3 0 0 1 0 1 0
Feature 12: Sub 4 0 0 0 0 0 0
Feature 12: Sub 5 0 0 0 0 0 0
# Tests Passed Feature 12: Sub 6 0 0 0 0 0 0
Feature 13 0 5 0 0 5 1
Total # Tests Totals 7 17 18 3 45 13
Executed Closed Since Ready for Reported Since
Closed Deferred Duplicate
8/18/2011 Retest 8/18/2011
18 245 9 0 13 5
24
10/5/2012 ©2012 BenchmarkQA, Inc. All rights reserved.
25. WAS OUR APP/RELEASE GOOD ENOUGH?
> Warranty Period
The Software Quality Experts
Quantity & Severity of Post Release Defects
Time for Post Production Fixes
> Defect Removal Rate
# of pre-release defects
(# of warranty period defects
+ # of pre-release defects)
> Cost of Production Defects
# hours for prod defects fixes
X average burden rate $$
25
10/5/2012 ©2012 BenchmarkQA, Inc. All rights reserved.
26. IS QUALITY IMPROVING?
> Trending Analysis
The Software Quality Experts
# of pre- and post-release defects found
and fixed per release (root cause
distribution analysis by release)
> Retrospectives
Burnup/Burndown over multiple sprints
User Stories/Tasks
26
10/5/2012 ©2012 BenchmarkQA, Inc. All rights reserved.
27. HOW EFFECTIVE ARE WE?
> Sprint delivery effectiveness
The Software Quality Experts
Burn down/burn up Velocity by Sprint
Retrospective
> Test coverage
> # Tests per resource per hour
27
10/5/2012 ©2012 BenchmarkQA, Inc. All rights reserved.
28. CAN I QUANTIFY THE IMPACT OF AN ISSUE?
> Frequency of Issue
The Software Quality Experts
Quantity of occurrences
Timeframe of each occurrence
> Severity of Issue
Empirical scales (show stopper, critical, high,
medium, low)
Subjective scales
> Impact of Issue
Importance to business (priority/severity)
Cost of not fixing
28
10/5/2012 ©2012 BenchmarkQA, Inc. All rights reserved.
29. AUTOMATION ROI
The Software Quality Experts
Gains – Investment Costs
ROI= Investment Costs
> Simple ROI: monetary savings due to test automation
PROs: Good overview for management
CONs: Oversimplified & need resource costs
> Efficiency Automation: time savings resulting from
test automation
PROs: Easy to gather data, shows team impacts
CONs: Oversimplified, assumes 100% test execution each
cycle
29
10/5/2012 ©2012 BenchmarkQA, Inc. All rights reserved.
30. SIMPLE ROI
Simple ROI
300.0%
The Software Quality Experts
250.0%
200.0%
% ROI 150.0% 18 Cycles
24 Cycles
100.0%
30 Cycles
50.0%
0.0%
1 2 3 4
Years
Task Manual Automation Manual Auto
Qty Rate Factor Total $$ QTY Rate Factor Total Initial Cost
Hardware 1 $ 1,000 - $ 1,000 2$ 1,000 $ 2,000
Software (Initial Costs) 0 $0 - $ - 2$ 4,000 $ 8,000 $ 1,000 $ 52,500
Software (Maintenance Costs) N/A $0 - $ - 1$ 8,000 0.2 $ 1,600
Build 500 Test Scripts N/A $0 - $ - 500 $ 85 1 $ 42,500
Execute & Analyze 500 Test Scripts 500 $60 0.17 $ 5,100 1$ 85 4 $ 340
Maintain 500 Test Scripts N/A $0 - $ - 1$ 85 8 $ 680
Execute Manual Test Suite (less Time to Execute
1000 $0 0.17 0.03
Automated) Automation
Total Cost of 24 Cycles of Manual Total Cost of 24 Cycles of Test
Testing $ 123,400 Automation $ 78,580
30
10/5/2012 ©2012 BenchmarkQA, Inc. All rights reserved.
31. EFFICIENCY ROI
Efficiency ROI
35.0%
The Software Quality Experts
30.0%
25.0%
20.0%
% ROI 18 Cycles
15.0%
24 Cycles
10.0% 30 Cycles
5.0%
0.0%
1 2 3 4
Years
Task Manual Automation Hours per Day
Qty Rate Factor Total Days QTY Rate Factor Total Days Manual Auto
Hardware 1$ 1,000 - 2$ 1,000 8 18
Software (Initial Costs) 0 $0 - 2$ 4,000
Software (Maintenance Costs) N/A $0 - 1$ 8,000 0.2
Build 500 Test Scripts N/A $0 - 500 $ 85 1 62.5
Execute & Analyze 500 Test Scripts 500 $60 0.17 1$ 85 4
10.6 0.5
Maintain 500 Test Scripts N/A $0 - 1$ 85 8
1.0
Execute Manual Test Suite (less Time to Execute
1000 $0 0.17 0.03
Automated) 21.3 Automation 0.8
Total Time in Days of 24 Cycles Total Time in Days of 24
of Manual Testing
765.0 Cycles of Test Automation
628.5
31
10/5/2012 ©2012 BenchmarkQA, Inc. All rights reserved.
32. ACTUAL VS. PROJECTED ROI
Actual vs. Projected ROI
The Software Quality Experts
150.0%
100.0%
50.0%
% ROI 0.0%
Projected ROI
-50.0%
Actual ROI
-100.0%
-150.0%
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
Year 1 Year 1
Month
• Visibility for ongoing value of automation
• Better understand/plan ROI for other
automation efforts
• Combat long term automation malaise
32
10/5/2012 ©2012 BenchmarkQA, Inc. All rights reserved.
33. REFERENCES
> Vanity Metrics vs. Actionable Metrics,
The Software Quality Experts
http://www.fourhourworkweek.com/blog/2009/05/19/vanity-metrics-vs-actionable-metrics/
May 19, 2009 by Eric Reis
> Dion Johnson “Test Automation ROI”
http://www.dijohn-ic.com/test_automation_roi.pdf
February 7, 2006 Whitepaper created
for SQE Webinar
33
10/5/2012 ©2012 BenchmarkQA, Inc. All rights reserved.
34. THANK YOU FOR ATTENDING!
For more information about BenchmarkQA and
the services we offer, please contact:
Molly Decklever
9523.392.2384
molly.decklever@benchmarkqa.com
10/5/2012 ©2012 BenchmarkQA, Inc. All rights reserved.