Benchmark METRICS THAT MATTER October 4 2012

METRICS THAT MATTER
Challenge Your Current Thinking!
(a.k.a. Selecting Valuable Metrics Instead of
Vanity Metrics)
Presented by: Betty Schaar & Jeff Roth

1
10/5/2012 ©2012 BenchmarkQA, Inc. All rights reserved.

METRICS THAT MATTER
The Software Quality Experts

“The only metrics that entrepreneurs should invest energy in

collecting are those that help them make decisions. Unfortunately,

the majority of data available in off-the-shelf analytics packages

are what I call ‘Vanity Metrics.’ They might make you feel good,

but they don’t offer clear guidance for what to do.”
Eric Ries

2

MEASUREMENT VS. METRIC
Measurement - The value of a dimension,

quantity, or capacity obtained by collecting
project data.
A single measurement does not provide support
for decision-making.

Metric - A comparison, ratio, or plot of a series
of measurements or an algorithm of two or
more measurements.
A metric or group of metrics can be used to
make decisions and manage a project.
3

ISSUES WITH METRICS
vs. Valuable

> Relative vs. Absolute
> Not enough context
> Systems don’t support data collection
> Not enough historical data to be valuable
> Fear causes “skewage” of data
> Outliers cause “skewage” of data
> The numbers can lie!
(or can at least be manipulated)

4

DETERMINING METRICS CONTEXT

5

CONTEXT TYPES

Environment/Technologies

SDLC/Methodology

Team/Organization

In Relation to
Other Information
Type of Project
6

CONTEXT – SDLC/METHODOLOGY

> Waterfall
> Agile
> Multiple phases/releases
> Pilot or prototype initially
> External constraints/factors
impacting delivery

7

AGILE SPECTRUM

© 2012 Impressum

8

CONTEXT - TYPE OF PROJECT

> In-house
 New development
 Maintenance
> COTS
> Outsourced
> SaaS
> New delivery platform, e.g.
mobile

9

CONTEXT - TEAM/ORGANIZATION

> Co-located team vs.
distributed team
> Internal vs. external
application users

10

CONTEXT – ENVIRONMENT/TECHNOLOGIES

> Regulated vs. non-regulated
> Mature vs. leading-edge
technologies

11

CONTEXT – IN RELATION TO OTHER INFO
> We have no open Severity 1 defects.

 Can we release?
∙ What if I told you we also have 250 Severity 2
defects?
> We ran 2,000 test cases.
 Can we release?
∙ What if I told you we ran out of time to test
the most recently added feature?
> We covered all of the critical
requirements.
 Can we release?
∙ What if I told you we only had time to run
positive tests?
∙ What if I told you we still have 2 open Critical
Severity defects?

12

CONTEXT – DON’T OVERTHINK IT

> SDLC/methodology

> Project type

> Team/organization

> Environment/ technologies

> Metrics in relation to other information

13

UNDERSTANDING YOUR AUDIENCE


ME
> QA Lead

 Test case building progress by tester
 Test execution progress by tester
 Defect close rate
 Test coverage ME

> Test Analysts
 Tests to build
 Feature coverage
 Test to execute
 Defects written by me
 Defects assigned to me

15

YOU
Development Lead

Customer
ME
YOU YOU

YOU
YOU

Business
Analyst YOU
Project
YOU
Manager Project
YOU YOU
QA Manager Sponsor

Product
Developer
Owner
16

US
> Project team

> Scrum team YOU ME

> QA team
YOU

YOU

YOU
YOU

17

THEM
> Enterprise

> Execute Suite / C-level THEM

> Divisional Managers THEM THEM

> Customers

ME

18

USE METRICS TO ANSWER A QUESTION


ARE WE DONE YET?
> Building Tests

 # of Total Tests vs. # Written/Approved
 Test Coverage Completeness
> Executing Tests
 # of Test Passed/Failed/Blocked/Not Run
 Automation vs. Manual
 Velocity/Rate of Execution

20


21

UAT PROCESS EXECUTION DASHBOARD

22

QUALITY CENTER TIP – VELOCITY OF TEST
Setup Result

23

ARE WE DONE YET?
> Sufficient Quality Feature Critical High Medium Low Total
Ready for
Retest

General 0 0 0 0 0 0
 # of Active Feature 1 0 0 0 0 0 0

Defects by Feature 2
Feature 3
0
0
1
0
1
0
0
0
2
0
2
1
Severity Feature 4
Feature 5
0
0
0
1
3
2
1
0
4
3
3
5
 Quality Bar Feature 6
Feature 7
0
3
0
1
0
1
0
0
0
5
0
0
Feature 8 0 0 0 0 0 0
Feature 9 2 3 5 0 10 0
Feature 10 0 1 0 0 1 0
QUALITY BAR Feature 11
Feature 12 - General
2
0
3
1
5
0
2
0
12
1
1
0
Feature 12: Sub 1 0 1 0 0 1 0
65% Feature 12: Sub 2 0 0 0 0 0 0
Feature 12: Sub 3 0 0 1 0 1 0
Feature 12: Sub 4 0 0 0 0 0 0
Feature 12: Sub 5 0 0 0 0 0 0
# Tests Passed Feature 12: Sub 6 0 0 0 0 0 0
Feature 13 0 5 0 0 5 1
Total # Tests Totals 7 17 18 3 45 13

Executed Closed Since Ready for Reported Since
Closed Deferred Duplicate
8/18/2011 Retest 8/18/2011
18 245 9 0 13 5

24

WAS OUR APP/RELEASE GOOD ENOUGH?
> Warranty Period

 Quantity & Severity of Post Release Defects
 Time for Post Production Fixes
> Defect Removal Rate
# of pre-release defects
(# of warranty period defects
+ # of pre-release defects)

> Cost of Production Defects
# hours for prod defects fixes
X average burden rate $$

25

IS QUALITY IMPROVING?
> Trending Analysis

 # of pre- and post-release defects found
and fixed per release (root cause
distribution analysis by release)
> Retrospectives
 Burnup/Burndown over multiple sprints
 User Stories/Tasks

26

HOW EFFECTIVE ARE WE?
> Sprint delivery effectiveness

 Burn down/burn up Velocity by Sprint
 Retrospective
> Test coverage
> # Tests per resource per hour

27

CAN I QUANTIFY THE IMPACT OF AN ISSUE?
> Frequency of Issue

 Quantity of occurrences
 Timeframe of each occurrence
> Severity of Issue
 Empirical scales (show stopper, critical, high,
medium, low)
 Subjective scales
> Impact of Issue
 Importance to business (priority/severity)
 Cost of not fixing

28

AUTOMATION ROI

Gains – Investment Costs
ROI= Investment Costs

> Simple ROI: monetary savings due to test automation
 PROs: Good overview for management
 CONs: Oversimplified & need resource costs
> Efficiency Automation: time savings resulting from
test automation
 PROs: Easy to gather data, shows team impacts
 CONs: Oversimplified, assumes 100% test execution each
cycle
29

SIMPLE ROI
Simple ROI
300.0%

250.0%

200.0%

% ROI 150.0% 18 Cycles
24 Cycles
100.0%
30 Cycles
50.0%

0.0%
1 2 3 4
Years

Task Manual Automation Manual Auto

Qty Rate Factor Total $$ QTY Rate Factor Total Initial Cost

Hardware 1 $ 1,000 - $ 1,000 2$ 1,000 $ 2,000
Software (Initial Costs) 0 $0 - $ - 2$ 4,000 $ 8,000 $ 1,000 $ 52,500
Software (Maintenance Costs) N/A $0 - $ - 1$ 8,000 0.2 $ 1,600
Build 500 Test Scripts N/A $0 - $ - 500 $ 85 1 $ 42,500

Execute & Analyze 500 Test Scripts 500 $60 0.17 $ 5,100 1$ 85 4 $ 340

Maintain 500 Test Scripts N/A $0 - $ - 1$ 85 8 $ 680

Execute Manual Test Suite (less Time to Execute
1000 $0 0.17 0.03
Automated) Automation
Total Cost of 24 Cycles of Manual Total Cost of 24 Cycles of Test
Testing $ 123,400 Automation $ 78,580
30

EFFICIENCY ROI
Efficiency ROI
35.0%

30.0%

25.0%

20.0%
% ROI 18 Cycles
15.0%
24 Cycles
10.0% 30 Cycles

5.0%

0.0%
1 2 3 4
Years

Task Manual Automation Hours per Day

Qty Rate Factor Total Days QTY Rate Factor Total Days Manual Auto
Hardware 1$ 1,000 - 2$ 1,000 8 18
Software (Initial Costs) 0 $0 - 2$ 4,000
Software (Maintenance Costs) N/A $0 - 1$ 8,000 0.2
Build 500 Test Scripts N/A $0 - 500 $ 85 1 62.5
Execute & Analyze 500 Test Scripts 500 $60 0.17 1$ 85 4
10.6 0.5
Maintain 500 Test Scripts N/A $0 - 1$ 85 8
1.0
Execute Manual Test Suite (less Time to Execute
1000 $0 0.17 0.03
Automated) 21.3 Automation 0.8
Total Time in Days of 24 Cycles Total Time in Days of 24
of Manual Testing
765.0 Cycles of Test Automation
628.5
31

ACTUAL VS. PROJECTED ROI
Actual vs. Projected ROI

150.0%

100.0%

50.0%

% ROI 0.0%
Projected ROI
-50.0%
Actual ROI

-100.0%

-150.0%
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
Year 1 Year 1
Month

• Visibility for ongoing value of automation
• Better understand/plan ROI for other
automation efforts
• Combat long term automation malaise

32

REFERENCES
> Vanity Metrics vs. Actionable Metrics,

http://www.fourhourworkweek.com/blog/2009/05/19/vanity-metrics-vs-actionable-metrics/

May 19, 2009 by Eric Reis
> Dion Johnson “Test Automation ROI”
http://www.dijohn-ic.com/test_automation_roi.pdf

February 7, 2006 Whitepaper created
for SQE Webinar

33

THANK YOU FOR ATTENDING!
For more information about BenchmarkQA and
the services we offer, please contact:

Molly Decklever
9523.392.2384
molly.decklever@benchmarkqa.com


Benchmark METRICS THAT MATTER October 4 2012

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Similar a Benchmark METRICS THAT MATTER October 4 2012

Similar a Benchmark METRICS THAT MATTER October 4 2012 (20)

Último

Último (20)

Benchmark METRICS THAT MATTER October 4 2012