1. April 1999, Vol. 12, No. 4
Software Metrics
What’s Practical About Software Measurement?
Joyce Statz 4
What Do Our Metrics Mean?
Richard E. Zultner 11
First, Get the Front End Right
Lawrence H. Putnam, Michael Mah, and Ware Myers 20
The Secrets of Highly Successful Measurement Programs
Carol A. Dekkers 29
Back to the Basics:
Metrics that Work for Software Projects
Dwayne Phillips 36

2. properly analyze any process metric: control
charts, from statistical process control. Not
only do control charts work well on soft-
ware metrics, they are essential to correctly
understanding what our process metrics are
telling us.
WHAT’S THE PROBLEM?
Most software organizations today know
that metrics are necessary for improvement.
Yet many do not have a clear idea of how to
analyze, or properly interpret, the metrics
they collect on their software processes. In
this article, we will focus on process metrics,
where we are measuring some attribute of
the software process (as opposed to the
software product produced by the process).
To support improvement efforts, our process
metrics must accurately detect changes in
the process being measured.
What Changes Must We Be Able to Detect?
There are only two changes that can
happen to any process, measured with
any process metric (see Figure 1).
1. The location (or “center”) of the
process can change. It can move higher
or lower — and whether the change is
favorable or not depends on what we’re
measuring. If the number of defects per
deliverable page increases, that’s bad.
Vol. 12, No. 4 April 1999 11
What Do Our
Metrics Mean?
by Richard E. Zultner
“If I had to reduce my message for management to just a few words, I’d say it
all had to do with reducing variation.” — W. Edwards Deming [3]
Acrucial question is being overlooked in many software organizations
pursuing metrics today: what do our metrics mean? If we cannot
draw correct conclusions about our metrics, then is it doubtful we can
take effective action based on them. And if we can’t do that, why bother
to collect them in the first place? Fortunately, a simple method exists to
Figure 1: TThheerree aarree oonnllyy ttwwoo cchhaannggeess tthhaatt ccaann ooccccuurr ttoo aa pprroocceessss.. On the left, the process
location — its center — has changed. On the right, the process dispersion — its variation —
has changed. For any process metric you collect, you should be able to tell if either change
has occurred. Can you?
©1999 by ZULTNER & COMPANY.
All rights reserved.

3. If we can’t tell signals
from noise, how will we
ever know if changes to
the process are
improvements —
or illusions?
If the number of programmer-hours
required to produce a release decreas-
es, that’s good. If we cannot tell if the
process center is shifting, then we
don’t know how the process is perform-
ing, we don’t know if it’s getting better
or worse, and we don’t know if our
actions had an effect — or how big
an effect.
Many organizations attempt to assess
this with a “percentage change”
calculation on their metrics. What
do you use?
2. The dispersion (or variation) of the
process can change. It can increase
(bad) or decrease (good). Processes
with less dispersion have less variability,
are more predictable, and generally
have lower costs and faster throughput.
If we cannot tell if the process variation
is changing, then we don’t know how
the process is performing. We don’t
know if it is getting better or worse,
and we don’t know if our actions had
an effect — or what that effect was.
Many organizations have no way of
assessing the variability of their process
from their metrics data. This means
they are blind to half the changes that
can occur in their processes — and half
the improvements as well. How do you
know if the processes your metrics
measure are increasing, or decreasing,
in their variability?
We must be able to detect both of these
changes in our process data, or our
metrics are not adequate for monitoring
or improvement.
What Makes It Hard to Detect
These Changes?
No matter what process metric you
measure, your data will vary. Variation is
inherent in all real-world processes and all
real-world measurement systems. So how
do we interpret our process metrics? How
do we know what is noise — the natural
variation that is an inherent part of the
process — and what is a signal (a sign the
process has changed)? If we can’t tell
signals from noise, how will we ever know
if changes to the process are improvements
— or illusions?
In many organizations, managers have
informal “mental limits” that they use to tell
if the changes in the data are “big enough”
to warrant investigation. Often these limits
are “±10%-15%.” How do you know, for
your own metrics data, what change is
significant?
Figure 2 shows a set of data presented at a
recent software development conference.
The presenter offered this data as evidence
that the process had changed and drew
attention to the “improvement” from the
first data point to the last data point (repre-
sented here by the dashed line) as a result
of changes made to the process. The only
other comment offered was that points 6
and 20 represented unusually small deliv-
eries, so that explained why their major
errors were so few.
Don’t these conclusions seem reasonable?
In many organizations, process metrics are
collected, plotted, and “eyeballed” to assess
the process. Unfortunately, it is very easy to
be misled by noise and either miss a signal
that the process has changed or see a false
signal when no change has actually
occurred. In this case, not only is there no
improvement, the process is out of control.
But the noise makes it very difficult to see
this by “eyeballing” the chart. How can we
avoid these all-too-common mistakes?
WHERE CAN WE LOOK FOR A SOLUTION?
If we cannot filter out the noise in our
process data, we will not be able to
12 April 1999 Vol. 12, No. 4 Get the Cutter Edge free: www.cutter.com/consortium/

4. correctly tell when the location or disper-
sion of our processes is changing.
Fortunately, the “Father of Quality,” Walter
Shewhart, solved this problem in the 1920s.
His solution became the foundation of the
quality field — statistical process control
(SPC). Developed by Shewhart at Bell
Laboratories while working in a Western
Electric manufacturing plant, the control
chart filters noise from signal — “uncon-
trolled from controlled variation” — and is
the heart of SPC [7, 8].
A process that exhibits only natural, or
controlled, changes is stable, and its perfor-
mance can be predicted. A process with
out-of-control changes, in either location or
dispersion, is unstable, and its performance
cannot be predicted. Metrics from an out-of-
control process predict nothing. The deter-
mination of whether a process is out of
control is not a matter of judgment or
opinion. It is not assessed by answers on a
questionnaire. It is determined by the same
simple statistical analysis that Shewhart
derived over 70 years ago (from both theo-
retical and empirical studies) on how to tell
if a process is in statistical control. So how
do the control charts of SPC tell us what our
processes are doing?
SOLUTION: CONTROL CHARTS
There are only two changes that can occur
in our data: the location and the dispersion.
We will use two control charts then, one to
detect each type of change.
Shewhart developed several different kinds
of control charts. The two that are most
appropriate for software metrics are the
Individual control chart (also called the “x”
Get the Cutter Edge free: www.cutter.com/consortium/ Vol. 12, No. 4 April 1999 13
Figure 2: EEvviiddeennccee ooff iimmpprroovveemmeenntt?? The data varies — as all real-world data does — but does
it show improvement or just noise? How can we tell? How do you know when “changes”
in your metrics are big enough to be meaningful?
Metrics from an
out-of-control process
predict nothing.

5. chart) and the moving Range control chart
(also called the “mR” chart). These partic-
ular charts make no assumptions about the
data or its distribution, so they can be used
with any software process metric.
A change in process dispersion is the more
damaging of the two possible process
changes. So first we will check if the process
dispersion has changed with the moving
Range (mR) chart. If any data points are
above the upper control limit (UCL), that is
a signal the process is not stable and has
changed.
In Figure 3, the same data from Figure 2 is
presented on an mR chart. The mR chart is
a plot of the successive differences between
the data points — the moving Range, a
measure of variability. All the data points are
below the UCL, so the process is stable.
There are no signals, no indication that the
process has changed (or improved) its vari-
ability. Since the process is stable, we can
predict that the process will continue to
produce an average variation of 0.50 major
errors per page and will approach a variation
of 1.62 major errors per page occasionally
— unless the process is changed.
Then we check for a change in process
location with the x chart. Shewhart devel-
oped four rules for the individual chart to
test if the process is in control:
The Four Western Electric Zone Rules
n Rule 1: One point more than three
standard deviations away from the
center line
n Rule 2: Two out of three successive
points more than two standard devi-
ations away from the center line on
the same side
n Rule 3: Four out of five successive
points more than one standard devi-
ation away from the center line on
the same side
Figure 3: MMoovviinngg RRaannggee ccoonnttrrooll cchhaarrtt.. Here there is only “common cause” or natural variation
— half the data is above average, half below average. All points are below the upper control
limit. The process is stable, predictable, and in statistical control.
14 April 1999 Vol. 12, No. 4 Get the Cutter Edge free: www.cutter.com/consortium/

6. Get the Cutter Edge free: www.cutter.com/consortium/ Vol. 12, No. 4 April 1999 15
n Rule 4: Eight consecutive points on
the same side of the center line
In Figure 4, the same data from Figure 2 is
presented on an x chart. The x chart is a
plot of the actual data points, a center line,
the control limit (at three standard devia-
tions), and the one- and two-standard devi-
ation lines. There is no signal that delivery
20 was different from the others. There is
no evidence the process has improved.
There are three signals that the process is
out of control. Points 1 and 16 are above the
upper natural process limit (a Rule 1 signal).
Points 6-13 (eight consecutive points)
appear on one side of the center line (a
Rule 4 signal). These points did not occur
by chance. They tell us that the process is
out of control — subject to special causes
— and that the location of the process is
wandering unpredictably. The process has
changed for the worse. Immediate action
should be taken to determine the cause
and stabilize the process. Because the
process is out of control, we cannot reliably
predict what the process will produce in the
future. The mean of 0.83 major errors per
page does not tell us what to expect on
average, and the upper natural process limit
does not tell us what the bounds are for
what the process will do in the future,
because the process is out of statistical
control.
Note that being in control (stable) or out of
control (unstable) tells us nothing about
whether the performance of the process is
acceptable or not. The process could be
stable at an unacceptably high number of
major errors per page or unstable around
an extremely low number of errors. Getting
the process stable is only the first step
toward real improvement.
Figure 4: IInnddiivviidduuaall ccoonnttrrooll cchhaarrtt.. Here there are multiple signals (circled) that “special causes”
are at work — and the process is out of control and unpredictable. No improvement has
occurred. Our metrics don’t tell us anything about what the process will do in the future.

7. MISCONCEPTIONS ABOUT
CONTROL CHARTS
Since control charts have been around for
70 years, why aren’t people using them for
software? Well, some people are using
them for software [1, 2, 4, 5, 14], but many
misconceptions remain. Here I will discuss
some of the most common questions raised
about software SPC and control charts.
“Aren’t control charts for manufacturing?
We don’t produce the same result over
and over as they do…” If you use a
defined process for software development
(a methodology), then you produce your
software products with a process, and
control charts will tell you how that process
is performing. If you have (or aspire to
have) ISO 9000 certification for your soft-
ware development process, then you must
describe your process and follow your
description. This is your defined process.
(If you don’t think you use a process to
develop your software, then what will you
be changing to build better software?
How will you improve? And what are you
measuring with your process metrics?)
Please understand that it is not necessary
to assume anything about your process to
use control charts. Whether you think it is
“defined” or not, the control chart will tell
you, based on your data, if your process is
stable (in control) or unstable (out of
control). Improving the definition of your
process is often helpful in getting an out-of-
control process into control. Control charts
don’t depend on the results being “the
same” but on the process being “the same”
— and they will tell you whether your
process is, or isn’t, “the same.” (By the way,
modern lean manufacturing, which
produces to order, doesn’t produce the
same result over and over either.)
Even if you use multiple processes during
the same project, control charts will tell you
if you really do have multiple processes, and
they will allow you to monitor, control, and
improve each process independently or
simultaneously. For example, if you suspect
that your experienced programmers use a
different process than your inexperienced
programmers, control charts can confirm
or disprove your hypothesis. If you think
complex modules are designed differently
than simple modules, control charts can tell
you if that is the case or not. If you think that
the size of your objects makes a difference
in how long it takes to design them, the
number of defects they contain, or any
other attribute of interest, control charts
can determine the magnitude of the differ-
ence — and whether it exists.
In more than a decade of working with
software organizations to improve their
processes, I have seen many “superstitions”
about software processes disproved (things
they “knew” made a difference — but
didn’t) and many significant factors revealed
(things that were not even suspected of
making a difference — but did). In order
to tell what is actually happening in your
processes, you must filter out the noise,
and only then can you really see if the
process changed. The control chart is the
oldest, simplest, and most robust tool for
properly analyzing process metrics.
Myths, or “Why we can’t use control
charts here.” Unfortunately there are many
common myths concerning control charts,
even in manufacturing organizations that
have used them for many years. One is that
the data must be normally distributed. A
second is that the control chart works
because of the central limit theorem. A third
is that the observations must be indepen-
dent. And a fourth is that the data must be
The control chart is the
oldest, simplest, and most
robust tool for properly
analyzing process metrics.
16 April 1999 Vol. 12, No. 4 Get the Cutter Edge free: www.cutter.com/consortium/

8. in control before plotting (including the use
of two sigma limits). Other misunderstand-
ings exist as well [11].
Some people may tell you that control
charts require lots of data — maybe more
than you have with your current metrics.
In fact, control charts can work with just
4-6 data points. Now if you only have a few
data points, you can only detect big process
changes. The more data you have, the
better you can detect small process
changes. For up to 16 data points, each
additional point is adding increasing resolu-
tion to your control chart. Beyond 16 data
points, each data point adds less and less to
your ability to detect small process changes.
And after 32 data points, you have effec-
tively reached the limit of your measure-
ment system to detect small process
changes [10].
“We aren’t ready. We’re not mature
enough…” Another myth-understanding is
that to use control charts or SPC, you must
be at some particular level of maturity with
your software process. Nothing could be
further from the truth. Maturity is a completely
separate issue from whether a process is in
control (or out of control), is producing
acceptable results (or not), or is capable of
being improved with SPC methods.
I have worked with a number of improve-
ment teams to improve specific software
processes (those that were most unaccept-
able from the standpoint of their customers)
with SPC and control charts. Most were at
CMM Level 1 when they began, and some
even had stable, but terrible, processes.
After making substantial (and, thanks to the
control chart, demonstrable) improvements,
their customers noticed the difference —
but they were still at Level 1 according to
the CMM. Control charts and SPC work
regardless of your “maturity.”
Don’t wait to use SPC and control charts.
If you want to improve, or just want to
know how your process is doing, do
control charts immediately with the data
you have right now.
APPLYING CONTROL CHARTS
TO YOUR DATA
Applying control charts to your data is easy
even if you do it manually — and it is very
easy with any spreadsheet. (The hardest
part is learning which options to set to get
your chart nicely formatted, but once you
have the format, you can reuse it.) Here is
an overview of the procedure.
Moving Range (mR) Chart
The first control chart you do is the mR
chart to check dispersion.
1. Calculate the moving Ranges: the
absolute value of the successive differ-
ences between each pair of data points.
So if your first five data points are 2.25,
0.85, 0.85, 1.20, and 0.95, then your mov-
ing Ranges will be __, 1.4, 0.0, 0.4, and
0.3. (The first blank is because you will
have one fewer moving Range than you
have data points.) Plot these moving
Ranges on your chart.
2. Calculate the mean of the moving
Ranges. (Add them up and divide by
how many there are.) For example, in
Figure 3, this came out to 0.50. Plot this
(“mR bar”) as the center line on your
chart.
3. Multiply the mean by 3.268. Plot this line
as the upper control limit. For example,
in Figure 3, 0.50 × 3.268 = 1.634. (With
all intermediate calculations carried out
by the spreadsheet with full precision,
the answer is 1.62 as plotted.) This line
is three standard deviations above the
mean. (There will not be a lower con-
trol limit on this chart, ever.)
Get the Cutter Edge free: www.cutter.com/consortium/ Vol. 12, No. 4 April 1999 17

9. Are all the moving Ranges inside the UCL?
Then the process dispersion is stable.
Individual (x) Chart
The second, and last, control chart you do is
the x chart to check location.
1. Plot the individual data points (2.25,
0.85, 0.85, 1.20, 0.95, and so on) on your
chart.
2. Calculate the mean of the individual
data points. For example, in Figure 4,
this came out to 0.82. Plot this (“x bar”)
as the center line on your chart.
3. Multiply the mean of the moving Range
(mR bar) by 2.660 and add that quantity
to the mean of the individual values (x
bar). This is the upper natural process
limit (UNPL). For example, in Figure 4,
(0.50 × 2.660) + 0.82 = 2.15. (With the
intermediate calculations carried out by
the spreadsheet with full precision, the
answer is 2.14, as plotted.) This line is
three standard deviations above the
mean. Plot it on your chart.
4. Multiply the mean of the moving Range
(mR bar) by 2.660 and subtract that
quantity from the mean of the individual
values (x bar). This is the lower natural
process limit (LNPL). For example, in
Figure 4, 0.82 − (0.50 × 2.660) = −0.51.
This line is three standard deviations
below the mean. If this line is above
zero (or your data can go below
zero), plot it on your chart. (In Figure
4 this line was not plotted, as it was
below zero.)
5. The distance from the center line to
the UNPL is three standard deviations.
Divide this interval by three to get one
standard deviation. For example, in
Figure 4, (2.14 − 0.82) ÷ 3 = 0.44, which
is one standard deviation. Draw a line
one standard deviation above and
below the center line. Draw another
line the same distance beyond those
lines. Now you should have lines drawn
at equal intervals of one, two, and three
standard deviations above and below
the center line. (If any of the lines is
below zero, and your data cannot go
below zero, you don’t need to draw
those lines.) Apply the four Western
Electric Zone Rules stated above. If
none of the points is a signal according
to the four rules, then the process loca-
tion is stable.
On the mR chart, we checked if the disper-
sion of the data is changing. On the x chart,
we checked if the location of the data is
changing. We now know objectively and
empirically if our process is in statistical
control and is therefore predictable.
CONTROL CHARTS WORK FOR SOFTWARE
PROCESS METRICS
Statistical process control is much more
than just doing control charts — it is a
complete system for process improvement
[6, 13]. But control charts are a vital
element, and they provide a simple, effec-
tive, and robust way to filter out the noise
that is inherent in every process and every
measurement system. They give you clear
signals using your own metrics as to
whether your process is stable or out of
control. If your process is stable, the control
charts allow you to predict what your
process can be expected to deliver in the
future. Can you do these things now?
If you do control charts on your key process
metrics, for your critical software processes,
then you will have repeatable, quantitative
evidence that your development process is
in statistical control. And once you achieve
such demonstrable stability of your critical
processes, you will have the solid founda-
tion needed to improve your performance
further to excellence.
Once you achieve
demonstrable stability of
your critical processes, you
will have the solid
foundation needed to
improve your performance
further to excellence.
18 April 1999 Vol. 12, No. 4 Get the Cutter Edge free: www.cutter.com/consortium/

10. Proven over 70 years, in use today in every
industry, the control chart is the simplest
tool available to give us essential insights
into the meaning of our own metrics. Why
not try one today?
REFERENCES
There is a lot more to SPC, and control
charts, than I covered here. I highly recom-
mend the books by Donald Wheeler [9-12],
starting with Understanding Variation [9].
For more information, including details and
an example of an Excel spreadsheet for soft-
ware control charts, e-mail a request to:
SoftwareSPC@Zultner.com.
1. Affourtit, Barba. “Statistical Process
Control Applied to Software.” In Total
Quality Management for Software, ed. G.
Gordon Schulmeyer and James I. McManus.
Van Nostrand Reinhold, 1992.
2. Card, David N., and Robert L. Glass.
Measuring Software Design Quality.
Prentice Hall, 1990.
3. Deming, W. Edwards. Out of the Crisis.
MIT Center for Advanced Engineering, 1986.
4. Grady, Robert B. Practical Software
Metrics for Project Management and Process
Improvement. Prentice Hall, 1992.
5. Grady, Robert B., and Deborah L. Caswell.
Software Metrics: Establishing a Company-
Wide Program. Prentice Hall, 1987.
6. Kume, Hitoshi. Statistical Methods for
Quality Improvement. Translated by John
Loftus. The Association for Overseas
Technical Scholarship, 1985. (Available from
ASQ Quality Press.)
7. Shewhart, Walter. Economic Control of
Quality of Manufactured Product. 1931. With
a foreword by W. Edwards Deming. Reprint,
ASQ Quality Press, 1980.
8. Shewhart, Walter. Statistical Method from
the Viewpoint of Quality Control. 1939. With
a foreword by W. Edwards Deming. Reprint,
Dover Publications, 1986.
9. Wheeler, Donald J. Understanding
Variation: The Key to Managing Chaos. SPC
Press, 1993.
10. Wheeler, Donald J. Advanced Topics in
Statistical Process Control. SPC Press, 1995.
11. Wheeler, Donald J., and Richard W.
Lyday. Evaluating the Measuring Process,
2nd ed. SPC Press, 1989.
12. Wheeler, Donald J., and David S.
Chambers. Understanding Statistical Process
Control, 2nd ed. With a foreword by W.
Edwards Deming. SPC Press, 1992.
13. Zultner, Richard E. “TQM for Technical
Teams.” Communications of the ACM, Vol.
36, No. 10 (October 1993), pp. 79-91.
14. Zultner, Richard E. “Software SPC.” In
Proceedings of the 4th International
Conference on Software Quality. ASQ
Software Division, 1994.
Get the Cutter Edge free: www.cutter.com/consortium/ Vol. 12, No. 4 April 1999 19
Richard “Show Me the Data”
Zultner is an international consul-
tant, educator, author, and
speaker. His primary focus is
on efficient software process
improvement — the rapid applica-
tion to software development of
daily management methods
such as statistical process control
(SPC), cross-functional manage-
ment techniques such as quality
function deployment (QFD), and
strategic improvement approaches
such as theory of constraints
(TOC). He is an avid and long-
time student of W. Edwards
Deming.
Mr. Zultner is a founder and
director of the QFD Institute, a
nonprofit research organization
dedicated to the advancement of
QFD. In 1998, for his work in
applying QFD to software, he
received the Akao Prize — one of
12 people in the world so honored
to date.
He holds a master’s degree in
management from the J.L. Kellogg
Graduate School of Management
at Northwestern University, and he
has professional certifications in
quality, software quality, project
management, software engi-
neering, and theory of constraints.
Mr. Zultner can be reached at
ZULTNER & COMPANY, 12
Wallingford Drive, Princeton, NJ
08540-6428 USA. Tel: +1 609 452
0216; Fax: +1 609 452 2643;
E-mail: Richard@Zultner.com;
Web site: www.zultner.com.