Software Testing has evolved and matured over the past few years on all the fronts, be it process or technical. When it comes to Code Coverage Testing, there always remain debates with novel & professionals alike each claiming its own definition of Code Coverage. The biggest universally accepted myth is that 100% code coverage guarantees exceptional quality. There are books and material available on the subject which are not wrong, but lack the clarity which is needed for actual implementation. The current paper is an attempt to tailor the entire subject in one crisp document and so goes the title, “Nuts and Bolts of Code Coverage Testing”. The paper starts with defining Code Coverage along with clearing out myths on Code Coverage. It demonstrates a real-time example of Code Coverage implementation and how/where other testing activities are involved and their relation with Code Coverage. It presents a methodical way to implement Code Coverage (highlighting the best practices along with pit falls to check out of). It also presents a real time case study of one of the heaviest products in the Industry, where it reached to a point where testing became almost untrack able and how Code Coverage Implementation helped to bring the Testing activity back on track.

1. Abhinav Vaid
IT Practitioner
Technical
Trends
Nuts and Bolts of Code Coverage Testing
Abstract— Software Testing has evolved and matured over the past few years on all the fronts, be it process or technical. When it
comes to Code Coverage Testing, there always remain debates with novel & professionals alike each claiming its own defi nition of
Code Coverage. The biggest universally accepted myth is that 100% code coverage guarantees exceptional quality. There are books
and material available on the subject which are not wrong, but lack the clarity which is needed for actual implementation. The current
paper is an attempt to tailor the entire subject in one crisp document and so goes the title, “Nuts and Bolts of Code Coverage Testing”.
The paper starts with defi ning Code Coverage along with clearing out myths on Code Coverage. It demonstrates a real-time example
of Code Coverage implementation and how/where other testing activities are involved and their relation with Code Coverage. It
presents a methodical way to implement Code Coverage (highlighting the best practices along with pit falls to check out of). It also
presents a real time case study of one of the heaviest products in the Industry, where it reached to a point where testing became
almost untrack able and how Code Coverage Implementation helped to bring the Testing activity back on track.
What is Code Coverage ?
Code Coverage is one of the measurement
criterion used to identify the Test
Coverage of Application Under Test. This
is typically used to publish the numbers
to the management/stake holders so that
they can make more informed decisions
before releasing a product.
Code Coverage is performed to
know the amount of Application Code
being exercised during testing (regardless
of Black Box, White Box, or Grey Box)
Testing.
This term is often confused with
Statistical Code Analysis which is a
completely di! erent activity done typically
by developers, dev. leads and some of the
most commonly techniques include code
inspections, walkthroughs etc.
Why Code Coverage?
Over the past few years, the software
development tools have evolved in terms
of maturity levels (simplicity from the
development perspective but at the cost of
increasing internal complexities). Most of
the development Tools are wizard driven
& feature rich but they come with a lot of
overhead. The codebase of the application
not only includes its own codebase, but
also of OS, support/3rd party libraries. It
makes Testing a daunting task as the size
of codebases associated with the product
remains complex and bulky. Development
Team does have Unit Test cases where
the scope is limited. The fi nal gate always
remains to be the Testing Team and no
matter how hard the Team tries, there
would always remain areas where the
code will be running multiple times and
on the other side there will be areas where
the application code will never be tested.
Historical data proves that it’s
impossible to release bug free products
which are destined to come when the
Product goes live. Manual Test Team
cannot be blamed as their job is to
execute the tests based on the Test Cases
generated from Functional Requirements
documentation. At the same time,
Automated Regression Tests can’t be
blamed as their scope is limited to ensure
nothing is broken where inputs are again
in terms of black and white (0’s and 1’s).
And similar is the case with system,
integration, grey box, and performance
testing.
Code Coverage: is the Test execution
performed on the Product with the
internal binaries hooked to a Coverage
Tool so that an accurate analysis can be
drawn as to how much code was actually
exercised during test execution. It draws
a measurement criterion of the untested
code of the application.
What Code Coverage Testing is not?
It is never intended to replace:
a. Manual Testing
b. Automated Testing
c. Grey Box Testing
d. Unit Testing
e. Static Analysis Tool – which includes
code walkthroughs, reviews etc.
Implementing Code Coverage
Following are the high level objectives
that should be thoroughly studied and put
together in order to implement e! ective
Code Coverage Testing.
1. Draw the right expectations –
It’s always important to clearly set
expectations so that there is no scope of
ambiguities in future. Following are the
key expectations to be considered -
a. Objective Indicator of Test Coverage
of application code
b. Links to uncovered Packages /
Classes / Methods / Branches
c. Links to uncovered Folders / Files /
Lines
d. Drill down support right from the
namespace to individual line of code
e. Early detections of uncovered Test
Coverage to avoid later investments
(which are higher multiple times as
the Project Progresses)
f. Remove redundancy (same code
being exercised in multiple tests) in
testing
g. Increased Confi dence for Releases
2.!Selecting the Tool: What to look for
while selecting a Code Coverage Tool –
there are many factors that need to be
considered when it comes to selecting the
Code Coverage Tool. The most important
key factors are mentioned below:
3.!Code Coverage Strategy (Short Term
and Long Term)
4."Execution and Reporting – This is one
of the most critical part which is often
neglected by the Engineering teams. The
product owners/stake holders need this
information. The information not only needs
to be accurate, but at the same time crisp
(with meaningful data) and presentable.
A Real Time Implementation
For the Purpose of demonstration, let’s do
a real-time implementation and see how
Code Coverage Testing is di! erent from
other types of Testing.
Application Under Test: Consider a web
or UI form (as displayed in Fig. 1) with only
3 controls, User ID, Password, and Login
CSI Communications | June 2014 | 14 www.csi-india.org

2. SNO Deciding Factor for Tool Selection Code Coverage tool
1. Uncovering Untested Areas Lines/nodes that remain untested
2. Execution modes For example, support for debug/run mode.
CSI Communications | June 2014 | 15
with an option to minimize, maximize
and close.
Black Box Testing: Enter a legitimate User
ID and the Password and click the submit
button. Expected outcome would be
getting to the home page.
Grey Box Testing: Would include going
to the database and validating that the
User ID and Password are authentic.
Expected behavior would be accurate
credentials.
Load/Performance Testing: Would
include multiple users logging in to the
form (with some ramp up patterns)
using their respective credentials,
followed by constant load and finally
ramp down. Expected outcome would
be conformance to SLA’s (be it response
time, CPU/memory utilizations etc.).
Automated Testing: A script would add
user id from an external fi le and then hit
the submit button in un-attended mode.
Expectation would be the execution report
marking the result as pass.
Code Coverage Testing: Code Coverage
can be generated for each of the Tests
mentioned above. The expectation would be
to publish the results and increase tests to
increase coverage. A typical Code Coverage
Report looks like shown in Fig. 2 and 3 below.
Code Coverage has a typical
jargon associated with it. It is not as
hard as it sounds at the first place. For
example, terms like Branch Coverage,
Sequence Point Coverage and so on.
The key terms are explained below
3. Identifying Repetitive Tested Areas
4. Quality of Test Reporting Metrics Should be intuitive/meaningful
5. Capacity to drill down from the top most module to pin
pointing the code/function
6. Accuracy of results during evaluation of the Tool before
making a recommendation
7. User Friendliness Should be intuitive/meaningful
8. Logging levels For example, basic, verbose etc.
9. Coverage Extensibility
10. Merging of Results Merging can be done manually as well as automated.
Popular Tools support both the modes of merging
Table 1
Fig. 1: Application Under Test

3. Fig. 2: Code Coverage Report
Fig. 3: Code Coverage Report 2
CSI Communications | June 2014 | 16 www.csi-india.org

4. CSI Communications | June 2014 | 17
Line Coverage - Line coverage captures
the number of lines that were executed
during testing activity. Later, the numbers
are compared with the total number
of executable lines (codebase of the
Application under Test). There is a
downside of Line Coverage because it
measures the line formats rather than the
code. For example, it is possible to format
a program in a single line and achieve
100% line coverage with only one test.
Statement/Branch Coverage - Statement
and branch testing are relatively stronger
than Line Coverage, but have the weakness
that interactions between decision
outcomes can mask errors during testing.
Decision Coverage - By requiring decision
outcomes to be performed independently
during testing, basis path testing can
expose additional errors.
For example, please refer to the
following code-func()
(
if (condition1)
a = a + 1;
if (condition2)
a = a -1;
)
If the expectation from the function is to
have the value of variable “a” unchanged
under all circumstances, it makes testing a
challenging task. Now let’s consider all the
Tests for testing this piece of code.
1." Branch Testing - The branch testing
can be accomplished by executing
two tests that do not detect error. The
tests would be -
a." The fi rst test to make both decision
outcomes to be false, in which case
the value of “a” will not get a! ected.
b." The second test to make both
decision outcomes to be true.
2." Statement Testing - The statement
testing can be accomplished only by
the later test
3."Basis Path Testing – From the above
example, it can be concluded that neither
statement nor branch testing is su# cient
to detect the error. In this particular
example, Basis Path Test will successfully
detect the error.
Code Coverage is measured as percentage
of application code executed during the
testing activities. Code Coverage can be
measured at various levels – in terms of
programming language like – Namespaces,
Classes, Methods, and Branches or in terms
of physical parameters like - Folders, Files
and Lines. And the idea is simple, generate
coverage add tests to increase coverage till
you reach the targeted coverage.
Note: 100% Code Coverage can never claim
that the product is bug free. It can help
ensuring that the code is 100% tested.
Case Study
A Case Study of Code Coverage
Implementation for Testing an
Application
Summary/Challenge: The codebase of
the Application was increasing from one
version to the next. This was primarily
because it had to support backward
compatibility (for legacy versions) as
well as add new features, do patch/SP
releases. The increasing complexities and
the bulkiness of product made testing a
challenge as well as an expensive task. It
needed around 8-9 months of e! ort to
just to test the regression suites manually
(with a team size of 70 engineers). In
order to control the situation, the test
suites were automated via as is basis, and
the execution time was reduced to 7 days
for un-attended test execution. The team
could spend qualitative time in enhancing
the existing automation suites and testing
for new feature/patch/bug fi x releases.
Later it was discovered that there was
no actual criterion to understand as to
how much application code was actually
tested. There were high possibilities of -
1. Cases where code was hitting the
same sequence hundreds of times
(maybe more)
2. Cases where the code was never
reaching out during Testing
Solution
A couple of popular Code Coverage
Tools were evaluated and the Application
was tested after confi guring the Code
Coverage. The Test Results of various Tools
were compared. The results of the Code
Coverage were alarming but accurate.
The best fi t Code Coverage Tool was
recommended, a Code Coverage strategy
was put together along with highlighting
the current as well as future targets.
The table mentioned below was put
together and shared in common dashboard.
The high level objectives included –
1. Highlighting and publishing the
current state of the Product
2. Use the number as a benchmark
to set the targets for subsequent
releases and
3. To bring back the clarity in Testing
with a continuous focus on
improvement
4. Highlight/bring in clarity in the Test
Organization as well as Product
Management team
The table below uses 10 %( a random
number) as targeted increased coverage
for subsequent releases
Release
Categorization
Functional/Manual
Testing
Automated Test
Execution
Performance and
Load Testing
Total % of Code
Coverage
Achieved
Release X(baseline
Coverage) A% -- Benchmark B% -- Benchmark C% -- Benchmark
Release Y
Targeted Coverage
Targeted Coverage – A
Targeted Coverage –
Actual Coverage
%+10%
B %+10%
Targets are normally not for
the increased coverage but
for application stability
Release Z
Targeted Coverage
Actual Coverage
Table 2: Code Coverage Dashboard Metrics

5. Recommended Best Practices of
Code Coverage Testing
1. Ensure covering all application
paths/workfl ows in terms of decision
trees in the code
2. Ensure covering all data values –
this can be done by patterns with di! erent
sets of data and can avoid extra fat/tests.
3. Code Coverage Tool should
determine where the controls/sequences
are being tested/not being tested, which
can save a lot of fat/extra tests being
tested by automated suites. It can cover
the classes, methods, branches but not
the business logic.
4. Give graphical representation of
the results in terms of real time charts &
metrics.
5. Education – Ensure that the stake
holders are aware of the shortcomings
and the benefi ts of implementing Code
Coverage so that the expectations are
clearly set and visible.
6. Testing is Proportional to
Complexity. It is a known fact that 80%
of the bugs are always found in 20% of
the code. Identifi cation, Planning, and
execution of E! ective strategy can make a
huge di! erence and bring in value to the
Organization.
Analysis
Some Important Points to recap –
1. Code Coverage can never replace any
other form of Testing
2. Testing activity is always proportional
to complexity of the application
3. Testing e! orts should be focused on
the error prone software and/or error
prone components.
4. Making informed decisions; whether
it is taking calculated risks or
confi dence in application stability. A
couple of examples include -
"a. Facilitating Quicker Release
Cycles
"b. Shrinking down the execution
time
"c. Percentage of un-tested code in
Application
"d. How have been the Code
Coverage’s from x Release x to
y release and what’s the target
benchmark set for forthcoming
release?
Take Away/Where to go from Here: The
Code Coverage is typically done by the
development Team. This is an attempt
to help manual/functional/automation
testers to get started on Code Coverage
Testing. I strongly recommend starting
with a real-time implementation taking
the sample project for getting started.
But before doing some tests, download
a couple of Code Coverage Tools and
learn how to hook on to the binaries of
the Application under Test. Aim for the
Tool that works best for your product
and justifi es business needs. Go through
the internals of the Application under
Test (an area which is always a black box
for a tester). This should be followed by
going through the Tools to narrow down
for selection. Once a Tool is selected and
basic Tests and some benchmarking is
done. You are good to go. There is enough
information available in the websites,
Books, journals to help you make a
baseline, creating a long term strategy
to make a signifi cant impact on the
whole testing lifecycle and eventually the
Product. n
Abhinav Vaid, is an IT Practitioner with 15 years of experience in various blue chip companies including Motorola,
McAfee, and Lotus. He is the author of building automated test systems and a regular writer in various technical and
research journals. He is also the Foundation Member of Indian ISTQB Certifi cation Board.
About the Author
CSI Communications | June 2014 | 18 www.csi-india.org