Pairwise testing (known as allpairs testing all pairs testing, pair-wise testing, and orthogonal array test design) is a simple but extremely powerful test design approach. It helps testers create tests faster, create more powerful and thorough tests, and reduce the number of tests they need to execute. But what does "pairwise coverage" mean? Exactly? What is 3-way coverage? This pairwise testing tutorial clearly explains these fundamental test coverage concepts.

1. Understanding
Combinations
(Pairs and
Triples)

2. Each of the end points on this mind map (like “Young” and
“Friendly”) are parameter values.
Parameters
Parameter Values

3. This is what they look like in Hexawise.
Parameters
Parameter Values

4. When you click on the “Create Tests” button...
1
Hexawise generates a set of 2-way tests that contains
every single pair of those test parameter values in at
least one of the tests.
2

5. 2-way coverage is useful because so many defects can be
quickly found by testing all pairs of parameter values.
Studies show that 84% of defects in production could have
been found by testing for every pair of parameter values.
1 input or 2 inputs (“pairwise)
3 inputs
4, 5 or 6 inputs
•
•
•
•
Medical Devices: D.R. Wallace, D.R. Kuhn, Failure Modes in Medical Device Software: an Analysis of 15 Years of Recall Data, International Journal of Reliability, Quality, and Safety Engineering, Vol. 8, No. 4, 2001.
Browser, Server: D.R. Kuhn, M.J. Reilly, An Investigation of the Applicability of Design of Experiments to Software Testing, 27th NASA/IEEE Software Engineering Workshop, NASA Goddard SFC 4-6 December, 2002 .
NASA database: D.R. Kuhn, D.R. Wallace, A.J. Gallo, Jr., Software Fault Interactions and Implications for Software Testing, IEEE Trans. on Software Engineering, vol. 30, no. 6, June, 2004.
Network Security: K.Z. Bell, Optimizing Effectiveness and Efficiency of Software Testing: a Hybrid Approach, PhD Dissertation, North Carolina State University, 2006.

6. But what does 2-way coverage mean
EXACTLY?

7. It means this...
Every single pair of parameter values appears together
in at least one of these six test cases:

8. For example...
Stylish
and
Tax Collector?

9. That’s a pair of values. So at least one test includes it.
Stylish
and
Tax Collector?
Test # 2

10. Old
and
Shabby?

11. That’s a pair of values. So at least one test includes it.
Old
and
Shabby?
Test # 4

12. Hostile
and
Skydiving
Instructor?

13. That’s a pair of values. So at least one test includes it.
Hostile
and
Skydiving
Instructor?
Also test # 4

14. Young
and
Hostile?

15. That’s a pair of values. So at least one test includes it.
Young
and
Hostile?
Test # 6

16. We could do this for quite a while.
Choose any two parameter values.
Any pair of test inputs you choose will appear in at
least one of the 2-way tests.

17. For every set of Parameter Values entered into Hexawise
there is a finite set of pairs that can be tested.
There are exactly 24 possible pairs of parameter values
in this simple example.

18. All of the pairs will be tested in at least one test.
All 24 of those possible pairs are tested in at least one
of the following 6 tests.

19. Again, the reason it is a good idea to test all possible pairs is
because...
It is an extremely efficient way to find defects given that:
• Studies show that 84% of defects in production could have been
found by testing for every pair of parameter values and
• 2-way coverage can be achieved in a relatively small number of tests.
1 input or 2 inputs (“pairwise)
3 inputs
4, 5 or 6 inputs
•
•
•
•
Medical Devices: D.R. Wallace, D.R. Kuhn, Failure Modes in Medical Device Software: an Analysis of 15 Years of Recall Data, International Journal of Reliability, Quality, and Safety Engineering, Vol. 8, No. 4, 2001.
Browser, Server: D.R. Kuhn, M.J. Reilly, An Investigation of the Applicability of Design of Experiments to Software Testing, 27th NASA/IEEE Software Engineering Workshop, NASA Goddard SFC 4-6 December, 2002 .
NASA database: D.R. Kuhn, D.R. Wallace, A.J. Gallo, Jr., Software Fault Interactions and Implications for Software Testing, IEEE Trans. on Software Engineering, vol. 30, no. 6, June, 2004.
Network Security: K.Z. Bell, Optimizing Effectiveness and Efficiency of Software Testing: a Hybrid Approach, PhD Dissertation, North Carolina State University, 2006.

20. But what about coverage of
combinations involving 3 or more
parameter values?

21. It depends!
Sometimes, a specific combination of 3
values will be included in your set of 2way tests.
Other times, combinations of 3 values
will not be included in your set of tests.
Let’s look at a few examples.

22. 3-way combinations involve 3 specific parameter values.
Young
and
Stylish
and
Skydiving
Instructor?

23. 3-way combinations involve 3 specific parameter values.
Let’s look at a few examples of 3-way combinations and
see if they are covered in our 2-way tests.
Young
and
Stylish
and
Skydiving
Instructor?
Shabby
and
Hostile
and
Tax Collector?
Old
and
Stylish
and
Skydiving
Instructor?
Young
and
Stylish
and
Hostile?

24. This 3-way combination does appear in the 2-way tests.
Young
and
Stylish
and
Skydiving
Instructor?
This 3-way
combination
does appear
in test # 3
We got lucky.

25. This 3-way combination does not appear in the 2-way tests.
Old
and
Stylish
and
Skydiving
Instructor?

26. This 3-way combination does not appear in the 2-way tests.
Shabby
and
Hostile
and
Tax Collector?

27. This 3-way combination does not appear in the 2-way tests.
Young
and
Stylish
and
Hostile?

28. If you want to make sure that all possible
“triples” like these are covered by at least
one of your tests, you will need to create
3-way tests.
Let’s now take a look at the 3-way tests
from Hexawise that includes at least one
test for every possible combinations of
“triples.”

29. These 3-way tests contain at least one test for every triple.
Some 3-way combinations are covered in 2-way tests.

30. All 3-way combinations are covered in 3-way tests.
Reviewing the same 3-way combinations as before,
we see that all of them are covered in the 3-way tests.
That’s because every single triple is included in the 3way tests.
Young
and
Stylish
and
Skydiving
Instructor?
Old
and
Stylish
and
Skydiving
Instructor?
Test # 6
Test # 8
Shabby
and
Hostile
and
Tax Collector?
Young
and
Stylish
and
Hostile?
Test # 3
Test # 4

31. For additional information, please see:
training.hexawise.com
or
help.hexawise.com