1. 101: “Let's Test Together”
Track: Hands-On Testing Techniques Lab
Tuesday, Oct. 19th 10:15am - 11:15am
Justin Hunter, CEO of Hexawise

2. Objectives
1. Introduce a test design method that
will help make you a more effective tester
2. Have you actively participate and
share your ideas in creating tests as we talk
3. Change the way you think about how
you should be testing software
2

3. Agenda
1. “Lessig-style” overview of benefits
2. Spools Exercise
3. Darts Exercise
4. Generating Tests
3

4. Lessig-style Overview of Benefits
[Launch Lessig-style presentation]
4

5. Spools Exercise
How many possible tests do 4
spools represent?
Credit Property
Rating (5) Income (6) Type (6) States (6)
A+ 0 1 Family CA
A 1-30K 2 Family IL
A- 30,001-50K 3 Family GA
B 50,001-100K 4 Family NY
<B 100,001-1Mil Coop TX
1 Mil - 50 Mil Condo WI
5

6. Spools Exercise
How many possible tests do 8
spools represent?
How many possible tests do all
the spools represent?
How long does it take before
your head is “ready to explode?”
6

7. Spools Exercise
How do you
cope?
http://www.flickr.com/photos/stewf/2579810818
7

8. Spools Exercise: ‘The Problem with Blinders’
Combinations of 2 test inputs are responsible
for many defects, including these ones:
1) Use shorten URL feature? = Y and
2) Type to shorten = already shortened URL
1) Take a photo? = Y and
2) In the midst of composing tweet = Y
8

9. Darts Exercise
With 3.7 million total possible tests,
how many tests are required to test
for all possible pairs of values?
http://www.flickr.com/photos/kanaka/1798327442
9

10. Hands on Test Generation
“Let’s Test Together”
First Set of Tests- Mortgage Example:
1. H/W and/or Software Configurations
2. User Types
3. Main User Actions
4. Sub-steps and Choices for User Actions
5. Who, What, When, Where, Why, How
6. Business Rules
10

11. Hands on Test Generation
“Let’s Test Together”
Second Set of Tests - (Participants to
Choose an Application):
1. H/W and/or Software Configurations
2. User Types
3. Main User Actions
4. Sub-steps and/or Choices for User Actions
5. Who, What, When, Where, Why, How
6. Business Rules
11

12. Appendix Slides
12

13. Want to hear more? Want to practice?
601: “5 Kick Ass Test Design Techniques
from Combinatorial Testing”
Track: Hands-On Testing Techniques Lab
Wednesday, Oct. 20th 2:30pm - 3:30pm
Justin Hunter, CEO of Hexawise

14. Overview of techniques discussed in tomorrow’s session
1 2
3
4
5
© Hexawise, 2010. All rights reserved.
14

15. Additional sources of information
Excellent introductory articles and instructional videos:
www.CombinatorialTesting.com
Also, please feel free to contact me if you have any questions.
I’d be happy to quickly review a test plan or two, answer your
questions, give quick pointers to help you run a pilot, etc.
.... Seriously.... I enjoy helping people get started with this
approach. Please don’t hesitate to reach out. There’s no charge
and no catch.
15

16. Practice Tips: 4-Step Process
Four-Step Process to Design Efficient and Effective Tests
Passive
Field
Treatment
1.
Plan Scope
Be
clear
about
single
or
mul3ple: Acceptable
op3ons:
Level
of
Detail
Dis3nguish
between
important
fields
(par3cularly
those
that
will
trigger
business
rules)
and
unimportant
fields
in
the
-­‐
Features
/
Func3ons
/
Capabili3es -­‐
High
level
“search
for
something” applica3on.
-­‐
User
types -­‐
Medium
level
“search
for
a
book”
Quickly
document
what
your
approach
will
be
towards
passive
-­‐
Business
Units -­‐
Detailed
“search
for
‘Catcher
in
the
Rye’
fields.
You
might
consider:
ignore
them
(e.g.,
don’t
select
any
-­‐
H/W
or
S/W
Configura3ons
by
its
3tle” Values
in
your
plan)
or
a
3
Value
approach
such
as
“valid”
“Invalid
(then
fix)”
and
“Blank
(then
fix)”
Ac3ons
2.
Create Configura3ons
First
add
hardware
configura3ons
Users
Next,
add
mul3ple
types
of
users
(e.g.,
Start
with
Big
Common
Ac3ons
made
by
users
administrator,
customer,
special
customer)
AZer
comple3ng
Big
Common
Ac3ons,
circle
back
and
add
Next
add
soZware
configura3ons
Consider
permission
/
authority
levels
of
admin
Small
Ac3ons
and
Excep3ons
users
as
well
as
business
rules
that
different
users
Remember
some
ac3ons
may
be
system-­‐generated
might
trigger
Itera3on
3.
Refine Business
Rules
Select
Values
to
trigger
bus.
rules
Gap
Filling
Iden3fy
gaps
by
analyzing
decision-­‐tree
outcomes
Refine
longest
lists
of
Values;
reduce
their
numbers
by
using
equivalence
classes,
etc.
sought
vs.
delivered
by
tests,
“gap
hun3ng”
Iden3fy
equivalence
classes conversa3ons
w/
SME’s,
etc. Create
Tests
with
and
w/out
borderline
Values;
consider
cost/
benefit
tradeoffs
of
addi3onal
test
design
refinements
Test
for
boundary
values Fill
gaps
by
either
(i)
adding
Parameters
and/or
Values
or
(ii)
crea3ng
“one-­‐off”
tests.
Consider
stopping
tes3ng
aZer
reaching
~80%
coverage
Mark
constraints
/
invalid
pairs
Consider
2-­‐way,
3-­‐way,
and
Mixed-­‐Strength
op3ons
Con3nuous
Improvement
4.
Execute
Auto-­‐Scrip3ng Expected
Results
Add
auto-­‐scrip3ng
instruc3ons
once;
apply
Export
the
tests
into
Excel
when
you’re
done
If
possible
measure
defects
found
per
tester
hour
“with
and
without
Hexawise”
and
share
the
results
those
instruc3ons
to
all
of
the
tests
in
your
itera3ng
the
plan
plan
instantly Add
inputs
based
on
undetected
defects
Add
complex
Expected
Results
in
Excel
post-­‐export
Don’t
include
complex
Expected
Results
in
Share
good,
proven,
plan
templates
with
others
auto-­‐scripts
© Hexawise, 2010. All rights reserved. 16

17. Practice Tips: Warning Signs
“You might be headed for trouble if...”
Passive
Field
Treatment
1.
Plan Scope
...
You
cannot
clearly
describe
both
the
Level
of
Detail
...
Parameters
with
the
most
Values
have
more
...
You
cannot
clearly
describe
your
strategy
to
deal
with
scope
of
your
test
plan
and
what
will
be
leZ
Values
than
they
require.
365
values
for
“days
of
unimportant
details.
If
Values
will
impact
business
rules,
focus
out
of
scope. the
year”
is
bad.
Instead,
use
equivalence
class
on
them.
If
Values
don’t
impact
business
rules,
consider
Values
like
“weekend”
&
“weekday.”
When
in
ignoring
them.
doubt,
choose
more
Parameters
and
fewer
Values.
Ac3ons
2.
Create Configura3ons Users
...
You
have
ignored
hardware
and
soZware
...
You
have
not
included
all
the
different
types
of
...
You
start
entering
Small
Ac3ons
(e.g.,
“search
for
a
hardback
configura3ons
without
first
confirming
this
users
necessary
to
trigger
different
business
rules.
science
book
by
author
name”)
before
you
enter
Big
Ac3ons
approach
with
stakeholders. What
user
types
might
create
different
outcomes?
(e.g.,
“Put
Something
in
Cart.
Buy
it.”)
First
go
from
beginning
Authority
level?
Age?
Loca3on?
Income?
to
end
at
a
high
level.
AZer
you’ve
done
that,
feel
free
to
add
Customer
status? more
details.
Itera3on
3.
Refine Business
Rules
...
You
forget
to
iden3fy
invalid
pairs.
-­‐
or
Gap
Filling
...
You
assume
that
the
test
condi3ons
coming
out
...
You
forget
to
look
at
the
Coverage
Analysis
charts.
If
you
-­‐
...
You
rely
only
on
Func3onal
of
Hexawise
will
be
100%
of
the
tests
you
should
achieve
80%
coverage
in
the
first
quarter
of
the
tests,
you
Requirements
and
Tech
Specs
w/out
run.
There
might
well
be
addi3onal
“one-­‐off”
should
measure
the
cost/benefit
implica3ons
of
execu3ng
the
thinking
hard
yourself
and
asking
ques3ons
things
that
you
should
test
and/or
a
few
nega3ve
last
3/4
of
the
tests.
to
SME’s
about
business
rules
and
tests
to
design
by
hand.
outcomes
that
are
not
yet
triggered.
Con3nuous
Improvement
4.
Execute Auto-­‐Scrip3ng Expected
Results
...
You
add
detailed
Expected
Results
in
the
...
You
invest
a
lot
of
3me
in
calcula3ng
and
...
You
don’t
ask
(when
defects
that
the
tests
missed
are
found
tests.
-­‐
or
-­‐
...
You
forget
that
this
feature
documen3ng
Expected
Results
before
you
have
post-­‐tes3ng)
“What
input
could
have
been
added
to
the
test
exists
and
find
yourself
typing
out
test-­‐by-­‐
test
instruc3ons
one-­‐by-­‐one. determined
your
“final
version”
Parameters
and
plan
to
detected
this?”
“Should
I
add
that
input
to
the
Hexawise
Values.
Last
minute
addi3ons
to
inputs
will
jumble
test
plan
now
to
improve
it
in
advance
of
the
next
3me
it
is
up
test
condi3ons
for
most
test
cases. used?”
© Hexawise, 2010. All rights reserved. 17