Manual functional testing is a slow, tedious, and error prone process. As we continue to incrementally build software, the corresponding regression test suite continues to grow. Rarely is time allotted to consolidate and keep these test cases in sync with the product under development. If these test cases are used as the basis for automation, the resulting suite is composed of very granular tests that are often quite brittle in nature. Using a case study, Anand Bagmar describes how behavior-driven testing (BDT) can be applied to identify the right type of test cases for manual and automated regression testing. Learn how the BDT technique can be applied in your context and domain, regardless of the tools and technologies used in your project and organization.

1. T14
Test Techniques
5/8/2014 1:30:00 PM
Build the Right Regression Suite
with Behavior-Driven Testing
Presented by:
Anand Bagmar
ThoughtWorks
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

2. Anand Bagmar
ThoughtWorks
Anand Bagmar is a hands-on and result-oriented software quality evangelist with seventeen
years in the IT field. Passionate about shipping quality products, Anand specializes in building
automated testing tools, infrastructure, and frameworks. He writes testing-related blogs and has
built open-source tools related to software testing-WAAT (Web Analytics Automation Testing
Framework), TaaS (for automating integration testing in disparate systems), and TTA (Test
Trend Analyzer). Anand is the lead organizer for vodQA, the popular testing conference in India.
Follow him on Twitter @BagmarAnand, email him at abagmar@gmail.com, or read his Essence
of Testing blog.

3. Building
the
“right”
regression
suite
using
Behavior
Driven
Tes5ng
(BDT)
Anand
Bagmar
So#ware
Quality
Evangelist
Principal
Consultant,
ThoughtWorks

4. • “Succeeding
with
Agile”
–
Mike
Cohn
• MarCn
Fowler
–
Test
Pyramid
– hFp://marCnfowler.com/bliki/TestPyramid.html
The
Test
Pyramid

5. Func%onal
Tests
(GUI)
• Developers
&
/
QA
team
Web
Service
Tests
• Developers
&
/
QA
team
Integra%on
tests
• Developers
Component
tests
• Developers
Unit
Tests
• Developers
Test
AutomaCon
Pyramid
Manual
/
Exploratory
TesCng
Product
under
test
Ideal
Test
Pyramid
Business-­‐
facing
Tests
Technology-­‐
facing
Tests
• Cost
• Effort
• Time

7. Test
Pyramid
reality
Development
team
Test
Pyramid
QA
team
Test
Pyramid

8. Func%onal
Tests
(GUI)
• Developers
&
/
QA
team
Web
Service
Tests
• Developers
&
/
QA
team
Integra%on
tests
• Developers
Component
tests
• Developers
Unit
Tests
• Developers
Manual
/
Exploratory
TesCng
Test
Ice-­‐cream
cone
Business-­‐
facing
Tests
Technology-­‐
facing
Tests
Test
Pyramid
–
AnC
PaFern
Product
under
test

9. Func%onal
Tests
(GUI)
• Developers
&
/
QA
team
Web
Service
Tests
• Developers
&
/
QA
team
Integra%on
tests
• Developers
Component
tests
• Developers
Unit
Tests
• Developers
Manual
/
Exploratory
TesCng
Product
under
test
Ideal
Test
Pyramid
Business-­‐
facing
Tests
Technology-­‐
facing
Tests
IdenCfy
the
right
type
of
tests
that
sit
on
top
of
the
Test
Pyramid

10. Case
Study
• MulCple,
long
running
projects
• Legacy
applicaCons
• Integrated

11. • Limited
AutomaCon
• Long
regression
cycle
• Huge
cost
of
fixing
defects
TesCng
Challenges

12. • No
visibility
into
what
is
tested
• Outdated
• BriFle
• LiFle
/
less
value
• Expensive
• Maintenance
nightmare
As
a
result

13. Biggest
problem
• Trust
deficit
on
the
team
– In
exisCng
AutomaCon
– In
manual
tesCng
• Finger
poinCng
/
blame
game

16. • Remove
the
ambiguity
• Same
visibility
for
all

17. EvoluCon
of
Test
Frameworks
Behavior
Driven
Development
(BDD)
OpCmized
scripts
Record
&
Playback
Core
Framework

18. Behavior
Driven
Development
(BDD)

19. RED
GREEN
REFACTOR

20. Behavior
Driven
TesCng
(BDT)

21. The
Big
Picture
• How
is
this
funcConality
going
to
be
used
by
the
end-­‐user?
• What
is
their
thought
process
going
to
be
when
using
this
product?
• What
is
the
“core-­‐business-­‐value”
I
am
delivering
to
the
end-­‐user?

22. Func%onal
Tests
(GUI)
• Developers
&
/
QA
team
Web
Service
Tests
• Developers
&
/
QA
team
Integra%on
tests
• Developers
Component
tests
• Developers
Unit
Tests
• Developers
Manual
/
Exploratory
TesCng
Product
under
test
Ideal
Test
Pyramid
Business-­‐
facing
Tests
Technology-­‐
facing
Tests
IdenCfy
the
right
type
of
tests
that
sit
on
top
of
the
Test
Pyramid

23. Expected
Func%onality:
The
customer
should
be
able
to
withdraw
money
from
his
account
via
an
ATM
machine
An
example

24. Given
the
account
has
balance
of
INR
5000
When
the
customer
requests
INR
1000
Then
the
account
is
debited
by
INR
1000
BDD
Example:

25. Given
the
account
is
in
credit
And
the
dispenser
contains
cash
When
the
customer
requests
cash
Then
ensure
the
account
is
debited
And
ensure
cash
is
dispensed
And
ensure
the
card
is
returned
BDT
Example:

26. New
Expected
Func%onality:
The
customer
should
be
able
to
withdraw
money
from
his
account
only
if
his
card
is
valid

27. Given
the
account
is
in
credit
And
the
card
is
valid
And
the
dispenser
contains
cash
When
the
customer
requests
cash
Then
ensure
the
account
is
debited
And
ensure
cash
is
dispensed
And
ensure
the
card
is
returned
BDT
Example
-­‐
updated:

28. Test
specificaCon
styles
• ImperaCve
• DeclaraCve

29. • Write
user
flows
based
on
personas,
and
how
they
use
the
business
funcConality,
or,
• Write
business
flows,
and
how
different
user
personas
can
use
that

30. Another
example
For
Indigo
Airlines
website
–
• Guest
User
is
able
to
search
for
a
flight
for
a
single
traveller
• Ability
to
specify
contact
informaCon
for
person
booking
the
flight

31. Search
Search
Results
Contact
Informa%on

32. ImperaCve
style
Given
I
am
a
guest
user
on
the
Indigo
home
page
And
I
select
“round”
trip
opCon
And
I
select
“Pune”
from
the
origin
dropdown
And
I
select
“Bangalore”
from
the
desCnaCon
drop
down
And
I
select
departure
date
as
“5
December
2013”
And
I
select
returning
date
as
“25
December
2013”
When
I
click
on
Search
Then
I
should
see
the
search
results
page
And
I
should
see
at
least
1
opCon
for
my
criteria
...

33. …
When
I
select
the
“first”
opCon
Then
I
am
on
the
Contact
InformaCon
page
When
I
enter
first
name
as
“foo”
And
I
enter
last
name
as
“bar”
…
…
And
I
click
the
“Select
and
Con%nue”
buFon
Then
I
should
be
on
the
next
page

34. Given
I
am
a
guest
user
When
I
search
for
flight
opCons
for
a
“one-­‐way”
trip
for
“1”
“Adult”
from
“Delhi”
to
“Bangalore”
And
I
select
the
“first”
flight
And
I
enter
“valid”
contact
details
for
“traveller1”
Then
I
am
able
to
Save
and
ConCnue
DeclaraCve
style

35. What’s
next?

36. DO
NOT
AUTOMATE
• Low
Value,
High
Cost
• Manual
Regression!
2nd
candidate
for
automa%on
• High
Value,
High
Cost
Automate
ONLY
if
you
have
%me
• Low
Value,
Low
Cost
• ?
Manual
Regression?
1st
candidate
for
automa%on
• High
Value,
Low
Cost
Cost
Value

37. Func%onal
Tests
(GUI)
• Developers
&
/
QA
team
Web
Service
Tests
• Developers
&
/
QA
team
Integra%on
tests
• Developers
Component
tests
• Developers
Unit
Tests
• Developers
BDT
helps
iden5fy
the
“right”
type
of
regression
tests!
Manual
/
Exploratory
TesCng
Product
under
test
Ideal
Test
Pyramid

38. A
real
world
Case
Study

40. How
did
we
accomplish
this?

42. Who
is
doing
the
acCon?
What
is
the
business
funcConality?
AcCons
and
verificaCons
happen
implicitly
Depicts
a
user
flow
Given
auc%oneer
creates
a
sale
And
5
items
are
added
to
the
sale
When
auc%oneer
starts
the
sale
And
auc%oneer
sets
a
starCng
bid
of
5000
And
buyer_1
bids
And
buyer_2
bids
Then
auc%oneer
Sells
the
item
And
buyer_2
wins
the
item

43. Final
state
Tests
running
on
Hudson
/
Jenkins
Since
1st
test
was
automated
Feature
files
33
Scenarios
65
Smoke
tests
execuCon
Cme
15
minutes
Regression
tests
execuCon
Cme
45
minutes
Test
case
repository
(manual
+
automated)
Cucumber
.feature
files
with
appropriate
tagging
Project
Management
tool
Mingle

44. ?
Incrementally
build
User
Flows
Regress
required
Product
Business
Value
Tests
in-­‐sync
EffecCve
Tests
Remove
ambiguity
Understand
new
funcConality
Living
DocumentaCon
Onboarding

45. Tools
to
enable
BDD
/
BDT
• Cucumber
• JBehave
• SpecFlow
• Twist
• Concordion

46. abagmar@gmail.com
@BagmarAnand
about.me/anand.bagmar
Thank
you