Model-based Testing: Taking BDD/ATDD to the Next Level

Model-based Testing:
Taking BDD/ATDD to the Next Level
Robert V. Binder
System Verification Associates
rvbinder@sysverif.com
#MoreModelsLessTests

Chicago Quality Assurance Association
February 25, 2014

The Incredible
Vastness of
Testing

ModelBased
Testing

Eaten Alive by the
Testing Backblob

Tell Me
A Story

More Models,
More Tests
Less
Models,
Less
Tests

Model Driven Development: Taking BDD/ATDD to the Next Level

1

The Incredible
Vastness of Testing

Saved by Revealing Sub-domains
• Grains of sand to fill the
Grand Canyon
~ 1.68 × 1025
~ 10 septillion

• Trivial Program
–
–
–
–

Data space: 21024
Sequences: 21024
Execution space: 22048
1.34 × 10616

Large subsets of any program’s execution space are revealing: if a bug exists in any
member of that space and it is executed, an observable failure results.

But, we rarely know exactly which sub-domains are revealing until we test.

3

It’s release time.
Do you know where your bugs are?
Aerospace
Medical

IT

0.7
0.8

Other

Agile

1.9
1.8
4.4

5.0

6.0
6.0

Mobile
Best In Class

Average bugs per
thousand
equivalent source
code lines,
post-release

4.7
Capers Jones’ statistics: 5 defects/FP, 95% Removal. Assumes 53 Java LOC/FP .


4

It’s release time.
Do you know where your bugs are?

Aerospace
Medical
IT

Best In Class

Other

325
375

Agile

Predicted total
latent bugs,
500 KLOC Java

925
875
2511
2202

2350
Capers Jones’ statistics: 5 defects/FP, 95% Removal. Assumes 53 Java LOC/FP


5

Driven to Distraction
• Test Driven Development
– TDD

• Behavior Driven Development
– BDD

• Acceptance Test Driven Development

• What is it?
• How does it work?
• What can it do for
me?

– ATDD

• Model-based Testing
– MBT

6

Eaten Alive
by the
Testing
Backblob
EATEN ALIVE BY THE TESTING
BACKBLOB

Where does the time go?
Total
Number
of Test
Cases

Sprint 1

Sprint 2

Sprint 3

Sprint 4

}

Tests/testing for
new features
Existing manual
Test Cases not
executed

Available
Dev/Test
Time

Existing automated
Test Cases not
maintained

8

The Attack of the BackBlob

http://www.youtube.com/watch?v=GODDLgM1gKo

Tell Me a Story

TELL ME A STORY

The Chat Server
•
•
•
•
•

Logon
List active users
Post a message
Broadcast message
Logoff

!!!
Client
Endpoint

Client
Endpoint


Chat
Server
11

TDD – BDD – ATDD
• Test Driven
Development

• Behavior Driven
Development
• Acceptance Test Driven
Development

• Programming strategies
– Test suites are scaffolding
– Test each method/feature at
least once

– Rerun after any add/change

• Not testing strategies


12

The TDD Red-Green Cycle
• Imagine a class and its
responsibilities
• Code its API only

Logon
Logon

GetStatus
GetStatus

Logoff
Logoff


13

The Red-Green Cycle
responsibilities



TestLogon

• Write and run a simple test – it
should fail

Logon

GetStatus

Logoff


14

The Red-Green Cycle
responsibilities



TestLogon

should fail

Logon

GetStatus

• Code implementation
• Rerun all tests – expect pass


Logoff

15

The Red-Green Cycle
responsibilities
should fail






TestLogon
Logon
TestGetStatus

GetStatus


Logoff

• Repeat until feature complete

16

The Red-Green Cycle
responsibilities
should fail





TestLogon
Logon
TestGetStatus

GetStatus





TestLogoff

Logoff

• Repeat until feature complete

17

TDD Limitations
Process,
Activities

Components,
Test Objects

• No explicit linkage for
user/customer needs,
requirements, or
stories

• Gaps
– UI automation
– Feature testing

18

Behavior Driven Development
Process,
Activities

Components,
Test Objects
Features,
Scenarios
Feature Files

 Scenario A

*DD Tool

 Scenario B
 Scenario C

Test Objects
*Unit
Components

19

Acceptance Test Driven Development
Process,
Activities

Features,
Scenarios

UI Test
Cases
Feature Files

 Scenario A

*DD Tool

 Scenario B
 Scenario C

Test Scripts
Test Objects
Test Tooling
*Unit
System
Under Test

20

Features and Scenarios
Feature: story
Scenario: instance of a story
Steps:
Given: pre-condition
When: trigger
Then: verification condition

Feature: Logon
In order to use the chat server,
As a validated chat client,
I want to submit my credentials and
establish a session
Scenario: Submit Good
Given A connection to
When I send the Login
Then I should see the


Credentials
the server is open
message
"Hi Chatter" message

21

Overview
Feature Files

 Scenario A

SpecFlow

 Scenario B
 Scenario C

Test Objects
Nunit
Visual Studio
Components

http://www.specflow.org

• Enter features, scenarios
• Generates skeleton test
code for each step
– Extracts step names
– Can run immediately

• Developer adds step code
• Test run reports
• CI integration, more


22

1. Create a
Feature File
• SpecFlow
generates a stub
file
• Edit as necessary
• One feature
• 1:N scenarios


23

2. Enter Scenario
Steps
• Given
– Precondition
– Data State

• When
– Trigger
– API call, GUI action

• Then
– Verification
conditions
(expected results)

24

3. Inspect Skeleton
Test Code

• SpecFlow parses
feature file
• Generates
skeleton test
code for each
scenario and step


25

3. Add Test Case
Code
• User adds more
code as needed
to implement
each test case


26

4. Run Test Suite
• The generated
test code can be
run immediately
• Supports the redgreen cycle


27

SpecFlow
•

BDD for Visual Studio
• Link Feature Files to test code
• Generates partial test code

Feature File contains statements in
Gherkin
–

Keywords + natural language

•
•

Generates test code skeleton from
steps

•

Feature
File

1:N Steps per Scenario

•

Feature

1:N Scenarios per Feature File

Steps may be parameterized

Steps

Rows

•

Variables and variable bindings in Rows

–

Scenarios

–

Test code generation using regular
expression patterns

Integration with TFS, CI, …


28

Modelbased
Testing
THE VERY MODEL OF A MODERN …

What is a Model?
• Focus
• Smaller, simpler
• In IT systems
– Behavior
– Structure
– Content

30

IT Models
Process,
Activities

Behavior,
Structure,
Content


Technology

31

IT Models
• Notional (design)
• As built (documentation)
• Test Models
– Similar to notional and as-built
– Unique aspects

Test Models
• Focus on aspects that
support testing goals
• Behavior
– Sequence and time

• Transformation
– Input, output function

• Verification (oracle)
• Coverage

• Risk Management

33

Chat Server
• Logon

Chat Server

– Must be first
– Can’t be repeated

Logon
GetStatus
Logoff

• GetStatus
– Only after logon
– Returns number of calls since
logon

• Logoff
– Only after logon

• Typical RFC status codes

34

Simple Test Model
Logoff

nc = 0
GetStatus

System Under
Test (SUT)

Logon

Logon

Logon

GetStatus

assert (200)

assert (401)

assert (401)

assert (401)

Logoff

nc++
GetStatus

Logoff
assert (200)

Logoff
assert (200)

assert (200)
assert (n == nc)
35

Chat Server Message Sequence
Login a 1234
Hi Chatter a
Login b secret
Hi Chatter b
List Users
Users: a, b
!!!
Client Endpoint

Client Endpoint


Chat Server

36

Chat Server Message Sequence
List Users
Users: a, b
Post “Pumpkin Latte #mypics”
Ack a1 “Pumpkin Latte #mypics”

Bcast a1 “Pumpkin Latte #mypics”
Logoff b
Ack Logoff b

!!!
Client Endpoint

Client Endpoint


Chat Server

37

Spec Explorer Demo
• Model-based testing tool
– Microsoft Research

– Used to test all MSFT APIs
– Robust and stable
– Visual Studio “power tool”

• C# code, not cartoons
• Generates standalone
executable test suite

38

1. Create Test
Model Actions
• [Rule] tags a
method for
exploration
• Requires defines
a exploration
precondition –
must be true to
explore

39

2. Define Accepting
State(s)
• Tag condition(s)
that terminate
an exploration


40

3. Create Cord File
• Set options
• Define machines
– Limits Actions used
in an exploration

• Similar to regular
expression
• Composable
• Very powerful!

41

4. Explore the Model
• Select any
explorable machine
– Explore
– Generate test suite
– Run on-the-fly test


42

5. Review
Exploration Graph
•

Shows action/event
sequence

•

Ellipse: model state
computed by model
program machine

•

Diamond: observation
“event” – get something
from SUT (input to the
model program)

•

Right click to inspect
any item


43

6. Define machine(s)
to generate test
suites

• Set options to
generate test
code
• Use any machine
or composite
machine


44

7. Explore the Test
Suite machine
• Each root to leaf
path becomes a
test suite


45

8. Generate Test Code
• Explores the test
machine
• Writes out test
code using VS
testing
framework


46

8. Generate Test Code
• Not intended to
be maintained
• Re-generate any
time
• Add helper code in
the model to
extend/customize

47

9. Run the Test Suite
•

Uses built-in Visual
Studio test runner

•

Adapters
– Start/stop chat clients
and chat server

– Other SUT
setup/cleanup

•


Test code sends
messages, checks
replies
48

Model State
(test case inputs)

10. Review Test
Results
• Detailed run log
–
–
–
–

Event Handler
(pass/fail)
Requirement
Trace

VS test runner
States
Events
Requirements

• Add any other
programmable
function


49

Spec Explorer
Exploration
• Interactive model development
• Generate test suites

•

Machines

Cord
Declarations
Model
Program

Options

Machines define composable
Action sequences

•

Options control exploration and
data generation

•

Actions are explored
– Rules define exploration conditions

Model Data
Structures

Rules
Actions
Events
Adapters

– Events define expected SUT
response

•

Data structures support exploration

•

Adapters abstract, control, and
observe any interface, any platform


50

MBT Driven Development
Requirements, Stories, Scenarios...
Explore

Ambiguous, missing,
contradictory, incorrect,
redundant, incomplete

Test Model
Run

Generate

Model error, omission

Test Suite
Inputs
(Test Sequences)

Control

Missing,
incorrect
Coverage
Requirements
Model

Evaluate

Expected Outputs
(Test Oracle)

SUT

Observe


Bug
51

More Models, Less Tests

MORE MODELS, LESS TESTS

Model-based Testing in the Real World
• Windows protocol compliance (Web services, RPC, DCOM)
• Total effort: 250 person years (mostly junior SDETs)
• Saved 50 person years with model-based testing
Model-based Testing
1.4 Days/Requirement

42% Less
Time Per
Requirement

Traditional Testing
2.4 Days/Requirement
Requirements Study

Modeling

Test Coding

Adapter Coding

Model Driven Development: Taking
BDD/ATDD to the Next Level

Test Execution
53

Scope of Test Asset Maintenance
Model-based Testing

Behavior Driven Development

Model
How much MBTgenerated test code
must be maintained?

Test Code

None!

Adapters

54

More Models, Less Tests
• More Better Testing

• Icing the BackBlob

– Rule-based

– Maintain model

– Systematic and deep
exploration

– Regenerate test suites

– Generate 1000s of
scenarios in seconds

– Models grow much
more slowly than test
code


55

Icing the Backblob

http://www.youtube.com/watch?v=0GEJgR-bX0w

Q&A
#MoreModelsLessTests
rvbinder@sysverif.com


57

Learn More - MBT
•

UCCAT (User Conference on Advanced Automated Testing) Presentations

http://ucaat.etsi.org/2013/program_conf.html
•

2012 Model-based Testing User Survey

http://robertvbinder.com/wp-content/uploads/rvb-pdf/arts/MBT-User-Survey.pdf
•

Open Source MBT tools

http://www.robertvbinder.com/open-source-tools-for-model-based-testing/
•

Overview of MBT, list of commercial MBT tools

http://www.slideshare.net/robertvbinder/model-basedtestingignite
•

Spec Explorer

http://visualstudiogallery.msdn.microsoft.com/271d0904-f178-4ce9-956b-d9bfa4902745
http://msdn.microsoft.com/library/ee620411.aspx

•

Microsoft Open Protocol Project

http://cacm.acm.org/magazines/2011/7/109908-microsofts-protocol-documentation-program/fulltext#!

58

Learn More: BDD, etc.
•

SpecFlow

http://www.specflow.org/
•

Using SpecFlow with WatiN

http://msdn.microsoft.com/en-us/magazine/gg490346.aspx
•

SpecFlow Quick Start

http://www.youtube.com/watch?v=SVwBz-ZTliw
•

Writing Cucumber Feature Files

https://github.com/cucumber/cucumber/wiki
•

Using Lava Lamps to indicate Build Status

https://wiki.jenkins-ci.org/display/JENKINS/Hudson+Build+Status+Lava+Lamps


59

Content and Image Sources
Grand Canyon Time Lapse. Ezra Kamer
http://www.youtube.com/watch?v=NF6O6HuNJ0k

Contrast (Order and Chaos). M. C. Escher, 1950. National Gallery of
Canada
http://www.gallery.ca/en/see/collections/artwork.php?mkey=29270

The Blob, Theatre Scene. Fairview Productions, 1958.
http://www.youtube.com/watch?v=GODDLgM1gKo
http://www.imdb.com/title/tt0051418/

Order and Chaos II. M. C. Escher, 1955. National Gallery of Canada
http://www.gallery.ca/en/see/collections/artwork.php?mkey=41980

Defect densities:
• Capers Jones, Software Quality: Analysis and Guidelines for Success
• Robert E. Park. Software Size Measurement: A Framework for
Counting Source Statements. Technical Report CMU/SEI-92-TR-020.
Software Engineering Institute, Carnegie Mellon University

Spec Explorer. Microsoft
http://visualstudiogallery.msdn.microsoft.com/271d0904-f178-4ce9956b-d9bfa4902745

SpecFlow
http://www.specflow.org/

The Blob, Final Scene. Fairview Productions, 1958.
http://www.youtube.com/watch?v=0GEJgR-bX0w

Spec Explorer Chat Model. Microsoft.

All other content Copyright © Robert V. Binder, 2014.


60

Model-based Testing: Taking BDD/ATDD to the Next Level

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (6)

Similar to Model-based Testing: Taking BDD/ATDD to the Next Level

Similar to Model-based Testing: Taking BDD/ATDD to the Next Level (20)

More from Bob Binder

More from Bob Binder (20)

Recently uploaded

Recently uploaded (20)

Model-based Testing: Taking BDD/ATDD to the Next Level