We’ve all been there. We work incredibly hard to develop a feature and design tests based on written requirements. We build a detailed test plan that aligns the tests with the software and the documented business needs. And when we put the tests to the software, it all falls apart because the requirements were changed without informing everyone. Mary Thorn says help is at hand. Enter behavior-driven development (BDD), and Cucumber and SpecFlow, tools for running automated acceptance tests and facilitating BDD. Mary explores the nuances of Cucumber and SpecFlow, and shows you how to implement BDD and agile acceptance testing. By fostering collaboration for implementing active requirements via a common language and format, Cucumber and SpecFlow bridge the communication gap between business stakeholders and implementation teams. In this workshop, practice writing feature files with the best practices Mary has discovered over numerous implementations. If you experience developers not coding to requirements, testers not getting requirements updates, or customers who feel out of the loop and don’t get what they ask for, Mary has answers for you.
The Real-World Challenges of Medical Device Cybersecurity- Mitigating Vulnera...
Implement BDD with Cucumber and SpecFlow
1. MC
Half-day Tutorials
Monday, April 30th, 2018
8:30 AM
Implement BDD with Cucumber and
SpecFlow
Presented by:
Mary Thorn
Ipreo
Brought to you by:
350 Corporate Way, Suite 400, Orange Park, FL 32073
888---268---8770 ·· 904---278---0524 - info@techwell.com - http://www.stareast.techwell.com/
2. Mary Thorn
Ipreo
Chief storyteller of The Three Pillars of Agile Testing and Quality, Mary Thorn is
director of agile practices at Ipreo in Raleigh, NC. Mary has a broad agile and
testing background that spans automation, data warehouses, and web-based
systems in a wide variety of technologies and testing techniques. During her
more than nineteen years of experience with healthcare, HR, financial, and SaaS-
based products, Mary has held manager- and contributor-level positions in
software development organizations. A strong leader in agile and testing
methodologies, Mary has direct experience leading teams through agile adoption
and beyond.
3. 4/3/18
1
Identifier
Implement BDD
with Cucumber
or Specflow
Mary Thron
marythorn@gmail.com
What is the problem?
• Have you ever had overlooked requirements?
• Have you ever built the wrong thing?
• Do you have tangible reference materials?
• Have you ever interpreted a requirement incorrectly?
4. 4/3/18
2
The Communication Problem
The Challenge:
● Today, the most difficult problem in software
development is knowing what to build, not how to build
it.
● Knowing what to build, requires knowing why it is being
built. In other words, understanding the goal.
The Communication Problem
The Problem:
▪ Stories typically concentrate on the what and the how, not the
why. Goals and examples are usually missing.
▪ Stories are usually a list of imperative requirements (acceptance
criteria).
▪ Imperative requirements are usually interpreted in subtly
different ways by each person. If goals and examples are
absent, misinterpretation is much more likely.
5. 4/3/18
3
The Communication Problem
Example:
▪ Requirement: Draw a star with 10
points.
OR
The Communication Problem
Solution: Specification by Example
▪ People understand requirements best using
concrete examples.
▪ Examples clear up ambiguity and
misunderstandings.
▪ Examples expose missing requirements.
▪ Examples force everyone to think harder about
a problem.
6. 4/3/18
4
The Communication Problem
Specification by Example
▪ This is the basis of acceptance testing and
Behavior Driven Design (BDD).
▪ It is critical that everyone participates in
creating the examples so that the
understanding is shared.
▪ Ultimately, acceptance testing is less about
the example itself and more about the
conversation required to create the example.
Simple picture
7. 4/3/18
5
Simple Example
Acceptance Criteria: Transactions are
rounded to the nearest cent.
Original Value Rounded Value
$0.021
$0.025
$0.029
$0.02
$0.02
$0.02
Simple Example
Why was there confusion?
The goal was missing…
Goal: Create a currency conversion system.
The Concrete Example clarified what was meant by the
acceptance criteria.
Scenario Outline: Verify Transactions are rounded down to the
nearest cent.
Given I am on the order screen
When I make a <transaction> of values
Then the <values> are rounded
Examples:
|transaction|values|
|.021|.02|
|.025|.02|
|.029|.02|
8. 4/3/18
6
Simple Example
▪ This simple example is a REAL EXAMPLE in a system
that was well-tested and approved by all parties.
▪ Attacker was able to steal > $15,000 starting with a
single cent using this process...
▪ $0.01 → 0.0051 € rounded to 0.01 €
▪ 0.01 € → $0.0196 rounded to $0.02
▪ A seemingly minor ambiguity in the requirements was
very expensive.
Take Aways
▪ Concrete examples drive understanding and
consensus.
▪ To create good representative examples,
everyone must understand the goal of the
story.
▪ For technical people to properly understand
the goal, they must understand the business
domain.
10. 4/3/18
8
Automated Testing Process
Process:
▪ Product Owner writes the acceptance criteria and creates the
feature file per story.
▪ During grooming the team discusses the scenarios and asks
questions. Team can add/remove scenarios during grooming or
team updates feature file after.
▪ Feature file can be taken to planning to assist in task creation.
▪ Once Sprint has started QA and Dev work on getting the feature
file 90% complete before the developer starts coding.
▪ The developers code to the scenarios in the feature file, and
can do TDD if possible.
▪ QA expands the scenarios in the feature files after development
if there are new findings. Begins to automate in parallel with the
developer and do manual testing so they make it work in
parallel.
▪ When testing is complete PO acceptance is done by running the
acceptance tests.
Specflow Feature File
Feature File Overview:
l Written in plain text
l Define the test suite and contain the actual
tests which are called Scenarios
l Written collaboratively by everyone on the team
(PO, QA, and Dev)
l QA can run these manually.
11. 4/3/18
9
Specflow Feature File
Feature:
l Suite is defined by keyword “Feature:” followed
by a description. This states the goal for the
Feature File. When the Feature File is complete,
a tag is included at this level so that all of the
test Scenarios contained within the file can be
run from a single command.
Feature: This tests the currency conversion
system
Specflow Feature Files
Background:
l “Background:” is run before each test so that
you don’t have to repeat steps. Equal to a setup
step.
Background:
Given I login with the test account
And I navigate to the Transaction Screen
Then I should see the Transaction logo
12. 4/3/18
10
Specflow Feature Files
Scenario:
l Individual tests are defined by “Scenario:”
followed by a unique name. Scenarios are
concrete examples of how we want the
software to behave. The first line states the
goal of the given Scenario, while every line
thereafter lists the steps needed to complete
the test.
l Scenario: Verify Transactions are rounded
to the nearest cent.
Specflow Feature Files
Scenario Outline:
l “Scenario Outline:” is used for data driven tests.
Scenario Outline: Verify Transactions are rounded to the nearest cent.
Given I am on the order screen
When I make a <transaction> of values
Then the <values> are rounded
Examples:
|transaction|values|
|.021|.02|
|.025|.02|
|.029|.02|
13. 4/3/18
11
Specflow Feature Files
Keywords:
• Tests have reserved words that begin each line:
Given, When, Then, And, But.
•Use Given to state a precondition
•Use When & And when taking an action
•Use Then when asserting a validation
Scenario: Verify Transactions are rounded to the nearest cent.
Given I am on the order screen
AND(GIVEN)
When I make a transaction of 9.025
And(WHEN) save the transaction
Then the value should equal 9.02
AND(THEN)
Specflow Feature Files
Tags:
• Tags a powerful tool to organize tests and
scenarios. A Scenario or feature can have as
many tags as you like. Just separate them with
spaces.
• A note to the wise: Keep it simple.
@tag @smoke_test @component
14. 4/3/18
12
Imperative vs Declarative
è The imperative style uses highly reusable granular steps which outlines
much of the user interface.
Write tests in Declarative format
è Declarative style is more aligned with User Stories in the agile sense
having more of the "token for conversation" feel to it.
è The first thing that you should observe about this style is how much
smaller it is than the imperative one.
è The imperative style tends to produce noisy scenarios that drown out
the signal. With the declarative style the goal of the scenario remains
clear. When a new field is added to the form the scenario does not
have to be modified.
15. 4/3/18
13
Shorter tests
è Shorter tests are better. Omit any details from the scenario
that are not necessary for the business scenario being
validated.
è BAD
è GOOD
Verify only one thing at a time
è Try to minimize each scenario and test one thing at a time
è Try to limit to 1 Given and 1 Then per scenario
BAD
Scenario: Test A
Given I login
When I goto the main screen
Then I can start
Given I have started
When I create x
Then it is verified
GOOD
Scenario: Verify that I can start
Given I have logged in
When I have started
Then it is verified
16. 4/3/18
14
Max number of sentences 10
è Test scenarios should consist of 3 max 10 conditions.
BAD
GOOD
Scenarios must be independent.
è Scenario should run independently without any dependencies on other
scenarios.
BAD
Scenario: Test A
Given I login
When I goto the main screen
Then I can start
Scenario: Test B
Given I have started
When I create x
Then it is verified
GOOD
Scenario: Verify that I can start
Given I have logged in
When I have started
Then it is verified
17. 4/3/18
15
Group related scenarios together in a
feature
è Related scenarios should be grouped together in
features.
Bad:
Feature: Verify you can post/edit/delete status update. Verify you
can change you account permissions of who can see me. Verify I
can create a facebook event. Verify I can create a facebook group.
Good:
Feature: Verify you can post/edit/delete status update. Verify you
can post/edit/delete a picture. Verify you can “checkin”
#TFS123, #TFS125
Cover both the happy and non-happy paths
è Scenarios should cover both happy and non-happy paths. A happy
path is a straightforward user journey while a non-happy path covers
different edge cases around the happy path, including invalid inputs etc.
HAPPY
UNHAPPY
18. 4/3/18
16
Do NOT Hardcode configuration data
è Assume your test will run in all environments therefore
nothing should be hardcoded for a specific environment
BAD
Given I login to QX environment with user ‘mthorn’
GOOD
Given I login to environment ‘environment’ enter the user ‘user’
and the password ‘password’
Make sure Data Driven Tests are Readable
è No more than 5 columns and 10 rows for example, else
call in from Excel
BAD
Scenario outline: Free delivery Given the threshold for free delivery is 10 books And the customer is
<type> with <books> in the cart When checking out Then free delivery is <free delivery>
Examples
| type | books | free delivery |
| VIP | 9 | not offered |
| VIP | 10 | offered |
| VIP | 11 | offered |
| Normal | 10 | not offered |
| Normal | 9 | not offered |
| Normal | 10 | offered |
| VIP | 11 | offered |
| Normal | 10 | not offered |
| STD| 9 | not offered |
| STD | 10 | offered |
| STD | 11 | offered |
| Normal | 10 | not offered |
19. 4/3/18
17
Don't include technical stuff in your sentences
è If it looks like a programmer wrote it then it’s wrong.
BAD:
Given group entity for the brand pid available in the PIPS datastore.
GOOD:
Given BBC TV brand is available to public or coming soon
Feature versus Scenario Hooks*
è A Tag placed on a Feature will apply to all Scenarios within
that Feature
è BeforeFeature and AfterFeature are “Feature only” Hooks.
Their corresponding Tags can only be placed before the
Feature keyword.
20. 4/3/18
18
Tags
Keep It simple stupid
è Functional area
è CI run
• @smoke
• @sprint
• @regression
Hook Order*
BeforeTestRun
BeforeFeature
BeforeScenario or Before
BeforeScenarioBlock
BeforeStep
AfterStep
AfterScenarioBlock
AfterScenario or After
AfterFeature
AfterTestRun
21. 4/3/18
19
Horizon – Real World Example
The Challenge:
How to effectively communicate
requirements to the team, to be
developed / tested / implemented while
exceeding business expectations
Story Title: As a Horizon User, I want to access a page with detailed application specific information so that I can learn
more about the Horizon's offerings
Description:
In lieu of creating an Information page (as detailed in HZN-6805), we will instead provide a link to open a new browser for the applications in Internet
Explorer, similar to how we will handle the news items. The application specific information pages contain a detailed summary of Horizon’s offerings and
serve as a Marketing tool for potential users. Ultimately, launching this page directly is a simpler and cheaper solution that building a new window.
Acceptance Tests:
1.) An "I" icon is shown on each application tile, in the All Apps
view. This icon is not displayed in the My Apps view.
2.) If I am entitled to an application and it is live in Production, or if
the application is coming soon, or if I am not entitled to it - then
clicking the “I” icon will open the info page for that application.
3.) If I am entitled to an application and it is live in Production, then
clicking anywhere else in the tile (ie: NOT on the "i" icon) will open
the application.
4.) If the application is coming soon, then the following message is
shown when they click anywhere else in the tile (ie: NOT on the "i"
icon):
This app is not available yet
Please select the info icon to view more information on the Risk
Intranet
5.) If the user is not entitled to the application, then the following
message is shown when they click anywhere else in the tile (ie: NOT
on the "i" icon):
You do not have access to view this app
Please select the info icon to view more information and request
access on the Risk Intranet
6.) If the application does not have an associated info page, then
clicking the “I” icon will open the Horizon home page
(http://risk.intranet.db.com/content/horizon_coo_30587.html)
7.) All scenarios from feature file have been implemented correctly.
22. 4/3/18
20
Feature File
Example 1:
Verify clicking directly on the “I” icon launches a new internet
page when the user is permissioned to the application and it is
live in production
Given that I'm on the My Apps page
And I am permissioned to an application that is live in production
When I click the “I” icon on the application tile
Then a new browser window opens
Example 2:
Verify clicking directly on the “I” icon launches a new internet
page when the application is coming soon
Given that I'm on the My Apps page
And an application is coming soon
When I click the “I” icon on the application tile
Then a new browser window opens
And the All Apps page state is retained
Example 3:
Verify clicking in an area outside of the “I” icon (but within the tile)
launches the application when the user is permissioned to the
application and it is live in production
Given that I'm on the My Apps page
And I am permissioned to an application that is live in production
When I click on an area outside of the “I” icon, within the tile
Then the application opens
Benefits
▪ Clear Expectations Communicated
▪ Defined Scope
▪ Reveals Overlooked Requirements
▪ Tangible Reference Materials
▪ Accurate Story/Task Estimations
▪ Deliberate Collaboration Between QA/DEV/PO
▪ Increased Productivity
▪ Commitments are met (and often times exceeded!)
▪ Higher quality b/c everyone shares the same understanding of the requirements and builds it
right the first time.
▪ Tests become the system's regression suite.
▪ Create an objective verification of “done-ness” for a story. A story is done when all tests
pass.
▪ Create transparency into progress on a story.
▪ Manual testers are part of the automation process.
▪ Allows for more exploratory testing b/c happy path is automated
24. 4/3/18
22
Write the Acceptance Criteria
Keywords:
Tests have reserved words that begin each line:
Given, When, Then, And, But.
• Use Given to state a precondition
• Use When & And when taking an action
• Use Then when asserting a validation
Share
QUESTIONS?
