Traditional test plans are incompatible with agile software development because we don't know all the details about all the requirements up front. However, in an agile software release, you still must decide what types of testing activities will be required—and when you need to schedule them. Janet Gregory explains how to use the Agile Testing Quadrants, a model identifying the different purposes of testing, to help your team understand your testing needs as you plan the next release. Janet introduces you to alternative, lightweight test planning tools that allow you to plan and communicate your big picture testing needs and risks. Learn how to decide who does what testing—and when. Determine what types of testing to consider when planning an agile release, the infrastructure and environments needed for testing, what goes into an agile “test plan,” how to plan for acquiring test data, and lightweight approaches for documenting your tests and recording test results.
Boost Fertility New Invention Ups Success Rates.pdf
Agile Testing Guide by DragonFire
1. 1/22/2013
Star Canada 2013
Janet Gregory, DragonFire Inc.
Copyright 2013
With material from Lisa Crispin
My experience comes …
As a tester, working on agile teams
Coaching and training, learning
Agile Testing: A Practical Guide for Testers and Agile Teams;
Addison Wesley 2009
Copyright 2013 DragonFire Inc.
1
2. 1/22/2013
How many years experience with agile?
◦
◦
◦
◦
◦
At least 2 years
Less than 2 years
Less than 1 year
Have done some reading, starting in a team
Completely new to the concept
Are you a ...
◦
◦
◦
◦
Tester
Test Manager
Programmer
Other
Copyright 2013 DragonFire Inc.
3
10 minutes ~1 minute / person
Name
Where you are from?
Why you are here?
Let everyone have a chance to introduce themselves
Copyright 2013 DragonFire Inc.
4
2
3. 1/22/2013
Copyright 2013 DragonFire Inc.
•
•
•
•
•
•
5
have short iterations
encourage active customer participation
demand whole team collaboration
test features & stories as they are coded
deliver business value at regular intervals
adapt their processes based on feedback
Copyright 2013 DragonFire Inc.
6
3
4. 1/22/2013
Sequential, phased – eg. waterfall
Requirements
Specifications
Code
Testing
Release
Time
D
C
C
A
It 0
B
It 1
A
B
It 2
A
Agile: iterative, incremental
F
E
D
C
B
It 3
A
– Each story is expanded, coded and
tested
– Possible release after each iteration
B
It 4
7
What concerns you
about agile
and
test planning?
Copyright 2013 DragonFire Inc.
4
5. 1/22/2013
Test Approach – The Agile Way
Project Initiation
Release/Project
Planning
Each Iteration
1
….
X
System Test /
End Game
Release to Prod/
Support
Get an understanding of the project (business)
Participate in sizing stories
Create Test Plan
Estimate tasks, Run regression tests
Collaborate with customers on acceptance tests
Write, automate and execute new story tests
Pair test with other testers, developers
Perform exploratory testing
Perform Final Load Test
Complete Final Regression Test
Perform UAT
Perform Mock Deploy
Participate in Release Readiness
Participate in Release to Prod
Participate in Retrospectives
Copyright 2013 DragonFire Inc.
9
Project
Test Plan
Release
Test Plan
Copyright 2013 DragonFire Inc.
Story
Tests
10
5
6. 1/22/2013
• Product Roadmap
◦ High level feature ideas
• Release Planning
◦ Backlog of sized and prioritized stories
• Iteration Planning
◦ Backlog of estimated tasks
◦ Estimations are in hours: how long will it take
Copyright 2013 DragonFire Inc.
11
• Types of testing needed
• Automation
• Complexity
• Size
More details later …
Copyright 2013 DragonFire Inc.
12
6
7. 1/22/2013
Instead of saying NO,
or being the gatekeeper
Be the information provider so
business can make the decisions
Questions… on agile approach to planning
Copyright 2013 DragonFire Inc.
13
Copyright 2013 DragonFire Inc.
14
7
8. 1/22/2013
Let’s talk a little
testing ….
What is enough
to keep you out
of trouble?
Copyright 2013 DragonFire Inc.
15
• Find bugs
• Check correctness
• Test reliability
• Check usability
• Answer “Is it done?”
• Learn about the application
• Feedback into future stories
What else???
Copyright 2013 DragonFire Inc.
16
8
9. 1/22/2013
• Each group, collaborate
• Write different types of testing that you do
• One type per sticky
• 5 minutes
Copyright 2013 DragonFire Inc.
17
• Model for classifying tests
• Looks at the purpose of the tests – the ‘why’
Copyright 2013 DragonFire Inc.
18
9
10. 1/22/2013
Agile Testing Quadrants
Brain child of Brian Marick
19
• Take your tests and put them in the quadrant
you think they belong in.
• Were you able to get them all classified?
• Add tests as you think about them.
Copyright 2013 DragonFire Inc.
20
10
11. 1/22/2013
Agile Testing Quadrants
(Brian Marick)
21
• Can be used as a communication tool
o
to explain testing in a common language
• Emphasize whole-team responsibility
o
o
focus on collaboration
whole team participation
• Help plan what tests to automate
o
and tools and infrastructure needed
Copyright 2013 DragonFire Inc.
11
12. 1/22/2013
For each story / feature
◦ No story is done until tested
◦ Customer needs captured as passing tests
◦ Automated regression tests
For release readiness
◦ Delivers value
◦ “Doneness” in all quadrants
Copyright 2013 DragonFire Inc.
23
Copyright 2013 DragonFire Inc.
24
12
13. 1/22/2013
Unit Tests
o
o
o
o
Tests developer intent - program design
Tests a small piece of code
Makes sure it does what it should
TDD
Component Tests
o
o
Tests architecture intent – system design
Tests that components work together correctly
Connectivity Tests
Copyright 2013 DragonFire Inc.
• Builds quality in
• Focus on internal code quality
• Builds testability into code
• Go faster, do more
o
Unit tests provide refactoring support
• Provides instant feedback
• TDD increases confidence in design
• Forms the foundation of automation suite
Copyright 2013 DragonFire Inc.
13
15. 1/22/2013
• Use to elicit requirements
• Acceptance Test (or Example) Driven Development
o
Allows developers to code until the tests pass
• Capture examples, express as executable tests
• User experience
o
wire frames; mock-ups / prototypes
Copyright 2013 DragonFire Inc.
• Help customers achieve advance clarity
• Gain shared common understanding of stories
• Drive development with business-facing tests
• Obtain enough requirements to start coding
• Focus is external quality
• Customer – developer – tester collaboration
◦ Power of Three
Copyright 2013 DragonFire Inc.
15
16. 1/22/2013
Can you think
of anything else
for Quadrant 3?
Discussions?
31
Copyright 2013 DragonFire Inc.
32
16
17. 1/22/2013
• Exploratory testing
o
what didn’t we think about
• Test for usability
o
understand end users, personas
• Tours
• User Acceptance Testing (UAT)
Copyright 2013 DragonFire Inc.
• Iteration reviews / demos
o
Builds confidence
o
Quick feedback loop
• Pair test with customers
• Informal demos
o
Pair exploratory testing with customer
Copyright 2013 DragonFire Inc.
17
18. 1/22/2013
• Provide feedback ….
o
Turn learnings into tests that drive new features
o
Change process as needed
• Evaluation of the product
• Recreate actual user experiences
• Realistic use
Copyright 2013 DragonFire Inc.
• Know your customers
• Make them real
• Plan your exploratory
testing using them
• Picture – from Jeff Patton’s
Pragmatic Personas weekly column
on Sticky Minds (1/25/2010)
Copyright 2013 DragonFire Inc.
18
19. 1/22/2013
• Think of some personas or characters and
devise exploratory scenarios which that role
might get into, for example:
Shopping website - Amazon
o
Senior citizen who never shopped on the web
o
Hacker looking to cause trouble
o
Working mom in a rush
o
Internet-savvy teen
37
Copyright 2013 DragonFire Inc.
37
Any questions about
Quadrant 3?
38
19
20. 1/22/2013
39
Copyright 2013 DragonFire Inc.
• Non-functional tests
• Performance, scalability, stress, load
• “ility” testing
• Automated deployments
• Memory management
• Infrastructure testing
• Security testing
o
Roles & permissions , system ‘hacking’
• Data migration
• Recovery
Copyright 2013 DragonFire Inc.
20
21. 1/22/2013
• May be higher priority than functional req’ts
• Makes the ‘finished’ product
• Transfer specialized expertise
Consider the four quadrants
• Would you add anything?
• Consider your quality attributes
• Do you need to move any tests?
• What tests are you missing?
• What is your team is not doing now?
Copyright 2013 DragonFire Inc.
42
21
22. 1/22/2013
Automate at
the feature
level
push
the
tests
lower
Automate
at the story
level
Automate
at the task
level
Sequential, phased – eg. waterfall
Requirements
All
automation
done here
Specifications
Code
Testing
Release
Time
D
C
C
A
It 0
B
It 1
A
B
It 2
A
Agile: iterative, incremental
F
E
D
C
B
It 3
A
– Each story is expanded, coded and
tested
– Possible release after each iteration
B
Automation
here
It 4
Automation
Automation here
Automation
here
here
44
22
23. 1/22/2013
Instead of saying NO,
or being the gatekeeper
Be the information provider so
business can make the decisions
Questions… on types of testing?
Copyright 2013 DragonFire Inc.
45
Copyright 2013 DragonFire Inc.
46
23
24. 1/22/2013
Release Level Test Planning
Copyright 2013 DragonFire Inc.
47
• Consider scope, priorities, risks
• Document only what is useful
• Consider all types of testing
• Budget time for:
o
o
o
infrastructure
tools
automation needs
• Strive for simplicity
Copyright 2013 DragonFire Inc.
48
24
25. 1/22/2013
49
• Sizing of stories
◦ think about impacts to system
• Ask questions
◦ that may impact the ‘bigness’ of the story
◦ to uncover hidden assumptions
◦ such as ‘what if?’, or “what happens if?”
• Not the time for details
• Create a project “test plan”
Copyright 2013 DragonFire Inc.
50
25
26. 1/22/2013
• Are we working with a vendor?
◦ If so, how do we coordinate testing?
• Are there dependencies between …
◦ features?
◦ teams?
◦ Stories?
Copyright 2013 DragonFire Inc.
51
26
27. 1/22/2013
• Should be project specific
• High-light critical information
◦ risks
◦ assumptions
◦ constraints
• Focus on what is really needed
• Put static information –in a Quality
Management Strategy document
Copyright 2013 DragonFire Inc.
53
Copyright 2013 DragonFire Inc.
54
27
28. 1/22/2013
Copyright 2013 DragonFire Inc.
55
A test matrix (one example)
◦ provides a view of the release
◦ can also be used as visible progress
But, remember
◦ the value is in the planning
Let`s look at an example
Copyright 2013 DragonFire Inc.
56
28
29. 1/22/2013
Copyright 2009 Janet
Copyright 2013 DragonFire Inc.
Gregory, DragonFire
57
Let’s do some collaboration,
& work together to find a solution
Copyright 2013 DragonFire Inc.
58
29
30. 1/22/2013
rules
Sub topic
rules
user name
save
password
Sub topic
first time
Sub topic
encryption
MAIN
new account
TOPIC
Sub topic
change
Sub topic
Copyright 2013 DragonFire Inc.
59
Mind map testing ideas for a feature
How many ideas can you generate
in 5 minutes?
Copyright 2013 DragonFire Inc.
30
31. 1/22/2013
Feature A
• As an on-line shopper, I want to provide my
shipping address when I check out so that my
order goes to the right location
Feature B
• As an on-line shopper, I want to provide
payment information when I check out so I am
billed correctly.
Copyright 2013 DragonFire Inc.
• Create a test matrix for the stories on the next
slide.
• Functionality down the left side
• Test conditions across the top
o
Think heuristics
o
Use mnemonics (ex. SFDPOT)
• Gray out the squares that are not applicable.
Copyright 2013 DragonFire Inc.
31
32. 1/22/2013
Feature A
• As an on-line shopper, I want to provide my
shipping address when I check out so that my
order goes to the right location
Feature B
• As an on-line shopper, I want to provide
payment information when I check out so I am
billed correctly.
Copyright 2013 DragonFire Inc.
• What value do you see in something like this?
• What did you learn?
• How would you use it?
Copyright 2013 DragonFire Inc.
32
33. 1/22/2013
Instead of saying NO,
or being the gatekeeper
Be the information provider so
business can make the decisions
Questions on test planning at the release level?
Copyright 2013 DragonFire Inc.
65
Copyright 2013 DragonFire Inc.
66
33
34. 1/22/2013
Iteration Planning:
• Stories tell us about the tip of the iceberg
• What don’t we know?
• What questions should we ask to find out?
Copyright 2013 DragonFire Inc.
68
34
35. 1/22/2013
Create a
user story
Write
Customer
(Q2) Tests
Expand
tests –
Story Tests
Automate
Q2 Tests
Start
thinking
how to code
Pair,
“Show Me”
TDD
Product owner
Product owner/ Tester
Tester
Tester/Programmer
Programmer
Exploratory
testing
Customer
User
Acceptance
Copyright 2013 DragonFire Inc.
69
• Be proactive – preplanning
• Try specifications workshops – Gojko Adzic
• Help customer achieve ‘advance clarity’ on
stories
◦
◦
◦
◦
Customers speak with “one voice”
Testable stories
Steel threads
Create acceptance tests
Copyright 2013 DragonFire Inc.
70
35
36. 1/22/2013
• Define high level story tests or examples
• Find hidden assumptions
• Define and estimate testing tasks – consider:
o
o
o
o
automation needs
test data,
exploratory testing
Q4 tests (‘ilities’, security, performance, etc.)
Copyright 2013 DragonFire Inc.
71
Ask questions
• What's the business goal?
• Can the user mess up?
• What’s the best thing?
• What’s the worst thing that can happen?
• Watch for scope creep or ‘bling’
• Is this story testable?
Copyright 2013 DragonFire Inc.
72
36
37. 1/22/2013
To
Review
Copyright 2013 DragonFire Inc.
Picture from
Mike Cohn’s
website
73
73
•
•
•
•
Express the intent of the story
Use examples as specific instances of a scenario
Think expected and unexpected behaviors
Gives a shared common understanding of the
story
• Feeds into TDD (Test Driven Development)
Copyright 2013 DragonFire Inc.
74
37
38. 1/22/2013
As a new user, I want to create an
account with a user name and
password so that only I can access
my information.
===============================
Copyright 2013 DragonFire Inc.
75
75
• To create my account, I enter a valid user name
and password, my information is saved and I am
logged into the system and on the home page.
• If I enter an invalid user name, I get an error
message “Invalid User Name”, and I am able to
try again.
• If I enter an invalid password, I get an error
message “Invalid Password”, and I am able to try
again.
Copyright 2013 DragonFire Inc.
76
38
39. 1/22/2013
BDD – Behavioural Driven Development
Given the user has no existing account
When she requests to create a new account,
Then she enters a valid user name and valid
password (rules defined)
And the information is saved upon submitting .
77
Copyright 2013 DragonFire Inc.
77
returnValue TestLogin ( userName, password)
User Name
Password
Expected result
comments
JanetGregory
Password
Access system as
JanetGregory
Valid combo saved
Janet Gregory
Password
Error
Space in user name
JanetGregory
Abc
Error
Invalid password
Copyright 2013 DragonFire Inc.
78
39
40. 1/22/2013
• Write Acceptance Test(s) for one of the two
stories – any format you like
• To help, try mind-mapping or draw a flow
diagram
• Ask your customer for examples.
• Think “Amazon”
• (~15 min)
Copyright 2013 DragonFire Inc.
Story a – Part of Feature A
• As the company shipper, I need to verify the
city, state and postal code so that the order
goes to the right location.
Story b – Part of Feature B
• As the company accountant, I need to ensure
the credit card information is correct so the
on-line shopper is billed correctly.
Copyright 2013 DragonFire Inc.
40
41. 1/22/2013
We walk out of the iteration planning
meeting ....
Now what?
Copyright 2013 DragonFire Inc.
82
41
42. 1/22/2013
• Start simple with the high level acceptance tests
◦ Add boundary, edge conditions, etc.
◦ Add non-functional tests
• Experiment to find the right detail level
• Review with programmers
◦ Then automate
• Keep adding until story is complete
• Consider exploratory test scenarios
Copyright 2013 DragonFire Inc.
83
Acceptance Tests – Fit Automation Style
-- Third thread
User Name
Password
Expected result
Comments
JanetGregory
Password
Access system as
JanetGregory
Valid combo saved
Janet Gregory
Password
Error
Space in user name
Janet#Gregory
Password
Error
Special char not
allowed
Password
Error
Blank user name
JanetGregory
Password
Error
User already exists
JanetGregory
Abc
Error
Not enough char in
password
Error
Blank password
JanetGregory
Copyright 2013 DragonFire Inc.
84
42
43. 1/22/2013
How do you know
you are done?
Copyright 2013 DragonFire Inc.
Description
85
Completed
1
Story tests reviewed, automated and pass
Yes
2
Exploratory testing complete
Yes
3
Unit tests reviewed and pass
Yes
4
Q4 tests complete
Yes
5
Acceptance tests pass
Yes
Copyright 2013 DragonFire Inc.
86
43
44. 1/22/2013
Instead of saying NO,
or being the gatekeeper
Be the information provider so
business can make the decisions
Questions on test planning at the story level?
Copyright 2013 DragonFire Inc.
87
Copyright 2013 DragonFire Inc.
88
44
45. 1/22/2013
• Understand your context
• Understand the purpose
• Consider ROI (return on investment)
• Push the tests lower
• Automate the repetitive and boring tests
• Plan, but document simply
• Plan, but plan for the appropriate level
Copyright 2013 DragonFire Inc.
• How much is too much?
• What is not enough?
• Why do you need them?
• What is the right information?
• Who looks at them?
• What do they use them for?
• What is the simplest way you can capture
them?
Copyright 2013 DragonFire Inc.
90
45
46. 1/22/2013
•
•
•
•
•
•
•
•
What problem are you trying to solve?
What measurements make sense?
What is the simplest way to gather them?
Who needs to see them?
What is the simplest way to make them visible?
How are you going to use them?
When will you know to stop?
Be Aware!!!
o
o
The wrong measurements can be deadly
Metrics can be misinterpreted
Copyright 2013 DragonFire Inc.
• How are you going to know when you are done?
• What is enough?
• Collaborate to decide what coverage you want
• Think risk – include the customer
• Functional coverage
• Lines of code? … be careful
• What tools can you use?
o
o
Heuristics
Mnemonics
Copyright 2013 DragonFire Inc.
46
47. 1/22/2013
• Understand the level of precision you need
◦ Think big picture in release planning
◦ Think tasks during iteration planning
◦ Think tests during story test planning
• Use ATDD for feature and story testing
• Make it visible
• Make it simple
• Make it valuable
• You want to be testing, not documenting
Copyright 2013 DragonFire Inc.
Instead of saying NO,
or being the gatekeeper
Be the information provider so
business can make the decisions
Are there still unanswered questions.. Concerns?
Copyright 2013 DragonFire Inc.
94
47
48. 1/22/2013
Now Available
Agile Testing: A Practical Guide for
Testers and Agile Teams
By Lisa Crispin and Janet Gregory
www.agiletester.ca
My contact info
www.janetgregory.ca
http://janetgregory.blogspot.com/
Email: janet@agiletester.ca
www.lisacrispin.com
http://lisacrispin.com
Email: lisa@agiletester.ca
Copyright 2013 DragonFire Inc.
95
•
•
•
•
•
•
•
•
www.lisacrispin.com
agile-testing@yahoogroups.com
http://www.exampler.com - Brian Marick’s web site
www.testobsessed.com (heuristics cheat sheet) – Elisabeth Hendrickson
http://www.shino.de/blog/ - Markus Gärtner’s blog
Gojko Adzic, Bridging the Communication Gap, 2009
Gojko Adzic, Executable Specifications, Manning, 2011
vwww.mountaingoatsoftware.com – Mike Cohn’s web site (and all his
books)
• http://www.developsense.com/2009/04/of-testing-tours-anddashboards.html
• James Whittaker, Exploratory Software Testing, 2009 Addison-Wesley
• http://www.stickyminds.com/ Pragmatic Personas - Jeff Patton’s weekly
column 1/25/2010
• Jean Tabaka, Collaboration Explained, 2006 Addison-Wesley
• Copyright 2013Manifesto: http://agilemanifesto.org/
Agile DragonFire Inc.
96
48