Software Testing with Agile Requirements Practices
1. Software Testing with Agile
Requirements Practices
Dr Syed Akhter Hossain
Fall 2012
Copyright 2003-2005, Rally Software Development Corp
2. Agenda
•Context for Agile Testing
•Technical Challenges
•Organizational Challenges
•Keys for Success
Copyright 2003-2005, Rally Software Development Corp
3. Popular Agile Methods
Dynamic System Development XP (Kent Beck)
Method (Dane Faulkner)
Adaptive Software Development Lean Software Development
(Jim Highsmith) (Mary Poppendieck)
Crystal (Alistair Cockburn) Feature Driven Development (Jeff
DeLuca)
Scrum (Ken Schwaber) Agile Rational Unified Process
(RUP)
Copyright 2003-2005, Rally Software Development Corp
4. Excerpts from the Agile Manifesto
• Our highest priority is to satisfy the customer through early and continuous
delivery of valuable software.
• Welcome changing requirements, even late in development. Agile processes
harness change for the customer's competitive advantage.
• Working software is the primary measure of progress.
• Deliver working software frequently, from a couple of weeks to a couple of
months, with a preference to the shorter timescale.
• Business people and developers must work together daily throughout the
project.
• Build projects around motivated individuals. Give them the environment
and support they need, and trust them to get the job done.
Copyright 2003-2005, Rally Software Development Corp
DSAH@Fall 2012 4
6. Agile Iteration Cadence
Requirements
Are Refined
Dev Feature Dev Feature
Accept
Accept
Priority 1 Priority 4
Auto. Tests Auto. Tests
Planning & Design
Detailed Iteration
Feature 1 Feature 4
Demo & Retro
Initial Elaboration Dev Feature
Dev Feature
Requirements
Accept
Accept
Priority 2 Priority 5
With Tests
Auto. Tests Auto. Tests
Feature 2 Feature 5
Dev Feature
Accept
Priority 3
Auto. Tests
Feature 3
Iteration N-1 Iteration N Iteration N+1
Copyright 2003-2005, Rally Software Development Corp
DSAH@Fall 2012 6
7. What’s Different about Testing in Agile?
• Just-In Time Requirements Elaboration
– No SRS-level waterfall documents to drive testing plan
– Requirements and Test Cases developed in parallel or test
first strategy
• More Frequent Iterations, More Frequent Releases
– Testing needs to happen Early and Often
– Frequent to continuous regression testing
– High need to automate nearly everything
– Everyone needs to Test
• Two Levels of Testing
– Iteration Vs. Release testing patterns
Copyright 2003-2005, Rally Software Development Corp
DSAH@Fall 2012 7
8. Technical Challenges
•Requirements are changing fast. How does test keep up?
•Test early and often. How exactly do we move testing forward?
•Need to move off manual testing and more into automation. How does this
happen?
•Different kinds of testing need to happen at different times. How do these
get managed?
Copyright 2003-2005, Rally Software Development Corp
9. Requirements are Changing
Code Code Code
& Deliver & Deliver & Deliver
s
date
Up
Pass
UC/SR Fail TCs & Accept
Up
da
te
s
Generate Update Run
TCs TCs TCs
Copyright 2003-2005, Rally Software Development Corp
DSAH@Fall 2012 9
10. Requirements Changing is a Good Thing?
• Probably the hardest agile principle for the
team to embrace.
– Need to elaborate the feature ahead of time
– There is minimal time to have the team review
before the start.
– Sometimes you have to rewrite
• Bottom-line: everyone collaborates to make
the feature as useful for the customer as
possible.
Copyright 2003-2005, Rally Software Development Corp
DSAH@Fall 2012 10
11. Requirements to Test Cases
• Use Case Scenario Tests are perfect Acceptance Tests
• Use Case A
– Scenario 1 Test Case 1
– Scenario 2 Test Case 2
• Declarative Requirements that further refine the Use
Case may be better suited to going directly to
automation
– Have one Test Case be the container for all of the
automation results.
– All automated tests have to pass before the Test Case
passes.
Copyright 2003-2005, Rally Software Development Corp
DSAH@Fall 2012 11
12. Need to Test early and often
• Need to test early in the Iteration – do not
want mini-waterfalls
• Need to test on check-in – Don’t break the
build
• Need to test nightly – Don’t wait for a
Regression Iteration
Copyright 2003-2005, Rally Software Development Corp
DSAH@Fall 2012 12
13. Mike Cohn’s Testing Pyramid
•Small number
GUI
•Automate many
Acceptance
Tests
•Find the right ones
FitNesse
Unit Tests •Largest numbers
•Foster Test Driven Design
Start Stop Look
Start Stop ?
Copyright 2003-2005, Rally Software Development Corp
DSAH@Fall 2012 13
14. Break the Manual Testing
Paradigm
•Easy to Create
Manual GUI
Acceptance
•Very familiar – what we always do
Tests •Typically tedious
•How do we know coverage?
•Need Automation specialists
Automated •Automation good for performance
GUI Tests •Seems like we always rewrite
Unit •Sometimes fragile
Tests
•What is Dev testing?
•How do we know what these are?
•How do we know when they fail?
Start Stop Look
Start Stop ?
Copyright 2003-2005, Rally Software Development Corp
DSAH@Fall 2012 14
15. Manual Testing Conundrum
• “You can never have too many manual acceptance
tests”
– Manual tests are cute little bunnies, before you know it
you have hundreds or thousands in your regression suite
– You inadvertently dig a hole you can never get out of
– Whole team had to help run regression suite
• Defect count typically is high
– Most defects were found as manual tests were elaborated
– Regression tests typically didn’t find many defects
– Commonly found defects – things we didn’t think of
Copyright 2003-2005, Rally Software Development Corp
DSAH@Fall 2012 15
16. Better, But Not Perfect Testing Architecture
Manual GUI
Acceptance •Still too many here
Tests
Automated
GUI Tests
& FitNesse •Add FitNesse
Unit Tests •Increase Coverage
•Increase Capability
Start Stop Look
Copyright 2003-2005, Rally Software Development Corp
DSAH@Fall 2012 16
17. Testing Types and Scheduling
Acceptance – •Minimize Manual •During the Iteration
GUI? •Generate off of Use
Cases to get scenario
tests.
Acceptance •Combination of Unit •Build Verification and Run
-Functional tests, FitNesse Nightly
Load & •Profiling and •Do it periodically
Performance Simulation Automation •Don’t wait till the end of the
Release cycle
Regression •Acceptance and •Run Nightly
Functional tests from
previous Iterations
Exploratory •Manual Group Explore •Before Releasing
•Roles and Personas
Copyright 2003-2005, Rally Software Development Corp
18. Keys to Overcome the Technical Challenges
• Continuous Builds
• Nightly Regression testing
• Make Unit Testing a priority
• From found defects – create automated tests
that go into Regression
Copyright 2003-2005, Rally Software Development Corp
DSAH@Fall 2012 18
19. Organizational Challenges
•Dev as Testers and Testers as Dev – how does that happen?
•Resistance to Change – how do we get the team to welcome and
embrace changes and not feel threatened?
•Testers are an integral part of the team- do we need to re-organize
to make this happen?
Copyright 2003-2005, Rally Software Development Corp
20. I’m a Developer, Not a Tester
• Pretty typical to hear push back from developers that they
– Don’t have time to do all of this testing
– Number of features delivered will go down
– Don’t really want to do all this testing
• Testers can help
– Provide guidance on how to break software, art of creative
destruction
– Pair testing with developers works well
• Have developers help out with manual regression testing.
– “Can’t I write a test for this instead of running it manually?”
Copyright 2003-2005, Rally Software Development Corp
DSAH@Fall 2012 20
21. I’m a Tester Not a Developer
• Pretty typical to hear from testers
– That they don’t feel comfortable or knowledgeable
about coding
– That they maybe won’t be needed anymore
• Developers can help
– Developers can create the fixtures (code running the
test) needed to make FitNesse testing work
– To make it easier to auto test the code at the GUI
level
Copyright 2003-2005, Rally Software Development Corp
DSAH@Fall 2012 21
22. Resisting Change
• Resistance is common
– It is easier to do what is familiar, than risk something
new
– Time-challenges may keep you doing the old way
– Fear of failing keeps you in the status quo
• Get the whole team involved in trying to change
– Team needs to figure what works best
– Don’t feel like you have to do everything all at once
– Keep learning and adapting
Copyright 2003-2005, Rally Software Development Corp
DSAH@Fall 2012 22
23. Testers on the Team
• Your organization may have testing as a separate
group – look for ways to integrate them into the
team
– Creating feature or component teams comprised of
all disciplines is one way
• Co-location is a great way to hear and share
information
• Daily stand-ups with the whole team keeps the
information current
Copyright 2003-2005, Rally Software Development Corp
DSAH@Fall 2012 23
24. Keys to Overcome the Organizational Challenges
• Have Dev help run manual Regression tests
• Pair Dev and Test on Unit and FitNesse Testing
• Co-location of all the team
• Daily Standups
• Do Retrospectives
Copyright 2003-2005, Rally Software Development Corp
DSAH@Fall 2012 24
25. Summary
• Agile Pulls Testing Forward
– You need to change your tools and approaches to
move it forward
– You might need to change the model/structure of
your team
• With Agile, you will create faster Release cycles,
shorter Iterations, more satisfied customers, and
team members that enjoy what they are doing
Copyright 2003-2005, Rally Software Development Corp
DSAH@Fall 2012 25
Notas del editor
Testing Architecture to Avoid at all costs
Evil rabbits Good News: The whole team became highly motivated to do it differently, and to find a better way