"We should just call it testing - when it's not exploratory testing it's not real testing anyway" -Twitter, Summer 2011 Lately many professional testers have started to make a clear distinctions between thing that we call testing (like TDD and BDD) and what they consider testing - referring to TDD and BDD mostly as checking. And actually I – and I would think many of you as well – have seen projects with a test coverage of 80% and more that still fail to meet the clients' needs. Even though they meet the specifications perfectly. This points to some value that could be added to techniques like BDD and TDD by embracing the ideas from people like James Marcus Bach, Paul Carvalho and Michael Bolton. After giving an overview of current trends in the testing community like ET (exploratory testing) and ATDD (Acceptance Test Driven Development) this session will try to do exactly that: discuss the - often missing - intersection between BDD and exploratory testing and suggest ways to fill it.
16. Players in the Game
• James Bach
• Michael Bolten (not the singer)
• Cem Kaner
• Paul Carvalho
Tuesday, June 5, 2012
17. Ressources
Human Resource
Paul Carvalho http://swtester.blogspot.de/
Michael Bolton http://www.developsense.com/blog/
James Bach http://www.satisfice.com/blog/
Elisabeth Hendrickson http://testobsessed.com/
Markus Gärtner http://www.shino.de/blog/
Tuesday, June 5, 2012
18. Ressources by James Bach
Topic Ressource
SBTM Session Based http://satisfice.com/articles/sbtm.pdf
Test Management
http://people.eecs.ku.edu/~saiedian/
Exploratory Testing Teaching/Fa07/814/Resources/
exploratory-testing.pdf
Hands-on Exploratory http://www.satisfice.com/tools/
procedure.pdf
Testing
Rapid Software Testing http://www.satisfice.com/rst.pdf
Tuesday, June 5, 2012
27. So what’s the problem?
■ Software testing, the process of verifying that a
software program works as expected
Source: http://en.wikipedia.org/wiki/Test
Tuesday, June 5, 2012
28. So what’s the problem?
... but ...
Tuesday, June 5, 2012
29. So what’s the problem?
■ Test
[...]
Etymology
From Old French test (“an earthen vessel,
especially a pot in which metals were tried”)
Source: http://en.wiktionary.org/wiki/test
Tuesday, June 5, 2012
38. Defensive programming
- Check all input parameters
- Ascertain availability of resources
etc.
Defensive programming defends against the currently
impossible (c2 Wiki)
Tuesday, June 5, 2012
39. Design by Contract
Betrand Meyer
Since 1986
Programming language: Eiffel
Tuesday, June 5, 2012
40. Design by Contract
Betrand Meyer
Since 1986
Programming language: Eiffel
Preconditions
Postconditions
Invariants
Tuesday, June 5, 2012
41. TDD
• Unit Tests
• Black-Box Tests
• White-Box Tests
• RSpec etc.
Tuesday, June 5, 2012
42. BDD
• Acceptance Tests
• (Mostly) Black-Box Tests
• Based on Scenarios
Tuesday, June 5, 2012
43. ATDD
• Bigger gap between test creation and
execution
• Exclusively Black Box Testing
Tuesday, June 5, 2012
44. An adage from agile software
development
Tuesday, June 5, 2012
45. An adage from agile software
development
INVEST in stories and
come up with SMART tasks
Tuesday, June 5, 2012
46. INVEST
Independent
Negotiable
Valuable
Estimatable
Small
Testable
Tuesday, June 5, 2012
48. Independent
Specific
Negotiable
Mesuarable
Valuable
Achievable
Estimatable
Realistic
Small
Tangible
Testable
Tuesday, June 5, 2012
49. Acceptance Criteria
• To accomodate pre-conditions
Acceptance Criteria can be expressed as
• Given [Precondition]
When [Actor and Action]
Then [Observable Result]
• Naresh Jain Slide 29
Tuesday, June 5, 2012
50. Specification by Example
... It is the result of a research on
how teams all over the world specify,
develop, test and deliver the right
software, without defects, in very
short iterative delivery cycles. ...
Tuesday, June 5, 2012
52. DbC vs. TDD
Design by Contract
implies automated defensive
programming
pre- and postconditions and Test Driven Design is
invariants are checked at mostly applied during design-
runtime time.
Expensive checks are Tests are executed whenever
inhibitive appropriate - but usually not
in the production
environment
Expensive checks are
possible but unwarranted
Tuesday, June 5, 2012
53. Checking vs. Testing
Checking is something that
we do with the motivation of
confirming existing beliefs
Testing is something that we
do with the motivation of
finding new information.
http://www.developsense.com/blog/2009/08/testing-vs-checking/
Tuesday, June 5, 2012
56. Red-Green-Refactor
Failing
Test
Simplify Coding
Remove Redundancy
Tuesday, June 5, 2012
57. Exploratory Testing
From: Paul Carvalho
http://swtester.blogspot.de/2012/05/what-is-exploratory-testing.html
Tuesday, June 5, 2012
58. So what is Exploratory Testing?
"Exploratory Testing is simultaneously
learning about the system while designing
and executing tests, using feedback from
the last test to inform the next."
Elisabeth Hendrickson, @testobsessed
Quoted by Paul Carvalho
Tuesday, June 5, 2012
59. Cone of uncertainty
Agile and Iterative Development: A Manager's Guide, Craig Larman 2003
Tuesday, June 5, 2012
61. Cone of test-cases
http://swtester.blogspot.de/2012/05/what-is-exploratory-testing.html
Tuesday, June 5, 2012
62. Agile testing quadrants
http://lisacrispin.com/wordpress/2011/11/08/using-the-agile-testing-quadrants/
Tuesday, June 5, 2012
63. The Prime Directive
Be Thoughtful and Methodical
Throughout the test procedure, as you complete the tasks, you
have lots of freedom about how you do the work. But you must
work methodically, and follow the procedure. In the course of
creating the result for each task, you’ll find that you have to
make a lot of guesses, and some of them will be wrong. But you
must think. If you find yourself making wild and uneducated
guesses about how the product works, areas of instability, or
anything else, stop and talk to the Test Manager.
James Bach, 08/26/99, http://www.satisfice.com/tools/procedure.pdf
Tuesday, June 5, 2012
64. Scientific Method
Formulate a question
Hypothesis
Prediction
Test
Analysis
Tuesday, June 5, 2012
65. Scientific Method
Formulate a question
Hypothesis
Prediction
Test
Analysis
Tuesday, June 5, 2012
66. Exploratory Testing
Complete these five tasks:
✓ Identify the purpose of the product.
✓ Identify functions.
✓ Identify areas of potential instability.
✓ Test each function and record problems.
✓ Design and record a consistency verification
test.
Source: http://www.satisfice.com/tools/procedure.pdf
pg. 11 to 22
Tuesday, June 5, 2012
67. Identify the purpose of the
product
• With a tangible result (i.e. Document)
• Value for others - verifiable etc.
• Value for the tester - builds
understanding
Tuesday, June 5, 2012
68. Identify functions
• By actually using (skimming) the product
• Explorative (not based on documentation)
• Tangible output (Function outline)
Tuesday, June 5, 2012
69. From James Bach:
http://www.satisfice.com/tools/procedure.pdf
Tuesday, June 5, 2012
70. Identify areas of potential
instability
• Results:
• Documented areas (before testing)
• Documented data for testing
• Documented Strategies
Tuesday, June 5, 2012
71. Test each function and record
problems
• Actually this means: Try to do things the
development teams did not think of.
• Remember: A Problem is just the
difference between the expected behavior
and the observed behavior - whether it is
cause for action has to be decided by the
customer
Tuesday, June 5, 2012
72. Design and record a
consistency verification test
• Try to extrapolate if the Software would
run in (slightly) modified environments.
(The original 5 steps are from 1999 (gasp)
and relate to specific windows versions -
the basic requirements still hold through)
Tuesday, June 5, 2012
73. Tool Support
Area Tool URL
http://
Recording spector www.spectorsoft.com/
http://
Recording BBTestAssistant www.bbsoftware.co.uk/
products.aspx
http://
Recording Wink www.debugmode.com/
wink/
http://www.sirius-
All in One TestExplorer sqa.com/
http://www.satisfice.com/blog/archives/77
Tuesday, June 5, 2012
74. Session Based Test Management
• Original description of ET : 1999
• Today : 2012
• SBTM: A way to organize exploratory
testing
• Working in 45 to 90 min Session based
on Charters
• See: http://satisfice.com/articles/sbtm.pdf
Tuesday, June 5, 2012
75. What are Charters
Counterpart to the use case / user story
Some of the same attributes
- Roles, activities, actors, systems
Examples:
- “Analyze <Product>’s View menu functionality and
report on areas of potential risk.”
Tuesday, June 5, 2012
82. ET-Spectrum
Exploratory Testing according to:
http://www.satisfice.com/rst.pdf
Tuesday, June 5, 2012
83. Can we prove it?
Testing shows the presence,
not the absence of bugs
J.N. Buxton and B. Randell, eds, Software Engineering Techniques,
April 1970, p. 16. Report on a conference sponsored by the NATO
Science Committee, Rome, Italy, 27–31 October 1969.
via http://en.wikiquote.org/wiki/Edsger_W._Dijkstra
Program testing can be used to
show the presence of bugs, but
never to show their absence!
Source: Notes On Structured Programming, 1972, at the end of section 3, On The
Edsger Wybe Dijkstra Reliability of Mechanisms. EWD249 (1970).
Tuesday, June 5, 2012
84. TDD
• Technique to drive the design of your code
y, t!
r n
sa cie
s fi
ce uf
e s
N ot
t n
Bu
Tuesday, June 5, 2012
85. BDD
• Technique to capture requirements and
drive desired functionality
y, t!
r n
sa cie
s fi
ce uf
e s
N ot
t n
Bu
Tuesday, June 5, 2012
86. ET
• Technique to drive the discovery of
unwanted, unexpected and unpredicted
behaviour
y, t!
r n
sa cie
s fi
ce uf
e s
N ot
t n
Bu
Tuesday, June 5, 2012
87. About me
“Consulting on software development
processes and software architecture since
the last millennium”
- @MMahlberg
Google me: “Michael Mahlberg”
Tuesday, June 5, 2012