Developers often have the unfortunate distinction of not thoroughly testing their code. It’s not that developers do not understand how to test well; it’s just that often they have not had an opportunity to understand how the product works. Kevin Dunne maintains that implementing a team-wide exploratory testing initiative can help build the collaboration and knowledge sharing needed to elevate all team members to the level of product master. Exploratory testing can be performed by anyone, but the real challenge is making sure that the process is properly managed, documented, and optimized. Kevin describes the tools necessary to drive a deeper understanding of software quality and to implement an effective and impactful exploratory testing practice. Creating better software is not just about writing code more accurately and efficiently; it is about delivering value to the end user. Well-executed exploratory testing helps unlock this capability across the entire development team.

1. Exploratory Testing: Make It Part of Your Test Strategy
Kevin Dunne
VP, Strategy and Business Development
QASymphony

2. Agenda
1. What is exploratory testing (ET)?
2. What is NOT ET?
3. Why should we do ET?
4. How can I integrate ET with BDD?
5. ET best practices – do’s and don’ts

3. What is exploratory testing?
1. Parallel test planning, test design, and test execution
2. Specific yet flexible
3. Aligned towards investigation of potential opportunities
4. Values depth and attention to detail during testing
5. Fosters knowledge sharing and awareness

4. Parallel Planning, Design & Execution
Unlike traditional testing techniques, planning, design, and execution happen
concurrently, allowing efficiencies of time as well as flexibility in approach
Plan Design Execute Report
Plan
DesignExecute
Report

5. Specific yet Flexible
Exploratory testing provides a specific lens through which to perform testing –
whether that be a user persona, functionality, criteria (i.e. localization), etc.
However, it allows testers to use the tool as an end user would, not necessarily
as the product owner envisioned it
Manual Scripted Testing
I tested the application as the script
prescribed
Exploratory Testing
I tested the application as the end user
would

6. Investigating Opportunities
Exploratory testing rewards testers who identify unknown areas of “opportunity”
within the application, as they are essential in maintaining a backlog of future
test charters
Manual Scripted Testing Exploratory Testing

7. Knowledge Sharing
Exploratory testing relies on knowledge sharing to reach full potential –
developing testers who understand the impact of more areas of the application
allows them to identify more areas of risk and opportunity
Plan
DesignExecute
Report
Transfer
Learning
Example Questions to Ask
• Have you seen this before?
• What am I not considering?
• Why would someone do this?
• How would you have tested this?

8. What is Not Exploratory Testing
1. Exploratory testing is NOT unstructured testing
2. Exploratory testing is NOT the only form of testing
3. Exploratory testing is NOT throwaway work
4. Exploratory testing is NOT impossible in a regulated
environment

9. ET is NOT Unstructured Testing
While exploratory testing allows for flexibility in the exact path of the application that is
tested, it is NOT unstructured, in that it still contains parameters such as:
1. A goal of the exploration
2. A log of the activity performed
3. A lens through which the testing is performed (i.e. a user persona)
Performing exploratory testing without involving some parameters such as the above
allows a greater risk of unsuccessful implementation of exploratory testing

10. ET is NOT the Only Form of Testing
Exploratory testing is best suited as a complement to automated and manual scripted
test cases. It can feed these types of testing to create greater depth in testing, and also
to identify any potential gaps in coverage.
Potential New Feature Testing Cycle
Code
Developer
Unit Test
Exploratory
Testing
Manual
Scripted
Test
Automation
Regression
Test
Exploratory
Testing
Feature “Delivered”
“Let’s make sure this
is worth writing
scripts against yet”
“Let’s make sure
were still testing all
aspects of this”

11. ET is NOT Throwaway Work
Exploratory testing does NOT need to be extra work done on top of other testing
methods – it can count on its own towards testing progress and coverage if properly
accounted for. Some of the necessary information needed to manage it:
1. Charter
2. Session Sheet
3. Oral report
4. Debrief
5. Data Files
6. Logs
"Any testing approach is manageable when you choose to manage it.” – Michael Bolton
http://www.developsense.com/blog/2010/01/exploratory-testing-is-accountable/

12. ET is NOT Impossible in a Regulated Environment
Contrary to rumor and popular belief, exploratory testing is not only allowed in most
regulated environments, it is also essential.

13. Why Should we do ET?
A 2007 controlled study found that:
• Testing with test cases vs. exploratory testing take almost 7 times longer,
due to the amount of time needed to write the tests and report results on
them
• Testing with test cases vs. exploratory testing finds more defects, and
does not miss many (if any) critical or severe defects in comparison to
test case testing
• Testing with test cases causes more false defect reports vs. exploratory
testing
Study link: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.167.3696&rep=rep1&type=pdf

14. Adding ET to your Test Strategy
1. Paired Testing – get quick feedback from product experts early in the development
cycle
2. Team Based ET – rally the entire team around product learning and exploration
3. UAT – focus valuable SME time on value added activity rather than documentation
4. Beta Testing – maximize the returns of beta testers, minimize duplication and
uncovered areas
5. Replacing Traditional Testing – shift “stale” manual scripted tests to drive net new
exploration and feedback

15. Traditional Problems in Siloed Testing
Unsure if problem is code or requirements
• Testing is removed from development and requirements
management
Bottlenecks in handoffs between test and dev
• Submitting defects, assisting in defect reproduction, notifying
tester of defect ready for retest
Too Much Time Passing Between Dev and test
• Developer is notified of bugs in their code days after testing
has been completed

16. Benefits of Paired Testing
Close integration of development and test
• Minimize the risk of miscommunication and incorrect
interpretation of requirements
Immediate feedback on new code
• Code is tested right away, meaning developers have less time
to continue to build on bad code that will need to be rewritten
Fix Problems Immediately and Move On
• Reduce the number of defects opened, decreasing the
number of issues that would make it into production

17. Team Based ET
Exploratory testing is something that the whole team can benefit from:
1. Easy to Learn – don’t need to be proficient in many tools,
languages, etc.
2. Benefits from multiple perspectives and viewpoints
3. Quick to start up and scale – reduced overhead to get process
set up and running
NOTE: just because some team members are not experienced testers,
that does not mean that you throw the basics of exploratory testing out
the window!

18. UAT with ET
UAT Challenge ET Benefit
UATer’s are unfamiliar test case syntax and need
continual clarification
Allow UATer’s to perform the business flows they
know well without test scripts
UATer’s are not trained on test case
management, automation tools, etc.
Focus UATer’s time on learning how to document
proper defects, reduce time to ramp
UATer’s have a shorter attention span – they are
not used to testing 6-8 hrs. per day
Allow UATer’s to veer off the rails from time to
time and investigate areas of interest
UATer’s have a short period of time in which to
provide feedback
Ensure that as much of the UATer’s time as
possible is dedicated to ET

19. Beta Testing Challenges Solved Through ET
Problem #1 - Even the most efficient beta testing shops rarely get feedback from more
than 30% of their users – meaning that 70% of the beta testers provide no feedback
Solution – provide specific ET charters to beta testers. Charters will keep the testers
focused on key objectives and will drive accountability that will increase participation
Problem #2 – There is a segment of use cases are either under or over tested, leaving
bugs undiscovered in production
Solution – prioritize particular features and use intelligent assignment of the specific
charters to make sure adequate coverage is planned across appropriate environments
and devices

20. Traditional Beta vs. ET Beta Testing
Traditional Beta ET Beta
Where should I focus testing? At the users discretion, where
they’d like
According to their assigned charter
How many times will this feature
get tested?
We don’t really know As many times as it is assigned
out to users
Will this feature get tested on all
environments?
We don’t really know We will assign it to environments
needing coverage
Are testers focusing their efforts on
new features being released?
We don’t really know Yes, assuming we assign them to
work in those particular areas

21. Replace Traditional Testing with ET
Workplace choice improves the employee experience, and adding exploratory testing to
the mix allows testers to have choice many times per day

22. Replace Traditional Testing with ET
There are typically two strategies, which can be used in conjunction to begin replacing
manual scripted tests with exploratory ones:
1. Look for tests that have resulted in the lowest failure rates, lowest defect detection
rates, or both. In priority order, transfer these tests to exploratory test charters, and
monitor the defect detection rates from the transition.
2. Look for tests that take the longest to run, and are run the most frequently. In priority,
transfer these tests to exploratory charters, monitor the time per execution, and
ensure that defect detection rates stay constant or improve.

23. How to best structure our exploratory testing?
Introducing exploratory testing within a framework will greatly increase the odds of
success, and will reduce fear and uncertainty among the practitioners as well as
executives.
Session Based Test Management is a popular framework for this, as it tracks all the
important data on testing:
More info on SBTM: http://www.satisfice.com/articles/sbtm.pdf
• Session charter (includes a mission
statement, and areas to be tested)
• Testers involved Date and time executed
• Task breakdown
• Data files
• Test notes
• Issues
• Bugs

24. Resources/Thought Leaders
• James Bach - http://www.satisfice.com/
• Jonathan Bach - https://jonbox.wordpress.com/
• Michael Bolton - http://www.developsense.com/
• Paul Holland -
http://testingthoughts.com/blog/author/testthought
• Keith Klain - http://qualityremarks.com/
• Brian Osman - https://bjosman.wordpress.com/

25. Questions?
Kevin Dunne
kevindunne@qasymphony.com
Twitter: @kevindunneQA
Linkedin: www.linkedin.com/in/kevindunneQA
Blog: http://www.qasymphony.com/blog/