Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.
Exploratory Testing: Make It Part of Your Test Strategy
Kevin Dunne
VP, Strategy and Business Development
QASymphony
Agenda
1. What is exploratory testing (ET)?
2. What is NOT ET?
3. Why should we do ET?
4. How can I integrate ET with BDD?...
What is exploratory testing?
1. Parallel test planning, test design, and test execution
2. Specific yet flexible
3. Aligne...
Parallel Planning, Design & Execution
Unlike traditional testing techniques, planning, design, and execution happen
concur...
Specific yet Flexible
Exploratory testing provides a specific lens through which to perform testing –
whether that be a us...
Investigating Opportunities
Exploratory testing rewards testers who identify unknown areas of “opportunity”
within the app...
Knowledge Sharing
Exploratory testing relies on knowledge sharing to reach full potential –
developing testers who underst...
What is Not Exploratory Testing
1. Exploratory testing is NOT unstructured testing
2. Exploratory testing is NOT the only ...
ET is NOT Unstructured Testing
While exploratory testing allows for flexibility in the exact path of the application that ...
ET is NOT the Only Form of Testing
Exploratory testing is best suited as a complement to automated and manual scripted
tes...
ET is NOT Throwaway Work
Exploratory testing does NOT need to be extra work done on top of other testing
methods – it can ...
ET is NOT Impossible in a Regulated Environment
Contrary to rumor and popular belief, exploratory testing is not only allo...
Why Should we do ET?
A 2007 controlled study found that:
• Testing with test cases vs. exploratory testing take almost 7 t...
Adding ET to your Test Strategy
1. Paired Testing – get quick feedback from product experts early in the development
cycle...
Traditional Problems in Siloed Testing
Unsure if problem is code or requirements
• Testing is removed from development and...
Benefits of Paired Testing
Close integration of development and test
• Minimize the risk of miscommunication and incorrect...
Team Based ET
Exploratory testing is something that the whole team can benefit from:
1. Easy to Learn – don’t need to be p...
UAT with ET
UAT Challenge ET Benefit
UATer’s are unfamiliar test case syntax and need
continual clarification
Allow UATer’...
Beta Testing Challenges Solved Through ET
Problem #1 - Even the most efficient beta testing shops rarely get feedback from...
Traditional Beta vs. ET Beta Testing
Traditional Beta ET Beta
Where should I focus testing? At the users discretion, where...
Replace Traditional Testing with ET
Workplace choice improves the employee experience, and adding exploratory testing to
t...
Replace Traditional Testing with ET
There are typically two strategies, which can be used in conjunction to begin replacin...
How to best structure our exploratory testing?
Introducing exploratory testing within a framework will greatly increase th...
Resources/Thought Leaders
• James Bach - http://www.satisfice.com/
• Jonathan Bach - https://jonbox.wordpress.com/
• Micha...
Questions?
Kevin Dunne
kevindunne@qasymphony.com
Twitter: @kevindunneQA
Linkedin: www.linkedin.com/in/kevindunneQA
Blog: h...
Próxima SlideShare
Cargando en…5
×

Exploratory Testing: Make It Part of Your Test Strategy

1.310 visualizaciones

Publicado el

Developers often have the unfortunate distinction of not thoroughly testing their code. It’s not that developers do not understand how to test well; it’s just that often they have not had an opportunity to understand how the product works. Kevin Dunne maintains that implementing a team-wide exploratory testing initiative can help build the collaboration and knowledge sharing needed to elevate all team members to the level of product master. Exploratory testing can be performed by anyone, but the real challenge is making sure that the process is properly managed, documented, and optimized. Kevin describes the tools necessary to drive a deeper understanding of software quality and to implement an effective and impactful exploratory testing practice. Creating better software is not just about writing code more accurately and efficiently; it is about delivering value to the end user. Well-executed exploratory testing helps unlock this capability across the entire development team.

Publicado en: Software
  • Inicia sesión para ver los comentarios

  • Sé el primero en recomendar esto

Exploratory Testing: Make It Part of Your Test Strategy

  1. 1. Exploratory Testing: Make It Part of Your Test Strategy Kevin Dunne VP, Strategy and Business Development QASymphony
  2. 2. Agenda 1. What is exploratory testing (ET)? 2. What is NOT ET? 3. Why should we do ET? 4. How can I integrate ET with BDD? 5. ET best practices – do’s and don’ts
  3. 3. What is exploratory testing? 1. Parallel test planning, test design, and test execution 2. Specific yet flexible 3. Aligned towards investigation of potential opportunities 4. Values depth and attention to detail during testing 5. Fosters knowledge sharing and awareness
  4. 4. Parallel Planning, Design & Execution Unlike traditional testing techniques, planning, design, and execution happen concurrently, allowing efficiencies of time as well as flexibility in approach Plan Design Execute Report Plan DesignExecute Report
  5. 5. Specific yet Flexible Exploratory testing provides a specific lens through which to perform testing – whether that be a user persona, functionality, criteria (i.e. localization), etc. However, it allows testers to use the tool as an end user would, not necessarily as the product owner envisioned it Manual Scripted Testing I tested the application as the script prescribed Exploratory Testing I tested the application as the end user would
  6. 6. Investigating Opportunities Exploratory testing rewards testers who identify unknown areas of “opportunity” within the application, as they are essential in maintaining a backlog of future test charters Manual Scripted Testing Exploratory Testing
  7. 7. Knowledge Sharing Exploratory testing relies on knowledge sharing to reach full potential – developing testers who understand the impact of more areas of the application allows them to identify more areas of risk and opportunity Plan DesignExecute Report Transfer Learning Example Questions to Ask • Have you seen this before? • What am I not considering? • Why would someone do this? • How would you have tested this?
  8. 8. What is Not Exploratory Testing 1. Exploratory testing is NOT unstructured testing 2. Exploratory testing is NOT the only form of testing 3. Exploratory testing is NOT throwaway work 4. Exploratory testing is NOT impossible in a regulated environment
  9. 9. ET is NOT Unstructured Testing While exploratory testing allows for flexibility in the exact path of the application that is tested, it is NOT unstructured, in that it still contains parameters such as: 1. A goal of the exploration 2. A log of the activity performed 3. A lens through which the testing is performed (i.e. a user persona) Performing exploratory testing without involving some parameters such as the above allows a greater risk of unsuccessful implementation of exploratory testing
  10. 10. ET is NOT the Only Form of Testing Exploratory testing is best suited as a complement to automated and manual scripted test cases. It can feed these types of testing to create greater depth in testing, and also to identify any potential gaps in coverage. Potential New Feature Testing Cycle Code Developer Unit Test Exploratory Testing Manual Scripted Test Automation Regression Test Exploratory Testing Feature “Delivered” “Let’s make sure this is worth writing scripts against yet” “Let’s make sure were still testing all aspects of this”
  11. 11. ET is NOT Throwaway Work Exploratory testing does NOT need to be extra work done on top of other testing methods – it can count on its own towards testing progress and coverage if properly accounted for. Some of the necessary information needed to manage it: 1. Charter 2. Session Sheet 3. Oral report 4. Debrief 5. Data Files 6. Logs "Any testing approach is manageable when you choose to manage it.” – Michael Bolton http://www.developsense.com/blog/2010/01/exploratory-testing-is-accountable/
  12. 12. ET is NOT Impossible in a Regulated Environment Contrary to rumor and popular belief, exploratory testing is not only allowed in most regulated environments, it is also essential.
  13. 13. Why Should we do ET? A 2007 controlled study found that: • Testing with test cases vs. exploratory testing take almost 7 times longer, due to the amount of time needed to write the tests and report results on them • Testing with test cases vs. exploratory testing finds more defects, and does not miss many (if any) critical or severe defects in comparison to test case testing • Testing with test cases causes more false defect reports vs. exploratory testing Study link: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.167.3696&rep=rep1&type=pdf
  14. 14. Adding ET to your Test Strategy 1. Paired Testing – get quick feedback from product experts early in the development cycle 2. Team Based ET – rally the entire team around product learning and exploration 3. UAT – focus valuable SME time on value added activity rather than documentation 4. Beta Testing – maximize the returns of beta testers, minimize duplication and uncovered areas 5. Replacing Traditional Testing – shift “stale” manual scripted tests to drive net new exploration and feedback
  15. 15. Traditional Problems in Siloed Testing Unsure if problem is code or requirements • Testing is removed from development and requirements management Bottlenecks in handoffs between test and dev • Submitting defects, assisting in defect reproduction, notifying tester of defect ready for retest Too Much Time Passing Between Dev and test • Developer is notified of bugs in their code days after testing has been completed
  16. 16. Benefits of Paired Testing Close integration of development and test • Minimize the risk of miscommunication and incorrect interpretation of requirements Immediate feedback on new code • Code is tested right away, meaning developers have less time to continue to build on bad code that will need to be rewritten Fix Problems Immediately and Move On • Reduce the number of defects opened, decreasing the number of issues that would make it into production
  17. 17. Team Based ET Exploratory testing is something that the whole team can benefit from: 1. Easy to Learn – don’t need to be proficient in many tools, languages, etc. 2. Benefits from multiple perspectives and viewpoints 3. Quick to start up and scale – reduced overhead to get process set up and running NOTE: just because some team members are not experienced testers, that does not mean that you throw the basics of exploratory testing out the window!
  18. 18. UAT with ET UAT Challenge ET Benefit UATer’s are unfamiliar test case syntax and need continual clarification Allow UATer’s to perform the business flows they know well without test scripts UATer’s are not trained on test case management, automation tools, etc. Focus UATer’s time on learning how to document proper defects, reduce time to ramp UATer’s have a shorter attention span – they are not used to testing 6-8 hrs. per day Allow UATer’s to veer off the rails from time to time and investigate areas of interest UATer’s have a short period of time in which to provide feedback Ensure that as much of the UATer’s time as possible is dedicated to ET
  19. 19. Beta Testing Challenges Solved Through ET Problem #1 - Even the most efficient beta testing shops rarely get feedback from more than 30% of their users – meaning that 70% of the beta testers provide no feedback Solution – provide specific ET charters to beta testers. Charters will keep the testers focused on key objectives and will drive accountability that will increase participation Problem #2 – There is a segment of use cases are either under or over tested, leaving bugs undiscovered in production Solution – prioritize particular features and use intelligent assignment of the specific charters to make sure adequate coverage is planned across appropriate environments and devices
  20. 20. Traditional Beta vs. ET Beta Testing Traditional Beta ET Beta Where should I focus testing? At the users discretion, where they’d like According to their assigned charter How many times will this feature get tested? We don’t really know As many times as it is assigned out to users Will this feature get tested on all environments? We don’t really know We will assign it to environments needing coverage Are testers focusing their efforts on new features being released? We don’t really know Yes, assuming we assign them to work in those particular areas
  21. 21. Replace Traditional Testing with ET Workplace choice improves the employee experience, and adding exploratory testing to the mix allows testers to have choice many times per day
  22. 22. Replace Traditional Testing with ET There are typically two strategies, which can be used in conjunction to begin replacing manual scripted tests with exploratory ones: 1. Look for tests that have resulted in the lowest failure rates, lowest defect detection rates, or both. In priority order, transfer these tests to exploratory test charters, and monitor the defect detection rates from the transition. 2. Look for tests that take the longest to run, and are run the most frequently. In priority, transfer these tests to exploratory charters, monitor the time per execution, and ensure that defect detection rates stay constant or improve.
  23. 23. How to best structure our exploratory testing? Introducing exploratory testing within a framework will greatly increase the odds of success, and will reduce fear and uncertainty among the practitioners as well as executives. Session Based Test Management is a popular framework for this, as it tracks all the important data on testing: More info on SBTM: http://www.satisfice.com/articles/sbtm.pdf • Session charter (includes a mission statement, and areas to be tested) • Testers involved Date and time executed • Task breakdown • Data files • Test notes • Issues • Bugs
  24. 24. Resources/Thought Leaders • James Bach - http://www.satisfice.com/ • Jonathan Bach - https://jonbox.wordpress.com/ • Michael Bolton - http://www.developsense.com/ • Paul Holland - http://testingthoughts.com/blog/author/testthought • Keith Klain - http://qualityremarks.com/ • Brian Osman - https://bjosman.wordpress.com/
  25. 25. Questions? Kevin Dunne kevindunne@qasymphony.com Twitter: @kevindunneQA Linkedin: www.linkedin.com/in/kevindunneQA Blog: http://www.qasymphony.com/blog/

×