2. Exploratory testing is a systematic
approach for discovering risks using
rigorous analysis techniques coupled
with testing heuristics.
3. “[Exploratory Testing is] simultaneously designing and executing
tests to learn about the system, using your insights from the last
experiment to inform the next”
James Bach
12. 12
We’re at a stage in software industry where
you can find 76 problems including a
showstopper in your first 4 hours of testing,
ever. Imagine how the testing profession is
changing when that is no longer true?
20. Bring structure to ET: Charters
Form a mission before you test
Think about the tools you want to use
Think about what you’re hoping to find out
Example Charter:
Explore ApprovalTests
By putting different variables in the
verifyObject method
To discover if you get any unexpected
behaviour
24. ET approaches, suggestions:
SFDPO (James Bach)
Sequences and interactions (Explore It!)
Entities: CRUD, follow the data (Explore It!)
States & transitions (Explore It!)
Tours: super model tour, clubbing tour, couch potato,
landmark tour, intellectual tour, FedEx tour
Test Heuristics cheat sheet
More suggestions?
25. Suggested Tools:
Browser: inspector, Postman, Bug Magnet, color picker,
empty cache/cookies, Spectrum
Logging/Proxy: Charles, Fiddler
Security: OWASP ZAP
Performance/load: Gatling, Jmeter
Git
Any other suggestions?
26. Tools to support documenting
Rapid Reporter (mac/windows)
Post It’s
Microsoft Exploratory Testing add-on
TextEdit
Screen recorders
A second (or 3rd, 4th…) pair of eyes
Charles/Fiddler to record you API calls
More ideas?
28. Automation ideas beyond regression
Fuzzing
Model-based Testing
Monitoring/analytics
Step to –tools
Chaos monkey
State control with mocks/stubs
Pairing with automation ‘tour bus’
Data generation
Mutation testing
28
29. Main characteristics of ET
Self-directedness
Freedom
Responsibility
Learning
Working with your own biases
Improving yourself as a tester
Contextual
30. 30
Maaret Pyhäjärvi
Email: maaret@iki.fi
Twitter: @maaretp
Web: maaretp.com
Blog: visible-quality.blogspot.fi
(please connect with me through Twitter or LinkedIn)
Maaike Brinkhof
Email: mbrinkhof@xebia.com
Twitter: @maaikees
Web: maaikebrinkhof.nl
Blog: maaikebrinkhof.nl
32. Example Charters
Explore input fields
With injection attacks
To discover security
vulnerabilities
Explore editing profiles
With invalid usernames
To discover if there are
any instances where
username constraints are
not enforced
33. Example Charters (2)
Explore editing profiles
With invalid usernames
To discover if there are
any instances where
username constraints are
not enforced
34. ET Definition
Cem Kaner:
“Exploratory testing is a style of software testing that emphasizes the personal freedom and responsib
of the individual tester to continually optimize the value of her work by treating test-related learning,
test design, test execution, and test result interpretation as mutually supportive activities that run in
parallel throughout the project.”
James Bach:
“Simultaneously designing and executing tests to learn about the system,
using your insights from the last experiment to inform the next”
35. Testing vs Checking debate
Checking: tests you design in advance, can be automated,
can be scaled, modeled, etc.
Testing: only a human can do testing, it involves
judgement on the fly, skill, sharp decision making. You
cannot test everything! So test wisely
36. Context Driven Principles
The value of any practice depends on its context.
There are good practices in context, but there are no best practices.
People, working together, are the most important part of any
project’s context.
Projects unfold over time in ways that are often not predictable.
The product is a solution. If the problem isn’t solved, the product
doesn’t work.
Good software testing is a challenging intellectual process.
Only through judgment and skill, exercised cooperatively throughout
the entire project, are we able to do the right things at the right times
to effectively test our products.
37. Agile Manifesto
Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
That is, while there is value in the items on
the right, we value the items on the left more.
38. That's where the power of
exploratory testing comes in: the
richness of this process is only
limited by the breadth and depth of
our imagination and our emerging
insights into the nature of the product
under test.
Notas del editor
Question: Which title do you want?Exploratory Testing Experienced and Explained? Or this one?
asking all the questions beforehand?
OR, asking all the questions at the time you need them and learning from the answer one by one
expecting results beforehand bias
testing is about trying to find the unexpected problems too
variables. Testing isn’t about coverage/covering every variable.
human memory is limited
*room for discussion*
There’s a group of people who think of testing as creating artifacts: test cases, test automation. Tests are something that is used later for testing to happen again. The modern way of thinking leads this group of people to mostly consider test automation, recognizing that making people run mundane repeatable scripts isn’t the goal. With the automation artifacts, we get four types of benefits. The tests work as specification, giving us a detailed example of something. The tests give us feedback when we’ve implemented something, to see if what we have is what we intended. The tests stay around to guard us against regression, us forgetting details when paying attention elsewhere. And the tests give us granularity, helping us see where the problem is when the tests fail.
Another group of people thinks of testing as performance. It’s learning in layers, to prepare for the ultimate show with all the rehearsals where continuous learning enables us to put our best performance forward towards the end. Artifacts might be useful, but they are often more of an output to document all the layers of learning rather than driving the thinking. With testing as performance, we also get four types of benefits. While this style of testing does not give us a spec, it gives us guidance. It’s often not a binary Yes/No, but more of a consideration if there could be a problem here. The guidance enabling us to learn about the unknown unknowns turn into specs as we make our decisions on how to react to feedback. This testing gives us understanding, in the wider context of use and value. This testing gives us models of what the world around the application looks like, enabling us to go back to lessons we’ve learned and fast-track future learning. Finally, this testing gives us serendipity, lucky accidents to find many of the things that we should know of, but turn blind to assuming we know what we’re looking for.
While exploratory testing starts with the testing as performance in mind, it also generates artifacts whenever deemed useful. Testing of an API is a great way to get to the core of this difference in how we think about testing.
All testing may be exploratory, but some of it is focused on creating artifacts. Saying “scripting is just an approach” is belittling when scripting can be the main approach to use the powers when testing.
“there’s a process of knowing” – learning
Does not give as regression; serendipity (safety against things happening randomly) / unwanted serendipity events.
This is what it is and what it could be. There’s a direction to it, not just statement of what it is.
Coaching is not just feedback, it’s pointing them to the right way.
Safety.
EXPERIENCE (the verb) rather than facts ; emotions over facts. REACTIONS.
HISTORY, Lessons learned, checklists. Modeling.
UNDERSTANDING – where you start (knowing the thing (code & environment), knowing the user, knowing the problems, knowing the developers (how to help them and what they do so that you can efficiently test), knowing the hackers (weird use cases outside common ‘have you tried reading it upside down’) , knowing all stakeholders, knowing the business priorities)
Uncovering things I cannot know, giving the application a change to reveal information for me.
This allows you to know things.
EACH MOUSE CLICK INTRODUCES THE NEXT PERSONS POSITION. ORDER: DRIVER, DESIGNATED NAVIGATOR, NAVIGATORS
Notice how easier the left side is to read and look at than the right side
Nog verder uitwerken, huiswerk voor Maaike
Deelnemers zelf laten kiezen welke ze willen doen!
http://searchsoftwarequality.techtarget.com/tip/Six-tours-for-exploratory-testing-the-business-district-of-your-application
http://reqtest.com/testing-blog/what-is-tour-testing-a-structured-approach-to-exploratory-testing/
Ik ga niet weer een definitie oplepelen hier, tenzij jullie vinden dat ik dat wel moet doen.
Wel definitie voorbereiden, op papier zelf. Mensen kunnen er om vragen.
Ik wil vooral benadrukken wat de mooie kanten van ET zijn t.o.v en in samenwerking met de rol die test automation heeft.
Documenteren in ET: vroeger was het ‘alleen bugs zijn boeiende dingen om te documenteren’.
Ik denk daar tegenwoordig anders over: als je heel erg leunt op scripted automated tests en daarnaast alleen dit doet moet je het slim aanpakken.
Zorg dat zo min mogelijk kennis verloren gaat en dat de leercurve in je tests omhoog blijft gaan. Kortom: leer van jezelf. Dit kan je alleen voor elkaar krijgen als je tijdens AL je test sessies wijselijk documenteert. Schrijf op WAT je hebt getest. Waarom is dat boeiend?? Omdat je dan weet wanneer je te weinig variatie in je tests hebt gebracht. Blijf de invoer en stijl veranderen, anders worden je test sessies minder waard.
“That's where the power of exploratory testing comes in: the richness of this process is only limited by the breadth and depth of our imagination and our emerging insights into the nature of the product under test.” http://www.satisfice.com/articles/what_is_et.shtml