Презентация доклада Майкла Болтона на SQA Days-15 www.sqadays.com

1. 03‐Jun‐14
1
What’s the Problem?
Michael Bolton
http://www.developsense.com
michael@developsense.com
James Bach
http://www.satisfice.com
james@satisfice.com
What Is A Problem?
A problem is…
• …something that will lose money!
• …something that doesn’t meet expectations.
• …something that annoys or frustrates a user.
• …an obstacle that hinders the achievement of a
particular goal, objective, or purpose.

2. 03‐Jun‐14
2
Problem
• “A difference between what is perceived and
what is desired.”1
• “An undesirable situation that is significant to
and maybe solvable by some agent, though
probably with some difficulty.”2
1. Dewey, J. (1933), How We Think: A Restatement of the Relation of Reflective
Thinking to the Educative Process
2. Smith, G.F. “Towards a Heuristic Theory of Problem Structuring”, Management
Science. Vol. 34, No. 12
Problems (bugs, issues) are not attributes of a product,
but relationships between products and (some) people.
A common element to all
problems is…
Desire

3. 03‐Jun‐14
3
How Do We Recognize Problems?
An oracle is…
• a means by which we recognize
a problem that happens during
testing.
• a way to recognize a problem
• something that tells us “problem” or
“no problem”
Familiar Problems
If a product is consistent with problems we’ve seen before,
we suspect that there might be a problem.

4. 03‐Jun‐14
4
Explainability
If a product is inconsistent with our ability to explain it,
we suspect that there might be a problem.
World
If a product is inconsistent with the way the world works,
we suspect that there might be a problem.

5. 03‐Jun‐14
5
History
If a product is inconsistent with previous versions of itself,
we suspect that there might be a problem.
Hey, I liked the
menu bar!
How the #&@
do I print now?
Image
If a product is inconsistent with an image that
the organization wants to project, we suspect a problem.

6. 03‐Jun‐14
6
Comparable Products
WordPad Word
When a product seems inconsistent with a comparable
product or algorithm, we suspect that there might be a problem.
Claims
When a product is inconsistent with claims that important
people make about it, we suspect a problem.

7. 03‐Jun‐14
7
User Expectations
When a product is inconsistent with expectations that a
reasonable user might have, we suspect a problem.
Purpose
When a product is inconsistent with its designers’ explicit
or implicit purposes, we suspect a problem.

8. 03‐Jun‐14
8
Product
When a product is inconsistent internally—as when it
contradicts itself—we suspect a problem.
Statutes and Standards
When a product is inconsistent with laws or widely
accepted or relevant standards, we suspect a problem.

9. 03‐Jun‐14
9
17
Oracles are Not Perfect
And Testers are Not Judges
• You don’t need to know FOR SURE if something is a bug;
it’s not your job to DECIDE if something is a bug.
• You do need to form a justified belief that it MIGHT be a
threat to product value in the opinion of someone who
matters.
• And you must be able to say why you think so; you must
be able to cite good oracles… or else you will lose
credibility.
• Linking feelings to principles and back to feelings is an
important testing skill
What Might Feelings Tell Us?
Impatience  a threat to performance?
Frustration  a threat to capability?
Fear  a threat to security?
Surprise  a threat to reliability?
Confusion  a threat to usability? to testability?
Annoyance  a threat to charisma?
Boredom  an insignificant test?
Tiredness  time for a break?
Anxiety  a need for a particular skill?
Curiosity  a pointer to useful investigation?

10. 03‐Jun‐14
10
Feelings as Heuristic Triggers for Oracles
• An emotional reaction or a feeling is a trigger to
attention and learning.
• Without emotion, we don’t reason well.
– See Damasio, The Feeling of What Happens
• When you find yourself mildly concerned about
something, someone else could be very concerned
about it.
• Observe emotions to help overcome your biases, to
evaluate significance, and to calibrate responses.
Emotions and feelings are signals.
Look into what they’re signalling.
Tacit Explicit
Other PeopleTester
Your
Feelings &
Mental Models
Shared Artifacts
(specs, tools, etc.)
Stakeholders’
Feelings &
Mental Models
Inference
Observable
Consistencies
ReferenceConference
Experience
Oracles from the Inside Out

11. 03‐Jun‐14
11
• A person whose opinion matters.
• An opinion held by a person who matters.
• A disagreement among people who matter.
• A reference document with useful information.
• A known good example output.
• A known bad example output.
• A process or tool by which the output is checked.
• A process or tool that helps a tester identify patterns.
• A feeling like confusion or annoyance.
• A desirable consistency between related things.
General Examples of Oracles
things that suggest “problem” or “no problem”
21
People
Mechanisms &
Artifacts
Feelings
Principles
Using and Choosing Our Oracles
• Some oracles are more authoritative
– but less predictable ... like the product owner
• Some oracles are more consistent
– but maybe not up to date … like the requirements document
• Some oracles are more immediate
– but less reliable … like our feelings
• Some oracles are more precise
– but less accurate … like an automated check with a bug in it
• Some oracles are more accurate
– but less precise … like our Blink oracles
• Some oracles are more available
– but less authoritative … like the programmer
• Some oracles are easier to interpret
– but more narrowly focused … like a good automated check

12. 03‐Jun‐14
12
All Oracles Are Heuristic
• We often do not have oracles that establish a definite correct or
incorrect result, in advance. Oracles may reveal themselves to us as
we’re testing, or even later. That’s why we keep our minds open and
focused on a search from problems.
• No single oracle can tell us whether a program (or a feature) is
working correctly at all times and in all circumstances. That’s why we
use a variety of oracles.
• Any program that looks like it’s working, to you, may in fact be failing
in some way that happens to fool all of your oracles. That’s why we
proceed with humility and critical thinking.
• We never know when a test is finished. That’s why we try to remain
professionally uncertain when everyone else on the project is sure.
• You (the tester) can’t know the deep truth about any result. That’s
why we report whatever seems likely to be a bug.
23
A Note On Problem Solving
• “In order to solve a problem one must be able to alter the
structure which the situation spontaneously presents to the
mind. To perceive is to grasp the salient features of a given
state of affairs, ways of altering relations, accents,
groupings, selections, etc. in such a way that the new
pattern yields the desired solution. … Far from being
arbitrary or nonsensical, the new, more appropriate
structure is discovered in the situation itself. A problem
solver does not reorganize what he sees without reason. He
is driven by the need to obtain from the given situation
something it seems unprepared to give.”
• Rudolph Arnheim, Visual Thinking