Rain is great for farmers and their crops, but terrible for tourists. Wind is essential for sailors and windmills but bad for the rest of us. Quality, like weather, is good or bad and that depends on who you are. Just like beauty, comfort, facility, flavour, intuitiveness, excitement and risk, quality is a concept that most people understand, but few can explain. It’s worse. Quality is an all-encompassing, collective term for these and many other difficult concepts. Quality is not an attribute of a system – it is a relationship between systems and stakeholders who take different views and the model of Quality that prevails has more to do with stakeholders than the system itself. Measurable quality attributes make techies feel good, but they don’t help stakeholders if they can’t be related to experience. If statistics don’t inform the stakeholders’ vision or model of quality, we think we do a good job. They think we waste their time and money.Whether documented or not, testers need and use models to identify what is important and what to test. A control flow graph has meaning (and value) to a programmer but not to a user. An equivalence partition has meaning to users but not the CEO. Control flow, equivalence partitions are models with value in some, but never all, contexts. If we want to help stakeholders to make better-informed decisions then we need test models that do more that identify tests. We need models that take account of the stakeholders’ perspective and have meaning in the context of their decision-making. If we measure quality using technical models (quality attributes, test techniques) we delude both our stakeholders and ourselves into thinking we are in control of Quality. We’re not. In this talk, Paul uses famous, funny and tragic examples of system failures to illustrate ways in which test models and (therefore testing) failed. He argues strongly that the pursuit of Quality requires that testers need better test models and how to create them, fast.

1. The Pursuit of Quality: Chasing
Tornadoes or Just Hot Air?
Gerrard Consulting Limited
PO Box 347
Maidenhead
Berkshire
SL6 2GU
Tel: +44 (0) 1628 639173
Fax: +44 (0) 1628 630398
Web: gerrardconsulting.com
Slide 1

2. Paul Gerrard
Slide 2
Paul is a consultant, teacher, author, webmaster, programmer, tester,
conference speaker, rowing coach and a publisher. He has conducted
consulting assignments in all aspects of software testing and quality
assurance, specialising in test assurance. He has presented keynote talks and
tutorials at testing conferences across Europe, the USA,Australia, South
Africa and occasionally won awards for them. In 2010 he won the Eurostar
EuropeanTesting Excellence Award.
Paul designed and built the Gerrard Consulting story platform on which the
maelscrum.com and businessstorymanager.com products are based.
Gerrard Consulting Limited hosts the UK Test Management Forum.

3. Agenda
• What is Quality?
• Models for quality and testing
• Examples of models
• Models and stakeholders
• Failures of systems, failures of models
• Close
Slide 3

4. Weather
• Rain is great for farmers and their crops, but
terrible for tourists
• Wind is essential for sailors and windmills but
bad for the rest of us
• Quality, like weather, can be good or bad and
that depends on who you are.
Slide 4

5. That’s
Fantastic!
That’s
Terrible!
Slide 5

6. Quality is a relationship
• Quality is not an
attribute of a system
• It is a relationship
between systems and
stakeholders who
take different views
• The model of Quality
that prevails has more
to do with
stakeholders than the
system itself
Slide 6

7. The concepts of quality, risk,
comfort, intuitiveness …
• Concepts that most
people understand, but
few can explain
• But it‟s a lot worse than
that
• Quality is an all-
encompassing, collective
term for these and many
other difficult concepts
• A term that means all
things to all people
• (I try and avoid the
Q-word).
Slide 7

8. Models for Quality and
Testing
Slide 8

9. Models
Slide 9
Models are everywhere

10. Models and reality
• In our minds we build mental models of everything we
experience (and also, many things we don‟t experience)
• When we pick up a glass of water, we build models
– The 3-dimentional location and relationship between the
glass, the water, the table it sits on and our body
– As we reach for the glass, our brain processes the signals
from our eyes, our muscles and the feelings in our
fingertips
– It continuously compares experience with the model and
adjusts/rebuilds the model many times
• … just to lift a cup of water – incredible!
Slide 10

11. Some familiar models
• The project plan is a model
– The resources, activities, effort, costs, risks and
future decision making
• System requirements are a model
– The “what and how” of the system
– What: the features and functionality
– How: how the system works (fast, secure, reliable)
• User personas (16 year old gamer, 30 year old
security hacker, 50 year old Man United fan).
Slide 11

12. Where quality comes from
• Quality is the outcome of a comparison
– Our mental model of perfection
– Our experience of reality
• Mental models are internal,
personal and unique to us
• We could share them using some
kind ofVulcan mind meld
• But usually, we can write them
down or we can talk about them
• However we communicate, there is noise and
information gets corrupted/lost in translation.
Slide 12

13. A quality model?
• The requirements and design describe the
behaviour of a system
• Functional
– Mapping test cases to requirements is all we need
• Non-Functional
– All technical attributes are defined and measured
• Quality and therefore testing assumes a model
– Often undocumented, the model may not be shared,
understood, complete, consistent, correct…
Slide 13

14. Test design is based on models
• Models describe the environment, system, usage, users,
goals, risks
• They simplify the context of the test - irrelevant or
negligible details are ignored in the model
• Focus attention on a particular aspect of the behaviour
of the system
• Generate a set of unique and diverse tests (within the
context of the model)
• Enable the testing to be estimated, planned, monitored
and evaluated for its completeness (coverage).
• Models help us to select tests in a systematic way.
Slide 14

15. Examples of test models
• A checklist or sets of criteria
– Goals, risks, process paths, interfaces, message type…
• Diagrams from requirements or design
documents
• Analyses of narrative text or tables
• Some models are documented, many models are
never committed to paper
– Can be mental models constructed specifically to
guide the tester whilst they explore the system under
test and guide their next action.
Slide 15

16. Sources of models
• Test Basis
– We analyse the text or diagrams or information that describe
required behaviour (or use past experience and knowledge)
• System architecture:
– We identify testable items in its user-interface, structure or
internal design
• Modes of failure (product risks):
– We identify potential ways in which the system might fail that
are of concern to stakeholders
• Usage patterns:
– We focus on the way the system will be used, operated and
interacted with in a business context using personas
• Everything looks fine – doesn‟t it?
Slide 16

17. But all models (over-)simplify
• But requirements are never perfect, not all
attributes can be meaningfully measured
• Models incorporate implicit assumptions and are
approximate representations
• All test models are heuristic, useful in some
situations, always incomplete and fallible
• Before we adopt a model, we need to know:
– What aspects of the behaviour, design, modes of
failure or usage the model helps us to identify
– What assumptions and simplifications it includes
(explicitly or implicitly).
Slide 17

18. Formality
• Formal test models
– Derived from analyses of requirements or code
– Quantitative coverage measure can be obtained from a formal
test mode (mostly)
• Informal test models
– E.g. some models are just lists of modes of failure, risks or
vulnerabilities.
– Informal models cannot be used to define quantitative coverage
measures
• Ad-hoc models
– Some models can be ad-hoc, invented by the tester just before
or even during testing
– Can be formal or informal.
Slide 18

19. Examples of Models
Slide 19

20. Basic test design techniques are
based on the simplest models
• Equivalence partitions and boundary values:
– Presume single input, single output responses
– All values in partitions are equivalent, but the
boundaries are the most important
• These techniques are useful, but they date
from the „green-screen‟ era.
Slide 20

21. “Green Screen” equivalence model
• Single input, single output
• All input is classified and
partitioned with rules
• One test per rule is
enough!
• But we don‟t consider:
– The state of the system
– Combinations of values.
Slide 21
If m<1 then
“Error”
Else if m>12 then
“Error”
Else
“OK”
Single Input
Single Output

22. StateTransitionTesting
Slide 22
Start
State
Room
Requested
Room
Booked
On
Waiting
List
Overnight
Stay
Booking
Cancelled
Checkout
Room available
Decrement room count
Room request
None
Customer arrives
None
Customer pays
Increment room
count
No room available
Add to waiting list
Customer cancels
Remove from waiting
list
Room available
Decrement room
count
Customer cancels
Increment room
count

23. But the number of states is infinite!
• State-Transition considers:
– The states of the system and
– The valid/invalid transitions between states
• Some systems have many, many states
– A real-time system e.g. telecoms switch may have
25,000 distinct states
– State may depend on many variables that can have
infinite values in combination
• How confident can we be in this model?
Slide 23

24. End-to-end/transaction-flow tests
• End–to-end tests can follow a path through a
process or a user journey
• The mechanics of the experience are
simulated but…
Slide 24

25. Bad experience leads to attrition
Slide 25
• Typical form-filling on government sites
intended to allow citizens to „apply online‟
Page 1 Page 2 Page 6Page 5Page 4Page 3 Page 7
45% 72% 48% 21% 85% 80% Conversion
by page
45% 32% 16% 3% 3% 2% Cumulative
• Every page „works‟ but the user-experience is so
poor that only 2% finish the journey
• Modelling the journey is good, but not enough…
• We need to model the experience too.

26. Models and
Stakeholders
Slide 26

27. Stakeholders and test models
• Stakeholders may not tell testers to use
specific test models; you need to explain them
to stakeholders so they understand
• The challenge(s):
– Stakeholders may be of the opinion that the
models you propose generate too few tests to be
meaningful or too many to be economic
– We need to engage stakeholders.
Slide 27

28. „Measuring quality‟ feels good
but…
• Measurable quality attributes make techies feel
good, but they don‟t help stakeholders if they
can‟t be related to experience
• If statistics don‟t inform the stakeholders‟
vision or model of quality
– We think we do a good job
– They think we waste their time and money.
Slide 28

29. Relevance
• Documented or not, testers need and use models
to identify what is important and what to test
• A control flow graph has meaning (and value) to a
programmer but not to an end-user
• An equivalence partition may have meaning to
users but not the CEO of the company
• Control flow, equivalence partitions are models
that have value in some, but never all, contexts.
Slide 29

30. Helping stakeholders to make
better decisions is the tester‟s goal
• We need models that
– Do more than identify tests
– Take account of the stakeholders‟ perspective and
have meaning in the context of their decision-
making
• If we „measure quality‟ using technical models
– We delude both our stakeholders and ourselves
into thinking we are in control of Quality
– We‟re not.
Slide 30

31. Failures of Systems,
Failures of Models
Slide 31

32. F-16 bug (found in flight)
• One of the early problems was that you could
flip the plane over and the computer would
gladly let you drop a bomb or fuel tank. It
would drop, dent the wing, and then roll off.
• http://catless.ncl.ac.uk/Risks/3.44.html#subj1.1
Slide 32
Poor test model

33. Slide 33
Poor test model

34. Slide 34
Poor test model

35. 2. Web Sub-System
Web Server
1. Application
(objects)
Sub-System
Database Server
Banking System
(Credit Card
Processor)
Legacy System(s)
3. Order Processing Sub-System
4. Full E-Business System
Scope of testing for E-Commerce
People
Process
Training
Environment
Slide 35

36. Test strategy
Slide 36
• Our test strategy must align with our model
of quality and our risk-assessment
Test Phase Focus
Requirements, design etc. Relevance, correctness, completeness, ambiguity etc.
Component Input validation, correct behaviour, output validation,
statement and branch coverage
Integration Correct, authorised transfer of control, exchange of
data, consistency of use and reconciliations
System (-system) End-to-end accuracy, consistency, security,
performance and reliability
Acceptance Alignment to business goals, end-to-end ease of use
and experience, successful outcomes, user personas
Every focus area
requires test
model(s)

37. Failure of testing is usually a failure
in a test model
• If the right models are selected, and commitment
is made to cover them
– The testing usually gets done
• But often, no model is explicitly selected at all
• Where a model fails, it is usually wrong because:
– The model does not represent reality
– The scope of the model is too narrow
– The model ignores critical aspects (context, people,
process, environment or training/capability).
Slide 37

38. Close
• We need to understand what quality is before we
can pursue and achieve it
• Testing often fails because test models are not
used or understood
• Testers need models to test but the „standard‟
quality models are too simple
• We need to take stakeholder views into account
to create relevant testing models
• Using models sounds techy, but it‟s completely
natural – it‟s part of what makes us human.
Slide 38

39. The Pursuit of Quality: Chasing
Tornadoes or Just Hot Air?
Slide 39
gerrardconsulting.com
testela.com
test-axioms.com
uktmf.com

40. http://catless.ncl.ac.uk/Risks/10.10#
subj6.1 (manufacturer response in RED)
• BA flight from NewYork and Fairbanks in US
• Co-pilot entered new navigational data into the system
and mis-typed a PIN code
• System response:
– “Invalid PIN number selected”
– “Access violation, contact your credit institution if you
believe there is an error.”
• All the plane's controls froze and it refused to respond
to commands
• SOS call to manufacturer at Aerospatiale in France…
• “The pasty little Englishman probably had too many
meat pies and Guiness".
Slide 40

41. Problem traced to the ATM-6000 INS
computer: a modified ATM product
• “The system will automatically remove the restrictions
at the start of the next banking day”
• Apparently, manual control could be re-set if a
crewmember went back of plane and operated the
elevators manually
• “There is nothing wrong with ze plane, that a little
pinch in the rear will not cure. Just like a woman. If
these English knew anything about women, they would
never have had to call us.”
• “The plane was able to safely land at Denver's
Stapelton airport, where the craft was repaired and all
crewmember's credit histories reviewed."
Slide 41