This is the talk Paul Gerrard gave at Eurostar 2010.

1. THE AXIOMS TESTING OF Advancing Testing Using Axioms Paul Gerrard

2. Paul Gerrard Paul is a consultant, teacher, author, webmaster, programmer, tester, conference speaker, rowing coach and a publisher. He has conducted consulting assignments in all aspects of software testing and quality assurance, specialising in test assurance. He has presented keynote talks and tutorials at testing conferences across Europe, the USA, Australia, South Africa and occasionally won awards for them. Educated at the universities of Oxford and Imperial College London, Paul was the founding chair of the BCS ISEB Testing Certificate Scheme and a member of the Working Party that produced BS 7925 – the Component Test Standard. Currently, he is Principal of Gerrard Consulting Limited and is the host of the UK Test Management Forum.

3. Agenda Axioms – a Brief Introduction Advancing Testing Using Axioms First Equation of Testing Test Strategy and Approach Testing Improvement A Skills Framework for Testers Quantum Testing Close

4. Test Axioms Formulated as a context-neutral set of rules for testing systems They represent the critical thinking processes required to test any system There are clear opportunities to advance the practice of testing using them Testers Pocketbook: testers-pocketbook.com Test Axioms Website test-axioms.com

6. They can help you to think critically about testing

7. They expose flaws in other people’s thinking and their arguments about testing

8. They generate some useful by-products

9. They help you to separate context from values

10. Interesting research areas!

11. First Equation of Testing, Testing Uncertainty Principle, Quantum Theory, Relativity, Exclusion Principle...

13. Stakeholder axioms

14. Testing needs stakeholders (p64) Summary: Identify and engage the people or organisations that will use and benefit from the test evidence we are to provide Consequence if ignored or violated: There will be no mandate or any authority for testing. Reports of passes, fails or enquiries have no audience. Questions: Who are they? Whose interests do they represent? What evidence do they want? What do they need it for? When do they want it? In what format? How often?

15. Design axioms Fallibility

16. Test design is based on models (p68) Summary: Choose test models to derive tests that are meaningful to stakeholders. Recognise the models’ limitations and the assumptions that the models make Consequence if ignored or violated: Tests design will be meaningless and not credible to stakeholders. Questions Are design models available to use as test models? Are they mandatory? What test models could be used to derive tests from the Test Basis? Which test models will be used? Are test models to be documented or are they purely mental models? What are the benefits of using these models? What simplifying assumptions do these models make? How will these models contribute to the delivery of evidence useful to the acceptance decision makers? How will these models combine to provide sufficient evidence without excessive duplication? How will the number of tests derived from models be bounded?

17. Delivery axioms Environment

18. Advancing Testing Using Axioms

19. First Equation of Testing Axioms+ Context+ Values+ Thinking =Approach

20. Why is the equation useful? Separation of Axioms, context, values and thinking Tools, methodologies, certification, maturity models promote approaches without reference to your context or values No thinking is required! Without a unifying test theory you have no objective way of assessing these products.

21. One context, multiple approaches Given context, practitioners can promote different approaches based on their values Valuesare preferences or beliefs Pre-planned v exploratory Predefined v custom process Requirements-driven v goal-based Standard documentation v face-to-face comms. Some contexts preclude certain practices “No best practices”

22. Axioms allow (ensure) different approaches and expose positions Separating axioms, context and values clarifies positions, for example: ‘Structured’ (certified?) test advocates have little (useful) to say about Agile contexts Exploratory test advocates have little (useful) to say about contract/requirements-based acceptance The disputes between these positions is more about valuesthan practices in context Is a consultant recommendation best for the stakeholders or the consultant?

23. Test Strategy and Approach Strategy is a thought process not a document

24. Contexts of Test Strategy Axioms Communication Early Testing Risks De-Duplication Test Strategy Opportunities Goals Automation Culture Contract User involvement Constraints Human resource Artefacts Skills Environment Process(lack of?) Timescales

25. IEEE 829 Test Plan Outline Test Plan Identifier Introduction Test Items Features to be Tested Features not to be Tested Approach Item Pass/Fail Criteria Suspension Criteria and Resumption Requirements Test Deliverables Testing Tasks Environmental Needs Responsibilities Staffing and Training Needs Schedule Risks and Contingencies Approvals Based on IEEE Standard 829-1998

26. I’m no fan of IEEE 829 Used as a strategy checklist Scarily vague (don’t go there) Used as a documentation template/standard Flexible, not prescriptive, but encourages copy and edit mentality (documents that no one reads) But many many testers seek guidance on What to consider in a test strategy Communicating their strategy to stakeholders and project participants

27. IEEE 829 Plan and Axioms Items 1, 2 – Administration Items 3+4+5 – Scope Management, Prioritisation Item 6 – All the Axioms are relevant Items 7+8 – Good-Enough, Value Item 9 – Stakeholder, Value, Confidence Item 10 – All the Axioms are Relevant Item 11 – Environment Item 12 – Stakeholder Item 13 – All the Axioms are Relevant Item 14 – All the Axioms are Relevant Item 15 – Fallibility, Event Item 16 – Stakeholder Axioms

28. A Better Test Strategy and Plan Stakeholder Objectives Stakeholder management Goal and risk management Decisions to be made and how (acceptance) How testing will provide confidence and be assessed How scope will be determined Design approach Sources of knowledge (bases and oracles) Sources of uncertainty Models to be used for design and coverage Prioritisation approach Delivery approach Test sequencing policy Repeat test policies Environment requirements Information delivery approach Incident management approach Execution and end-game approach Plan (high or low-level) Scope Tasks Responsibilities Schedule Approvals Risks and contingencies

29. Testing Improvement Test process improvement is a waste of time

32. Actually its 11 (most were not software related)

33. The delusion of process models(e.g. CMM) Google search “CMM” – 22,300,000 “CMM Training” – 48,200 “CMM improves quality” – 74 (BUT really 11 – most of these have NOTHING to do with software) A Gerrard Consulting client… CMM level 3 and proud of it (chaotic, hero culture) Hired us to assess their overall s/w process and make recommendations (quality, time to deliver is slipping) 40+ recommendations, only 7 adopted – they couldn’t change How on earth did they get through the CMM 3 audit?

34. “Test Process Improvement is a Waste of Time” Using process change to fix cultural or organisational problems is never going to work Improving test in isolation is never going to work either Need to look at changing context rather than values…

35. Why you are where you are Context+ Values+ Thinking =Approach <- your context <- your values <- your thinking <- your approach

36. Where maturity models come from Context+ Values+ Thinking =Approach <- someone else's <- someone else's <- someone else's <- someone else's

37. Making change happen Axioms+ Context+ Values+ Thinking =Approach <- recognise <- could change? <- hard to change <- just do some <- your approach

38. Using the axioms and questions Axioms represent the critical things to think about Associated questions act as checklists to: Assess your current approach Identify gaps, inconsistencies in current approach QA your new approach in the future Axioms represent the WHAT Your approach specifies HOW

39. Eight stage change process (after Kotter) Mission Coalition Vision Communication Action Wins Consolidation Anchoring Changes identified here If you must use one, this is where your ‘process model’ comes into play

40. A Skills framework for testers Axioms indicate WHAT to think about... ...so the Axioms point to SKILLS

41. Test design is based on models (p68) Summary: Choose test models to derive tests that are meaningful to stakeholders. Recognise the models’ limitations and the assumptions that the models make. Consequence if ignored or violated: Tests design will be meaningless and not credible to stakeholders. Questions: Are design models available to use as test models? Are they mandatory? What test models could be used to derive tests from the Test Basis? Which test models will be used? Are test models to be documented or are they purely mental models? What are the benefits of using these models? What simplifying assumptions do these models make? How will these models contribute to the delivery of evidence useful to the acceptance decision makers? How will these models combine to provide sufficient evidence without excessive duplication? How will the number of tests derived from models be bounded?

42. Test design and modelling skills A tester needs to understand: Test models and how to use them How to select test models from fallible sources of knowledge How to design test models from fallible sources of knowledge Significance, authority and precedence of test models How to use models to communicate The limitations of test models Familiarity with common models Is this all that current certification provides?

43. Training and certification must change Intellectual skills and capabilities are more important than the clerical skills Need to re-focus on: Testing thought processes (Axioms) Testing Stakeholder relationship management Testing as an information provision service Goal and risk-based testing Real-world examples, not theory Practical, hands-on, real-world training, exercises and coaching.

44. Quantum Testing If evidence arrives in discrete quanta... ...can we assign a value to it?

45. How testing builds confidence As tests are run, every individual test has some significance Some tests expose failures but ultimately we want all tests to PASS When all tests pass – the stakeholders are happy, aren’t they? Can we measure confidence by counting tests? Not really...

46. The value of a test varies by… Coverage model: A test could cover one or hundreds of functional conditions, ten thousand program statements or ten Objective: Criticality of the business goal it examples Criticality of the risk it informs Precedent: The first end-to-end test pass is significant The 100th variation of a similar test is less significant Functional dependence: A test of shared functionality used thousands of times per hour could be much more important than a peripheral feature used once/day Stakeholder: Are customers tests more or less significant than supplier tests? Context: The same test run at different times in different environments can have different value.

48. Testers can’t quantify value, but could define significance

49. A test is significant (to stakeholders) if it:

50. Can be related to a meaningful test objective

51. Increases coverage with respect to a meaningful test model

52. Is considered in an acceptance decision (at any level)

53. Significance is a Boolean but could be 0 or 1

55. The significance of significance Test coverage models and goals that generate uniform distributions of tests are inefficient and uninformative We need more and better test models Models that are meaningful in context Significance varies with context and can be used to explain why e.g. some tests aren’t useful as regression tests How much testing is enough? Can never be answered by coverage alone.

56. Using significance (as booleans) A test objective could be deemed as met when a set of significant tests are passed Test objectives can often be aligned with business goals (or risks) e.g.: Faultless end to end or straight-through processing in an ERP system Customer order capture, money-in or complete financial reconciliations without failure Could the business put a value on these goals being achieved? Perhaps the business case provides this level of information?

57. Our current understanding of test value measurement In terms of a mathematical treatment of testing we are some way off Compare with: Understanding of heat and energy in 18thcentury? Economics in the early 20thCentury? A complete theory of test value may never be achieved Quantum testing suggests that: Stakeholders judge value (relatively speaking) Testers judge significance.

58. Close Axioms are context-neutral rules for testing The Equation of Testing Separates axioms, context, values and thinking We can have sensible conversations about process Axioms and associated questions provide context neutral checklists for test strategy, assessment/improvement and skills QT separates significance from value Could it tell us, “how much testing is enough”? Right now, we can ‘feel the heat’, but not measure temperature.

59. THE AXIOMS TESTING OF testaxioms.com testers-pocketbook.com gerrardconsulting.com uktmf.com Thank-You!