The levels 2 and 3 of TMMi, the successor of TMM, are launched in 2008. By now, the company of the presenter performed almost 20 assessments, most informal assessments. Assessments where performed in telecom, utilities, financial services and (semi) government. The first conclusions about the maturity of testing (in the Netherlands) can be drawn. In the presentation the structure of TMMi will be introduced briefly, as well as the way the assessments where done. Then the results will be presented, as well as the conclusion. The overall picture is that test process improvement is something testers do in there cave. To improve testing rigorous we have to get out of our cave. At the end of the presentation instructions are given how to do this.

1. Tester, get out of your cave!
Jan Jaap Cannegieter
Vice President SYSQA B.V.

2. ©SYSQAAlmere
2

3. ©SYSQAAlmere
3
Why are we here?

4. ©SYSQAAlmere
4
Agenda
• TMMi in under 10 minutes
• Data from 20 TMMi workshops
• Conclusions
• How to get out of your cave

6. ©SYSQAAlmere
6
TMMi – essentials
• Test maturity framework
• Benchmark model for assessments
• 5 levels
• Based on:
• TMM
• Evolutionary Testmodel (Gelperin / Hetzel)
• CMMI for Development
• In line with ISTQB terminology
• Developed by the TMMi-foundation
• Public domain
• www.tmmifoundation.org

7. ©SYSQAAlmere
7
TMMi levels
1. Initial
2. Managed
Test Policy and Strategy
Test Planning
Test Monitoring and Control
Test Design and Execution
Test Environment
3. Defined
Test Organization
Test Training Program
Test Life Cycle and Integration
Non-Functional Testing
Peer Reviews
4. Management
and Measurement
Test Measurement
Software Quality Evaluation
Advanced Peer Reviews
5. Optimization
Defect Prevention
Test Process Optimization
Quality Control

8. ©SYSQAAlmere
8
How I got my data
• Goals
• Learn
• Presales
• Assignments
• 20 organizations
• Telecommunication: 2
• Government: 7
• Financial services: 9
• Industry: 2
• Analysis

9. ©SYSQAAlmere
9
Quick scans – a demo

10. Results
Fasten your seatbelts

11. ©SYSQAAlmere
11
Test policy and strategy - goals
• Establish a test policy
• Establish a test strategy
• Establish test performance indicators

12. ©SYSQAAlmere
12
Test policy and strategy: A: 30% V: 14%
0
10
20
30
40
50
60
70
80
90
100
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Test policy and strategy
Organizations
Percentageofspecificgoalsmet

13. ©SYSQAAlmere
13
Test planning - goals
• Perform a product risk assessment
• Establish a test approach
• Establish test estimates
• Develop a test plan
• Obtain commitment to the test plan

14. ©SYSQAAlmere
14
Test planning: A: 41%, V: 13%
0
10
20
30
40
50
60
70
80
90
100
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Test planning
Organizations
Percentageofspecificgoalsmet

15. ©SYSQAAlmere
15
Test monitoring and control - goals
• Monitor test progress against plan
• Monitor product quality against plan and
expectations
• Manage corrective action to closure

16. ©SYSQAAlmere
16
Test monitoring and control: A: 47%, V: 15%
0
10
20
30
40
50
60
70
80
90
100
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Test monitoring and control
Organizations
Percentageofspecificgoalsmet

17. ©SYSQAAlmere
17
Test design and execution - goals
• Perform test analysis and design using test design
techniques
• Perform test implementation
• Perform test execution
• Manage test incidents to closure

18. ©SYSQAAlmere
18
Test design and execution: A: 60%, V: 8%
0
10
20
30
40
50
60
70
80
90
100
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Test design and execution
Organizations
Percentageofspecificgoalsmet

19. ©SYSQAAlmere
19
Test environment - goals
• Develop test environment requirements
• Perform test environment implementation
• Manage and control test environments

20. ©SYSQAAlmere
20
Test environment: A: 59%, V: 9%
0
10
20
30
40
50
60
70
80
90
100
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
Test environment
Organizations
Percentageofspecificgoalsmet

21. ©SYSQAAlmere
21
Overview
0
10
20
30
40
50
60
70
80
90
100
Per process area

22. ©SYSQAAlmere
22
My conclusion
60%: Test design and execution
30%: Test policy and strategy
41%: Test planning
47%: Test monitoring and control
59%: Test environment
Inside
Outside

23. ©SYSQAAlmere
23
We are in a cave!

24. ©SYSQAAlmere
24

25. ©SYSQAAlmere
25
In case you disagree…

26. ©SYSQAAlmere
26
Best practices ‘outside of your cave’
• Perform a product risk assessment and establish a test
approach with relevant stakeholders
• Obtain commitment for your plan from relevant stakeholders
• Plan stakeholder involvement in your project
• Monitor stakeholder involvement in your project
• Make release advice with people outside of testing
• Establish and maintain a test strategy with relevant
management
• Define, measure and report test performance indicators to
relevant stakeholders
• Establish and maintain a test policy with higher management
(business and IT)

27. Stakeholder management
27
The basics

28. ©SYSQAAlmere
28
Stakeholder
A person or group of persons who will be involved in or
effected by your test project or improvement project

29. ©SYSQAAlmere
29
Stakeholder management
• Stakeholder identification
• Stakeholder analysis
• Stakeholder involvement

30. ©SYSQAAlmere
30
Stakeholder aspects
• Organizational level
• Power
• Business interest
• Mindset
• Ambition
• Behavioral style
• Stakeholder type

31. ©SYSQAAlmere
31
Type of stakeholders
Demanding Discretionary
Sleeping
Dependent
Dangerous Dominant
Definitive
Urgency Legitimacy
Power
Mitchell, Agle, Wood, 1997

32. ©SYSQAAlmere
32
Stakeholdercard – an example

33. Please think this over
I’ll turn on some inspiring music

34. ©SYSQAAlmere
34

35. ©SYSQAAlmere
35

36. ©SYSQAAlmere
36
Stuart Reid’s test competence model for
testers
Test skills
IT skills
Domain
knowledge
Soft skills

37. ©SYSQAAlmere
37
My test competence model for test
consultants
Test skills
IT skills
Domain
knowledge
Soft skills
Political
skills

38. ©SYSQAAlmere
38

39. ©SYSQAAlmere
40
Email for the stakeholder card:
jcannegieter@sysqa.nl

40. ©SYSQAAlmere
41