How to improve your testing in an agile environment

1. Test Improvement 4 Agile

2. The Evolution of IT and Testing
Pioneering
Structuring
Op>mizing

3.
Prototyping
Waterfall
Agile/Scrum
V-­‐model
Spiral
model
DEVOPS
Itera>ve
RAD
TDD
ATDD
BDD
W-­‐model
XP
Con>nuous
Integra>on/Delivery
MBT
Exploratory
CDT
RST
RSTM
TMap
ISEB
ISTQB
TMap-­‐Next
Packages
SOA
Devices
Social
media
Virtualiza>on
Cloud
Mobile
Internet
of
Things
Web
Localiza>on
Big
Data
The Evolution of IT and Testing

4. Manifesto for Agile Software Development
We are uncovering better ways of developing
software by doing it and helping others do it.
Through this work we have come to value:
Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
That is, while there is value in the items on
the right, we value the items on the left more.

6. Test Improvement 4 Agile roadmap
Quick
wins,
good
prac>ces
Goals,
scope
Interviews,
mee>ngs
Assessment
Awareness,
commitment,
buy-­‐in
Implement
and
evaluate

7. Test Improvement 4 Agile roadmap
Cost

8. Agile testing maturity levels
Forming Norming Performing

9. Agile testing maturity levels
Forming Norming Performing
Set the basis and the first
steps towards working in
an Agile manner
Adopt a process
that facilitates
the Agile view
on working
Continuously
improve the way
you work by
living the Agile
way

10. Assessment model
Key area Forming Norming Performing
1 Key area 1 1 2 3 4 1 2 3 4 1 2 3 4
2 Key area 2 1 2 3 4 1 2 3 4 1 2 3 4
3 Key area 3 1 2 3 4 1 2 3 4 1 2 3
… …… …
…
…
…
… … … … … … … …
n Key area n 1 2 3 1 2 3 1 2 3 4
• Where am I?
• Where do I want to be?
• How do I get there?

13. Test process
• What and when to test?
• Finish what you start
• Fully integrated
• Everybody tests
Forming Norming Performing
TI4Agile
Risk based
testing
Testing in and
over sprints
Everybody
tests

14. Keys to success
• Automated tests: unit ß à acceptance
• Non-functional testing
• Test data
• Regression testing
• Fit for purpose & context

15. Test management
• People management
• Generic test approach / strategy
• Risk analysis
• Release planning
• Birds eye view:
Cross teams – sprints – projects
Forming Norming Performing
TI4Agile
Generic
process
Support the
process
Support the
people

16. Generic test approach / strategy
Keys to success
Risks
Maturity
Skills
Product
Size
Time
Culture
Bandwidth

17. The wheel of risk
Core
Layer 1
Layer 2
Core
Layer 1
Layer 2
Core
Layer 1
Layer 2
Core
Core
Core

18. Planning & Estimation
• Work breakdown & plan for all activities
– Preparation, Review, Execution, Retest, Unit test, Bug
fixing
• It is a team effort!
Forming Norming Performing
TI4Agile
Know what to
plan and
estimate
Measurable,
small parts
Time is fixed,
scope is
flexible

19. System
Item Item Item Item
Item Item Item Item
Item Item Item Item
Keys to success
13
820
13

20. Defect management
• Use one system for all defects and ensure traceability
• Decide what and when to log
Forming Norming Performing
TI4Agile
Defects are
logged
Not all defects
are logged
Focus on
working
software

21. Keys to success
• Take into account:
– Co-location
– Maturity of team
– Number of defects
– Organizational influence
– Drive for metrics
– Size of development
– Complexity of development

22. Test environment
• Self controlled
• Production like
• Version controlled
• Continuous integration
Forming Norming Performing
TI4Agile
There is an
environment
The team is
stakeholder
The
environment is
flexible

23. Keys to success
Virtualization Version control
1
4
9
2
3
6
7
5
8
T1
T2 10
Tags
Trunks
Branches
Merges
Discontinued
development
branch

24. Test automation
• At every level
• Important part of the sprint
• Risk based
• Maintainable
Forming Norming Performing
TI4Agile
Test
automation
provides
added value
Test
automation at
different test
levels
Maintainable
and continuous

25. Keys to success
Maintainable Continuous integration
Testware
Test scripts – Test cases – Test data – Test
environment
System Under Test
Software – Service – Platform - Infrastructure
Management Reporting
Projects
Changes
Patches
Fixes
Test Automation

26. Regression & E2E testing
• Focus wider than current sprint
– Probably wider than the team
• Assign where and when
• Business value
Forming Norming Performing
TI4Agile
Regression
and E2E in
focus
In and over
iteration
Proactively
addressed

27. Keys to success
• Understandable, visualized, clear E2E information
• Optional:
– Hardening Sprint focused on improving the product
• E2E, Regression, Loose ends, Shortcuts, Non-functionals

28. People
Forming Norming Performing
TI4Agile
People are
motivated
People
embrace
changes in the
product
People initiate
process
changes

29. Keys to success
Feedback
• State something positive related to the subject
• State your criticism objectively
• Don’t use the word “but”
• State the effect
• Suggest an improvement
Thank you for delivering a lot of information in the meeting. I noticed that you
were talking a lot, this provided me little room to give my opinion. You might
want to ask others for their input in the future.

30. Keys to success
• Know the context
• Assist others
• Know your own strengths
• Leave your comfort zone
30

31. Interaction
• Actively share information
• Open and honest
• Use meetings for their purpose
– Understandable for all attendees
Forming Norming Performing
TI4Agile
Traceable
decisions
Focussed
meetings
Ad-hoc, when
needed

32. Keys to success
Communication = essential • SCRUM-board
32

33. Teamwork
• Respect and trust each other
• Commitment as a team
– Work towards a team goal
• Help each other where possible
– Multi disciplined
Forming Norming Performing
TI4Agile
The team
executes a
task
The team is
committed
The team is
self managing

34. Keys to success
• Keep the goals visible • Team development
Bruce W. Tuckman:
Stages of group development
34

35. Test profession
• Test skills
• Soft skills
• Technical skills (drivers, stubs, logging, …)
• Share test knowledge
– Within the discipline, over teams
– Within team, over disciplines
Forming Norming Performing
TI4Agile
Proficient in
testing
Focus on soft
and technical
skills
Profession
includes more
than just
testing

36. Key to success
• Charters, SBTM, …
• Exploratory testing
– Simultaneously learn, plan, design, execute and report

37. Stakeholder commitment
• Deliver acceptance criteria and share the context
• Recognise the value of the team and of each role
• Prioritise backlog, risks and defects
• Create an effective environment
• Participate in acceptance
• Allow freedom of choice
Forming Norming Performing
TI4Agile
Stakeholders
start projects
Stakeholders
help projects
Stakeholders
participate in
projects

38. The path to commitment
Contact
Awareness
Understanding
Posi2ve
percep2on
Adop2on
Ins2tu2onali-­‐
za2on
Internaliza2on
Level
of
commitment

39. Assessment model
Key area Forming Norming Performing
1 Stakeholder commitment 1 2 3 4 1 2 3 4 1 2 3
2 Planning & Estimation 1 2 3 4 1 2 3 4 1 2 3 4
3 People 1 2 3 4 1 2 3 4 1 2 3
4 Interaction 1 2 3 4 1 2 3 4 1 2 3 4
5 Teamwork 1 2 3 1 2 3 4 1 2 3 4
6 Test process 1 2 3 1 2 3 4 1 2 3
7 Test management 1 2 3 4 1 2 3 1 2 3
8 Test profession 1 2 3 4 1 2 3 4 1 2 3
9 Test automation 1 2 3 1 2 3 4 1 2 3 4
10 Regression & E2E testing 1 2 3 4 1 2 3 1 2 3 4
11 Defect management 1 2 3 1 2 3 4 1 2 3
12 Test environment 1 2 3 1 2 3 1 2 3 4

40. Transitioning from traditional to agile
Process Waterfall
Development
Iterative &
Incremental
Agile
Development
Measure of
Success
Management
Culture
Requirements &
Design
Coding &
Implementation
Test & Quality
Assurance
Planning &
Scheduling
Conformance
to plan
Response to change,
working code
Command &
control
Leadership collaborative,
empowerment of teams
Big & upfront
documentation
Continuous / emergent /
just in time elaboration
Code all features in
parallel. Test later
Code & unit test,
deliver serially
Big, planned / test late Continuous & concurrent
testing starting early
detailed / fixed scope,
estimate time & resources
Two-level plan / fix
date, estimate scope
(Scaling Software Agility: Best Practices for large Organisations – DeanLeffingwell 2007)

41. Key area Forming Norming Performing
1 Stakeholder commitment 1 2 3 4 1 2 3 4 1 2 3
2 Planning & Estimation 1 2 3 4 1 2 3 4 1 2 3 4
3 People 1 2 3 4 1 2 3 4 1 2 3
4 Interaction 1 2 3 4 1 2 3 4 1 2 3 4
5 Teamwork 1 2 3 1 2 3 4 1 2 3 4
6 Test process 1 2 3 1 2 3 4 1 2 3
7 Test management 1 2 3 4 1 2 3 1 2 3
8 Test profession 1 2 3 4 1 2 3 4 1 2 3
9 Test automation 1 2 3 1 2 3 4 1 2 3 4
10 Regression & E2E testing 1 2 3 4 1 2 3 1 2 3 4
11 Defect management 1 2 3 1 2 3 4 1 2 3
12 Test environment 1 2 3 1 2 3 1 2 3 4
Questions?

42. Test Improvement 4 Agile
Ruud Teunissen
Senior Test Consultant
Polteq Test Services BV
ruud.teunissen@polteq.com – @RuudTeunissen

43. 01. Stakeholder commitment –
Forming
1. The principal stakeholder is defined and known by the
testers
2. Stakeholders deliver the committed resources
3. Stakeholders actively acquire information on the
progress of the project
4. Stakeholders are willing to adapt their way of working
to the test process

44. 01. Stakeholder commitment –
Norming
1 The stakeholders provide a mandated representative
(PO) to the team
2 The stakeholders define business value and provide a
prioritized product backlog
3 The stakeholders define acceptance criteria for the
items on the product backlog
4 Stakeholders attend the review meeting

45. 01. Stakeholder commitment –
Performing
1 The product owner is actively participating in the
team activities
2 Stakeholders trust and value the team(s)
3 Stakeholders share responsibility for the quality of the
product

46. 02. Planning & Estimation –
Forming
1. Each test activity is planned (prepare - define -
execute)
2. Test levels are defined and overlap is minimized (UT –
ST - AT)
3. A technique for test effort estimation is applied
4. Planning is agreed with the stakeholders

47. 02. Planning & Estimation –
Norming
1 Estimation is used to estimate team effort (cross
functional)
2 The planning meeting is used to elaborate on user
stories
3 Testing work is broken down into measurable, small
tasks (preferably 8hrs or less)
4 Burn-down (or up) charts are used to keep track of
progress

48. 02. Planning & Estimation –
Performing
1 Scope is adjusted when estimation seems to be
incorrect, since time boxes are fixed
2 Changes on user stories are fit into the planning
wherever possible
3 The team(s) decide what is delivered in an iteration
4 Team velocity is used in planning

49. 03. People –
Forming
1. People are well trained and/or experienced in their
functions
2. People are willing to put in extra effort when needed
(commitment)
3. People can explain their value in the project context
4. People take full responsibility for their work

50. 03. People –
Norming
1 People understand SCRUM terminology and know the
purpose of the different SCRUM meetings
2 SCRUM master keeps track of the process
3 SCRUM master removes roadblocks outside the team
4 People have a positive attitude towards change

51. 03. People –
Performing
1 People proactively provide feedback
2 People know how to handle feedback and use the
feedback to improve
3 People are able to help with tasks outside their main
area of expertise (T-shaping)

52. 04. Interaction –
Forming
1. It is possible to trace back points of action,
agreements and decisions
2. Progress, product quality and risks are communicated
3. Meetings are only attended by relevant people
4. Tooling to support the necessary communication
within the project is supplied

53. 04. Interaction –
Norming
1 The stand-up meeting is used to actively monitor
progress and adjust the planning accordingly
2 The planning meeting is used to discuss and identify
risks
3 The retrospective deals with positive points and
improvement suggestions
4 Documentation reflects the current state of the
product

54. 04. Interaction –
Performing
1 People use ad-hoc meetings to obtain or communicate
relevant information
2 Testers share their test ideas as soon as possible
3 Documentation is fit for purpose (just enough)
4 Good practices are shared over the teams (and
projects)

55. 05. Teamwork –
Forming
1. The team is co-located
2. Each team member is involved in important decisions
3. The team understands its goal

56. 05. Teamwork –
Norming
1 Testing is an integrated task of a multi disciplinary
team
2 The team commits to the scope in a sprint
3 SCRUM coaching is provided to improve the teamwork
4 The team contains all skills necessary to deliver a
working product

57. 05. Teamwork –
Performing
1 Team members know and value each other
2 Specialists share their expertise to improve the
knowledge in the team
3 The team is responsible for a sufficient quality
product
4 Team members can take on different roles instead of
being limited to their function

58. 06. Test process –
Forming
1. The effort for test tasks is based on product risks
2. Test design techniques are applied when creating test
cases
3. Testing considers different quality characteristics

59. 06. Test process –
Norming
1 Non-functionals are addressed in and over sprints
2 Pairing is used to improve the test quality
3 The Definition of Done includes items related to
testing
4 Developers use simple techniques for unit testing

60. 06. Test process –
Performing
1 Testing debt is monitored and managed
2 User acceptance is fully integrated in the process
3 All team members have a responsibility for the test
process

61. 07. Test management –
Forming
1. Test management allocates the proper test staff
2. A generic test approach is available
3. Test management provides birds eye view on quality
and risks in the project
4. Test management contributes to the release planning
by adding a quality view to high level estimation

62. 07. Test management –
Norming
1 Test management initiates risk identification at
product backlog level
2 Test management supports the product owner(s) to
identify dependencies between user stories prior to
finalizing the release planning
3 Test management supports the SCRUM master in
removal of test roadblocks

63. 07. Test management - Performing
1 Test management facilitates test meetings to share
the testing pitfalls and lessons learned over teams
(and projects)
2 Test management provides mentoring to the teams to
improve the quality of testing
3 Test management increases awareness and
understanding of Agile testing within the organization

64. 08. Test profession –
Forming
1. Testers have received specific test training and/or
have sufficient experience in the field of structured
testing
2. Testers can explain the rationale behind chosen
techniques that have been applied
3. Test functions are part of the organizations career
development
4. Testers can explain their added value

65. 08. Test profession –
Norming
1 Testers have received training(s) regarding soft skills
2 Technical activities are part of the testing job (e.g.
working with drivers/stubs, code reviews)
3 Testers provide support to testing activities of other
team members
4 Testers use exploratory testing in addition to using
structured techniques

66. 08. Test profession –
Performing
1 Testers are proficient in at least one programming
language
2 Testers are proactively gathering relevant information
to be able to execute their tasks and to minimize
waiting time
3 Testers strive to continuously improve their testing

67. 09. Test automation –
Forming
1. The organization acknowledges the importance of test
tooling and allocates sufficient budget
2. Test cases are automatable by describing them using
‘inputs - actions - expected results’
3. Decisions on what to automate are based on risks and
return on investment

68. 09. Test automation –
Norming
1 Automated unit testing is part of the development
process
2 Tests are automated to save time for manual testing
3 Test cases are designed for automation
4 Test automation is seen and treated as software
development

69. 09. Test automation –
Performing
1 Automation is modular and maintainable
2 Requirements are set up to be executable
3 Teams are allowed to try different tooling to reach
their own optimal setting
4 Test automation is used to support continuous
integration

70. 10. Regression & E2E testing –
Forming
1. There is a basic strategy for regression testing
2. There is a basic strategy for E2E testing
3. Test sets are maintained and updated
4. Responsibilities for E2E and regression testing are
clear

71. 10. Regression & E2E testing –
Norming
1 Regression testing is part of every iteration
2 Regression and E2E testing is applied over multiple
iterations and when needed over multiple teams and
multiple projects
3 The user stories contain information on relevant E2E
functionality

72. 10. Regression & E2E testing –
Performing
1 The team is aware of the value of and need for E2E
and regression testing
2 All testing is used to address current business value
3 Regression test sets can be easily scoped to address
specific areas of functionality
4 E2E test possibilities are utilized continuously

73. 11. Defect management –
Forming
1. All persons involved in logging and/or tracking defects
use the same defect tracking tool
2. Defects have a common set of characteristics (e.g.
unique id, description, status)
3. All raised defects are followed up

74. 11. Defect management –
Norming
1 There are simple criteria for deciding when to log a
defect
2 Defects that can not be fixed within the sprint have to
be added to the product backlog
3 Defects, either logged or not logged, are
communicated with the complete team
4 Impact of defects on the product, but also on the
sprint needs to be taken into account

75. 11. Defect management –
Performing
1 The team strives to fix defects as soon as possible
(<1 day)
2 The person that logged the defect helps to solve the
defect
3 Defects are logged with the minimal set of
information needed to follow up the defects

76. 12. Test environment –
Forming
1. The environment is managed (setup, maintenance,
version management, etc.)
2. The environment is sufficiently representative for the
test
3. The data used in the environment can easily be
refreshed

77. 12. Test environment –
Norming
1 The test environment is always available to the team
2 Changes to the environment are communicated to the
involved teams
3 The version management system used for the
environment supports branching

78. 12. Test environment –
Performing
1 Testing takes place on all relevant environments
2 The environment supports the use of continuous
integration
3 The team manages its own environments
4 A mechanism to efficiently support testing in different
configurations is in place (e.g. virtualization)